How to Deploy Loki with Persistent Storage on AWS EFS Using Helm and Custom Security Context**

Question:

I'm trying to set up Loki for log aggregation using AWS EFS as persistent storage in my Kubernetes cluster. I followed the steps to configure Loki with EFS-backed persistence, but I encountered several issues along the way. Here's a detailed overview of the process, including my values.yaml, Helm command, and the key steps involved.

Can someone provide insights or improvements on how I can ensure my Loki container pods are using the EFS volume and have appropriate security permissions to access and write to the persistent storage?

Answer:

Here’s a step-by-step guide based on my experience setting up Loki with AWS EFS using Helm and custom security settings. I'll also explain some of the important details and configuration steps in the values.yaml file.

Prerequisites:

• EFS CSI Driver pods should be running in your Kubernetes cluster.
• AWS Load Balancer Controller should be used to set up ingress for accessing Grafana (if required).
• Ensure IAM permissions are properly configured to allow Kubernetes to mount EFS.

1. Setting Up EFS

Before starting, you need to ensure that your AWS EFS file system is properly set up and mounted. Here's the command I used to mount the EFS to the /data/loki directory:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-0bbf29876ed6298ee.efs.us-east-1.amazonaws.com:/ /data/loki

2. Configuring the User for Loki

To ensure Loki has proper permissions to write to the EFS directory, I created a separate user for Loki (in my case, I used loki with user ID 1002). I then added the user to the sudoers file to grant required privileges. Here's the process:

1. Create a Loki user with ID 1002:

   sudo useradd -u 1002 loki
   

2. Add the Loki user to the sudoers:

   sudo visudo
   # Add the following line to allow 'loki' to perform necessary file system operations
   loki ALL=(ALL) NOPASSWD: ALL
   

3. Create /data/loki directory and set permissions for Loki user:

   sudo mkdir -p /data/loki
   sudo chown -R loki:loki /data/loki
   

3. The values.yaml Configuration

Below is the key configuration for values.yaml that I used to set up Loki with persistent EFS storage.

test_pod:
  enabled: true
  image: bats/bats:v1.1.0
  pullPolicy: IfNotPresent

loki:
  enabled: true
  isDefault: true
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  persistence:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    existingClaim: loki-pvc-now
    storageClassName: "efs-sc"
    mountPath: /data            # Mount path for EFS
    subPath: "/mike"                  # Optional subPath
  securityContext:
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002
  logDirectory: /data/mike
  storage:
    chunks:
      directory: /data/chunks
    indexes:
      directory: /data/indexes
  initContainers:
    - name: init-fs
      image: busybox:latest
      command: ["sh", "-c", "sudo mkdir -p /efs/chunks /efs/indexes && chown -R 1002:1002 /data"]
      volumeMounts:
        - name: loki-storage
          mountPath: /data/mike

promtail:
  enabled: true
  config:
    logLevel: info
    serverPort: 3101
    clients:
      - url: http://{{ .Release.Name }}:3100/loki/api/v1/push

grafana:
  enabled: false
  image:
    tag: 8.3.5

proxy:
  http_proxy: ""
  https_proxy: ""
  no_proxy: "loki"

4. Explanation of the values.yaml Settings:

• Persistence Configuration:

  • We use efs-sc as the storageClassName, which corresponds to the EFS storage class.
  • mountPath is set to /data (this is where the EFS file system will be mounted inside the Loki container).

• Security Context:

  • The Loki container runs as the loki user (with UID 1002), and we grant it access to the /data directory through the fsGroup setting.

• Init Containers:

  • The init-fs container initializes the directory structure inside /data (such as chunks and indexes) before the main Loki container starts. It also ensures that the correct permissions are set for the directories.

• SubPath and Mount Path:

  • I used a subPath (/mike) in the mountPath to isolate Loki data. You can adjust this based on your specific needs.

5. Helm Installation Command

To install the Loki stack with the above values.yaml, I used the following Helm command:

helm install loki grafana/loki-stack --namespace=monitoring -f values.yaml

6. Troubleshooting Tips

• EFS Mounting Issues: Make sure your EFS file system is mounted correctly before starting the Loki containers. Double-check the mount options and file system ID.
• Permissions: Ensure that the loki user has proper permissions (read/write) for /data/loki and subdirectories. Misconfigured permissions are a common source of issues.
• SecurityContext: Double-check the securityContext settings. If the runAsUser, runAsGroup, or fsGroup aren't set correctly, the Loki container may not have the required permissions to write to the mounted EFS directory.

Conclusion

By following these steps, you can successfully deploy Loki with AWS EFS as persistent storage. If you face any issues, carefully check the EFS mounting, Loki container permissions, and Helm values to ensure everything is set up correctly.