The architecture

Three pieces:

  • Loki — the database. Stores log streams indexed by labels, with content compressed and chunked into object storage.
  • Promtail / Grafana Alloy — the agent. Reads files / journald / Docker sockets / Kubernetes pods, applies labels, ships batched lines to Loki.
  • Grafana — the UI. Same Grafana you'd use for Prometheus metrics; Loki is added as a data source.

In 2026 Promtail is being superseded by Grafana Alloy (the merger of Promtail, Grafana Agent, and the OpenTelemetry collector). The Promtail-specific config below still works; Alloy uses the same concepts in a slightly different config language.

Loki: single-binary install

For a single-node setup with local filesystem storage:

sudo useradd -r -s /sbin/nologin loki
sudo mkdir -p /var/lib/loki /etc/loki
sudo chown -R loki:loki /var/lib/loki

LV=3.4.0
curl -L -o /tmp/loki.zip \
    "https://github.com/grafana/loki/releases/download/v${LV}/loki-linux-amd64.zip"
unzip /tmp/loki.zip -d /tmp
sudo mv /tmp/loki-linux-amd64 /usr/local/bin/loki

# Sample config — single binary, filesystem storage
sudo tee /etc/loki/loki.yaml <<'EOF'
auth_enabled: false

server:
  http_listen_port: 3100
  log_level: info

common:
  path_prefix: /var/lib/loki
  replication_factor: 1
  ring:
    kvstore: { store: inmemory }
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory:  /var/lib/loki/rules

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 90d
  ingestion_rate_mb: 10

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  delete_request_store: filesystem
EOF

# systemd unit
sudo tee /etc/systemd/system/loki.service <<'EOF'
[Unit]
Description=Loki
After=network-online.target

[Service]
User=loki
Group=loki
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki.yaml
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now loki

For real production, replace filesystem with S3 / MinIO storage (see MinIO setup) — same schema_config, different common.storage block.

Promtail on each log source

PV=3.4.0
curl -L -o /tmp/promtail.zip \
    "https://github.com/grafana/loki/releases/download/v${PV}/promtail-linux-amd64.zip"
unzip /tmp/promtail.zip -d /tmp
sudo mv /tmp/promtail-linux-amd64 /usr/local/bin/promtail

sudo tee /etc/promtail/promtail.yaml <<'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.lab.example.com:3100/loki/api/v1/push

scrape_configs:
  # systemd journal
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: HOSTNAME-PLACEHOLDER
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label:  'unit'
      - source_labels: ['__journal_priority_keyword']
        target_label:  'level'

  # nginx access logs
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job:  nginx
          host: HOSTNAME-PLACEHOLDER
          __path__: /var/log/nginx/*.log
EOF

# Replace HOSTNAME-PLACEHOLDER with the actual hostname
sudo sed -i "s/HOSTNAME-PLACEHOLDER/$(hostname)/g" /etc/promtail/promtail.yaml

sudo tee /etc/systemd/system/promtail.service <<'EOF'
[Unit]
Description=Promtail
After=network-online.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail.yaml
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now promtail

The label discipline

Loki's whole performance model relies on a small, bounded number of label values. Do not put high-cardinality data (request IDs, user IDs, URLs) into labels — they explode the index. Put them in the log line content; LogQL can still filter on them at query time.

Good labels: job, host, container_name, namespace, severity, environment. Each should have a handful of values across the fleet, not thousands.

Wiring Grafana

Install Grafana the usual way (apt repo / docker). Add a Loki data source: Configuration → Data sources → Loki → URL http://loki.lab.example.com:3100. Test → Save.

Now in Explore → pick the Loki data source. The label browser shows everything Promtail has shipped so far.

LogQL by example

# All logs from nginx on web1
{job="nginx", host="web1"}

# nginx logs containing "500 "
{job="nginx"} |= "500 "

# nginx logs that match a regex and parse it as JSON
{job="nginx"} | json | status >= 500

# Top 10 source IPs by request count over 5min
topk(10, sum by (remote_addr) (rate({job="nginx"} | json [5m])))

# Error rate per service over 1m
sum by (job) (rate({severity="error"}[1m]))

# Tail the live log of a specific container
{container_name="myapp"} |= "" | line_format "{{.log}}"

The pattern is similar to PromQL: stream selector in {}, then filters (|= contains, != not contains, |~ regex, !~ not regex), then parsers (| json, | logfmt, | pattern), then aggregations.

Alerts

Grafana's unified alerting can rule on LogQL expressions. Example: alert if "out of memory" appears more than 5 times in 5 minutes on any host:

sum by (host) (count_over_time({severity="critical"} |= "out of memory" [5m])) > 5

Or as a Loki ruler rule (managed by Loki itself, without Grafana):

# In Loki's rules directory
groups:
  - name: oom
    rules:
      - alert: HostOOMSpotted
        expr: sum by (host) (count_over_time({severity="critical"} |= "out of memory" [5m])) > 0
        for: 0m
        annotations:
          summary: "OOM on {{ $labels.host }}"

Retention and storage

Loki indexes are tiny compared to ElasticSearch's, but the log chunks themselves still grow. With retention_period: 90d in limits_config and the compactor running, old chunks are deleted automatically. For longer retention without the cost, send Loki's chunk storage to S3/MinIO + lifecycle rules that move cold months to Glacier-tier.

Worth knowing

  • Don't over-label. The Loki failure mode is "the index is huge because every request_id became a label." Keep it small.
  • Promtail file positions — if Promtail crashes or restarts, it resumes from the position file. Delete that file to force a re-read of all matched files.
  • For containers, Promtail's docker_sd_configs or Kubernetes service discovery picks up new containers automatically and labels them with their metadata. For Podman, use the journal scraper since Podman logs to journald by default.
  • Cost vs ElasticSearch: at the ingest rate and retention most homelabs and small teams hit, Loki is 3–10× cheaper to run in storage and orders of magnitude cheaper in CPU. The trade-off is no full-text search on old chunks — Loki scans chunks linearly within a label-narrowed time window, which is fine for small queries and gets slow for "search everything for this string across 90 days."