Skip to main content
Helm chart
This page is for engineering teams self-hosting their own Lightdash instance.
By default, Lightdash processes all queries on the main API server. NATS workers move query execution onto dedicated pods, improving responsiveness under load and letting you scale query capacity independently. Lightdash uses NATS — a lightweight, high-performance messaging system — with JetStream, its built-in persistent streaming layer, to distribute work between the API server and worker pods. NATS powers two opt-in features in Lightdash:

Warehouse workers

Process interactive and background warehouse queries on dedicated pods.

Pre-aggregate workers

Materialize pre-aggregates and serve queries from DuckDB.

​
Requirements

  • Helm chart version 2.7.2 or later
  • Lightdash version 0.2675.0 or later. Older images will fail with MODULE_NOT_FOUND.
Upgrading the Helm chart alone does not change how Lightdash works. NATS features are entirely opt-in — your existing deployment will behave exactly the same until you explicitly enable the new Helm values described below.

​
Architecture

The Lightdash API publishes jobs to NATS JetStream. Worker pods consume messages from their stream and process them concurrently (default 100 concurrent jobs per pod).

​
Enabling NATS

nats:
  enabled: true
  config:
    cluster:
      enabled: false
    jetstream:
      enabled: true
      fileStore:
        enabled: false
      memoryStore:
        enabled: true
        maxSize: 1Gi
The JetStream configuration shown above reflects the defaults when nats.enabled is set to true. This deploys a NATS StatefulSet and sets NATS_ENABLED=true on the backend, which means the backend will start routing queries through NATS. You should always enable at least a warehouse worker alongside NATS to process those queries — otherwise queries will be enqueued with no worker to pick them up.
Do not enable nats.enabled: true without also enabling warehouseNatsWorker.enabled: true. The backend routes queries to NATS when NATS_ENABLED is set, so queries will stall if no worker is running to process them.

​
Auto-configured environment variables

The chart automatically sets these environment variables in the shared ConfigMap — you do not need to set them manually:
VariableSet whenValue
NATS_ENABLEDnats.enabled: true"true"
NATS_URLnats.enabled: truenats://<release>-nats:4222
Additional environment variables are auto-configured per worker deployment — see Warehouse workers and Pre-aggregate workers for details.

​
NATS JetStream configuration

JetStream supports two storage backends — we recommend memory store, but you can use file store depending on your needs.

​
Memory store vs file store

Memory store (recommended)File store
How it worksMessages are held in RAMMessages are persisted to disk
PerformanceFaster — no disk I/O overheadSlower — writes go through disk
PersistenceMessages are lost if NATS restartsMessages survive NATS restarts
InfrastructureNo PersistentVolumeClaim neededRequires a PersistentVolumeClaim
When to useMost deployments. Lightdash messages are small (just a query UUID) and are deleted once processed.High message volume exceeding available RAM, or if you need messages to survive NATS pod restarts.
For more details, see the NATS documentation on JetStream and stream storage types.

​
Configuration reference

nats:
  enabled: true
  config:
    cluster:
      enabled: false          # single-node NATS, no clustering
    jetstream:
      enabled: true
      fileStore:
        enabled: false         # no disk persistence
      memoryStore:
        enabled: true
        maxSize: 1Gi           # max memory for message storage
SettingRecommendedDescription
nats.config.jetstream.memoryStore.enabledtrueEnable memory-backed storage
nats.config.jetstream.memoryStore.maxSize1GiMaximum memory for JetStream message storage
nats.config.jetstream.fileStore.enabledfalseEnable disk-backed storage
nats.config.cluster.enabledfalseSingle-node NATS (no clustering)

​
Pod disruption

NATS is a stateful component — if the NATS pod restarts, in-flight messages are lost (queries will be retried by users). The chart protects against unplanned eviction with:
  • cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation
  • PodDisruptionBudget with maxUnavailable: 0