Skip to main content
Helm chart Warehouse workers process interactive and background SQL queries against your data warehouse. When a user runs a query in Lightdash, the API server publishes a job to the NATS warehouse stream, and a warehouse worker picks it up, executes the SQL, and returns the results.

ā€‹
Example configuration

A complete Helm values configuration with NATS and a warehouse worker: 
nats:
  enabled: true
  config:
    cluster:
      enabled: false
    jetstream:
      enabled: true
      fileStore:
        enabled: false
      memoryStore:
        enabled: true
        maxSize: 1Gi

warehouseNatsWorker:
  enabled: true
  replicas: 1
  concurrency: 100
  resources:
    requests:
      cpu: 250m
      memory: 1.5Gi
    limits:
      memory: 1.5Gi
The chart auto-configures NATS_ENABLED=true and NATS_URL for you. See the overview for details on JetStream configuration options.

ā€‹
Configuration reference

All configuration is set through your Helm values.yaml under warehouseNatsWorker:

ā€‹
Scaling

Helm valueDefaultDescription
warehouseNatsWorker.replicas1Number of worker pods. Scale horizontally for more parallel query capacity.
warehouseNatsWorker.concurrency100Maximum concurrent jobs per pod. Maps to NATS_WORKER_CONCURRENCY env var.

ā€‹
Resources

Helm valueRecommended (request)Recommended (limit)Description
warehouseNatsWorker.resources.requests.cpu250mā€”CPU request per pod
warehouseNatsWorker.resources.requests.memory1.5Gi1.5GiMemory request and limit per pod

ā€‹
Auto-configured environment variables

These are set by the Helm chart:
VariableSet fromValue
NATS_ENABLEDnats.enabled: true"true"
NATS_URLnats.enabled: truenats://<release>-nats:4222
NATS_WORKER_CONCURRENCYwarehouseNatsWorker.concurrency100

ā€‹
Optional environment variables

These can be set via extraEnv or configMap if you need to override the defaults:
VariableDefaultDescription
NATS_QUEUE_TIMEOUT_MS180000 (3 min)How long a message can wait in the queue before being discarded.

ā€‹
Troubleshooting

ā€‹
Queries timing out

If workers are busy, messages may expire before being processed. Scale up replicas or concurrency.

ā€‹
Worker OOM kills

Increase the memory request and limit. Large query result sets are held in memory during processing.

ā€‹
NATS connection errors

  1. Confirm the NATS pod is running and healthy
  2. Verify that NATS_URL is set correctly on the warehouse worker pod (should be nats://<release>-nats:4222)
  3. Check that your network policies allow traffic between the worker pods and the NATS service on port 4222