Skip to main content
Enterprise Pre-aggregates need a valid LIGHTDASH_LICENSE_KEY set on your instance before any of the configuration below takes effect. See enterprise features for applying the key.
Pre-aggregates materialize aggregated tables to S3 and route matching queries to DuckDB instead of your warehouse, so common dashboard queries return in milliseconds without hitting the warehouse. To enable them on a self-hosted instance you need an S3-compatible bucket for the materialized data and at least one pre-aggregate worker to run the materializations.

​
Prerequisites

  • Enterprise license — LIGHTDASH_LICENSE_KEY must be set on your instance.
  • S3-compatible storage — a bucket Lightdash can write Parquet files to. This can be the same bucket used for external object storage, or a dedicated one.
  • NATS enabled — set NATS_ENABLED=true and NATS_URL. Materializations run through the NATS pre-aggregate stream.
  • At least one pre-aggregate worker — see pre-aggregate workers for the worker deployment.

​
Enable pre-aggregates

Set the main switch, keep Parquet materialization on, and point Lightdash at the bucket:
Authenticate to the bucket with PRE_AGGREGATE_RESULTS_S3_ACCESS_KEY and PRE_AGGREGATE_RESULTS_S3_SECRET_KEY, or omit them to use an IAM role or the default AWS credential chain. PRE_AGGREGATES_PARQUET_ENABLED=false keeps pre-aggregates in-memory only, lost on restart. To bound DuckDB memory per worker, set PRE_AGGREGATE_DUCKDB_QUERY_MEMORY_LIMIT (e.g. 4GB). The full variable list is in the pre-aggregates reference. Deploy at least one pre-aggregate worker and restart the backend. Project developers can then define pre-aggregates in dbt and Lightdash will materialize and serve them.

​
Verifying pre-aggregates

  1. Add a pre-aggregate definition to a dbt model and deploy.
  2. Trigger a materialization from the Lightdash UI or wait for the scheduled run.
  3. Confirm a Parquet file lands in the configured bucket.
  4. Run a query that matches the pre-aggregate — the query timing panel shows pre-aggregate as the source.

​
Troubleshooting

Materialization jobs stay queued forever. No pre-aggregate worker is consuming the NATS stream. Confirm at least one worker pod is running against the same NATS URL and is subscribed to the pre-aggregate stream. Materializations fail with an S3 permissions error. The credentials in PRE_AGGREGATE_RESULTS_S3_ACCESS_KEY / _SECRET_KEY (or the IAM role) need s3:GetObject, s3:PutObject, and s3:ListBucket on the bucket. Queries never route to a pre-aggregate. Check the pre-aggregate definition matches the query shape — see pre-aggregate query matching.