Skip to main content
Enterprise Pre-aggregates need a valid LIGHTDASH_LICENSE_KEY set on your instance before any of the configuration below takes effect. See enterprise features for applying the key.
Pre-aggregates materialize aggregated tables to S3 and route matching queries to DuckDB instead of your warehouse, so common dashboard queries return in milliseconds without hitting the warehouse. A self-hosted instance needs two things to run them: the license key and an S3-compatible bucket for the materialized data. A pre-aggregate worker is recommended for large materializations, and required when NATS is enabled.

​
Prerequisites

  • Enterprise license — LIGHTDASH_LICENSE_KEY must be set on your instance.
  • S3-compatible storage — a dedicated bucket Lightdash can write Parquet files to. Don’t reuse the results cache bucket: query history cleanup would delete active materializations. See S3 storage configuration for the bucket variables and provider examples.

​
Where pre-aggregates run

Where materializations and pre-aggregate queries execute depends on whether NATS is enabled:
With NATS enabled and no worker on the pre-aggregate stream, queries that match a pre-aggregate wait in the queue for NATS_QUEUE_TIMEOUT_MS (3 minutes by default) and then fail. Nothing falls back to the warehouse.

​
Enable pre-aggregates

Set the main switch, keep Parquet materialization on, and point Lightdash at the bucket. Lightdash refuses to start when PRE_AGGREGATES_ENABLED is true without a bucket.
Authenticate to the bucket with PRE_AGGREGATE_RESULTS_S3_ACCESS_KEY and PRE_AGGREGATE_RESULTS_S3_SECRET_KEY, or omit them to use an IAM role or the default AWS credential chain. With PRE_AGGREGATES_PARQUET_ENABLED unset, Lightdash stores materializations as JSON Lines files, which DuckDB reads more slowly; keep it true. To bound the memory DuckDB uses to serve queries, set PRE_AGGREGATE_DUCKDB_QUERY_MEMORY_LIMIT (for example 4GB). The full variable list is in the pre-aggregates reference. Restart the backend, and the scheduler and workers if you run them. Project developers can then define pre-aggregates in dbt and Lightdash materializes and serves them.

​
Switching pre-aggregates off

Once the instance has the license key and the bucket, pre-aggregates are on for every organization through the pre-aggregates feature flag. To switch them off without removing the configuration, add the flag to the disable list and restart:
The disable list wins over every other setting. On Lightdash Cloud the same flag switches a single organization off.

​
Verifying pre-aggregates

  1. Add a pre-aggregate definition to a dbt model and deploy.
  2. The first compile that includes it materializes it. Later compiles rebuild it only when its definition changes; refresh.cron and manual refreshes cover the rest. See Scheduling refreshes.
  3. Confirm a Parquet file lands in the configured bucket.
  4. Run a query that matches the pre-aggregate — the query timing panel shows pre-aggregate as the source.

​
Troubleshooting

Materialization jobs stay queued forever, or matching queries fail after 3 minutes. NATS is enabled and no worker consumes the pre-aggregate stream. Confirm a worker pod runs against the same NATS URL and was started either without --stream or with --stream pre-aggregate. A worker started with --stream warehouse never picks these jobs up. Materializations fail with an S3 permissions error. The credentials in PRE_AGGREGATE_RESULTS_S3_ACCESS_KEY / _SECRET_KEY (or the IAM role) need s3:GetObject, s3:PutObject, and s3:ListBucket on the bucket. Queries never route to a pre-aggregate. Check the pre-aggregate definition matches the query shape — see pre-aggregate query matching.