Skip to main content
Enable managed pre-aggregates on your self-hosted instance with object storage and, when NATS is enabled, a dedicated worker. This page covers deployment setup; use the feature documentation for definitions, query matching, and refresh schedules.

​
Prerequisites

  • Enterprise license — LIGHTDASH_LICENSE_KEY must be set on your instance.
  • S3-compatible storage — a dedicated bucket Lightdash can write Parquet files to. Don’t reuse the results cache bucket: query history cleanup would delete active materializations. See S3 storage configuration for the bucket variables and provider examples.

​
Where pre-aggregates run

Where materializations and pre-aggregate queries execute depends on whether NATS is enabled:
With NATS enabled and no worker on the pre-aggregate stream, matching queries remain queued until they expire. Warehouse execution fallback handles serving errors, not a missing worker.

​
Enable pre-aggregates

Set the main switch, keep Parquet materialization on, and point Lightdash at the bucket. Lightdash refuses to start when PRE_AGGREGATES_ENABLED is true without a bucket.
Configure the storage endpoint and authentication using S3 storage configuration. Bucket credentials fall back to the base S3 credentials; use the default AWS credential chain when no explicit credentials are configured. Keep PRE_AGGREGATES_PARQUET_ENABLED=true for Parquet materializations. Set a DuckDB query memory limit using the worker sizing guidance. The full variable list is in the pre-aggregates reference. Restart the backend, scheduler, and any workers that use this configuration. Follow define pre-aggregates to configure a materialization.

​
Turn pre-aggregates off

Pre-aggregates are enabled for the whole instance, with no per-organization switch. To turn them off, set:
Restart the backend, scheduler, and workers that use this configuration.

​
Verify pre-aggregates

After deploying a definition, confirm a Parquet file appears in the configured bucket. Use monitoring and debugging to check materialization status and query routing.

​
Troubleshooting

Materialization jobs stay queued, or matching queries expire. NATS is enabled and no worker consumes the pre-aggregate stream. Confirm a worker pod runs against the same NATS URL and was started either without --stream or with --stream pre-aggregate. A worker started with --stream warehouse never picks these jobs up. Materializations fail with an S3 permissions error. Check the endpoint, credentials, and bucket permissions in S3 storage configuration on every process that materializes or serves queries. Queries never route to a pre-aggregate. Check the pre-aggregate definition matches the query shape — see pre-aggregate query matching.