Availability: Pre-aggregates are a Beta feature available on Enterprise plans only.
Managed and external pre-aggregates
Pre-aggregates come in two flavors, distinguished by who owns the underlying table:- Managed pre-aggregates are the default. Lightdash materializes the rollup on your warehouse, stores the result, and serves matching queries from that stored copy. You only write the definition.
- External pre-aggregates delegate the materialization to you. You point a pre-aggregate at a warehouse table you build and refresh yourself, and Lightdash uses the definition only for matching and routing. See External pre-aggregates for the full workflow.
How it works
Pre-aggregates follow a four-step cycle:- Define — You add a
pre_aggregatesblock to your dbt model YAML, specifying which dimensions and metrics to include. - Materialize — Lightdash runs the aggregation query against your warehouse and stores the results. This happens automatically on compile, on a cron schedule you define, or when you trigger it manually.
- Match — When a user runs a query, Lightdash checks if every requested dimension, metric, and filter is covered by a pre-aggregate.
- Serve — If a match is found, the query is served from the materialized data instead of hitting your warehouse.
Example
Suppose you have anorders table with thousands of rows, and you define a pre-aggregate with dimensions status and metrics total_amount (sum) and order_count (count), with a day granularity on order_date.
Your warehouse data:
Lightdash materializes this into a pre-aggregate:
Now when a user queries “total amount by status, grouped by month”, Lightdash re-aggregates from the daily pre-aggregate instead of scanning the full table:
This works because
sum can be re-aggregated — summing daily sums gives the correct monthly sum.
Defining pre-aggregates
Pre-aggregates are defined under thepre_aggregates key in your model configuration.
If you’re using Lightdash YAML instead of dbt model YAML, see the Lightdash YAML syntax guide for the surrounding model structure.
- dbt v1.9 and earlier
- dbt v1.10+ and Fusion
- Lightdash YAML
Configuration reference
If you specify
time_dimension, you must also specify granularity, and vice versa.Query matching
When a user runs a query, Lightdash checks whether a pre-aggregate can serve it first. A pre-aggregate matches when the query fits inside it on each axis:- Fields are available — every dimension, metric, and filter dimension in the query exists somewhere in the pre-aggregate.
- Grain is reachable — if the query uses a time dimension, its granularity is equal or coarser than the pre-aggregate’s, so the rows can be rolled up. Month is a coarser grain than day. Day is a coarser grain than hour.
- Scope is compatible — if the pre-aggregate defines its own
filters, the query includes an equal or narrower filter, so the subset can be filtered from the pre-aggregate base. - Metrics re-aggregate cleanly — all metrics are supported types. Non-additive metrics like
count_distinct,median, andpercentilecan’t be faithfully re-computed from stored rows, so they only match on an exact match of the pre-aggregate. Anything resolved at runtime — raw SQL table calculations,sql_filtermetrics, or SQL dependent on Parameters and user attributes — is never eligible.
Exact match queries
A query is an exact match of a pre-aggregate when its selected dimensions are set-equal to the pre-aggregate’s dimensions and its time dimension is at exactly the pre-aggregate’s granularity. On an exact match, each result row is served from a single materialization row without any re-aggregation. This unlocks metric types that can’t otherwise be re-aggregated — see Non-additive metrics on exact matches. For a query to count as an exact match:- Every pre-aggregate dimension must appear in the query’s selected dimensions, and vice versa.
- The time dimension must be selected at exactly the pre-aggregate’s granularity — not coarser, not finer.
- Filters on selected dimensions are allowed. They only subset the stored rows, so the match still holds.
- A pre-aggregate dimension referenced only by a query filter does not count as selected, and the query is no longer an exact match.
- A dimension reached through a custom bin does not count as selected either — bins collapse groups and break the one-row-per-result guarantee.
Filtered pre-aggregates
A pre-aggregate can define staticfilters so it materializes only a slice of the source data for a common query pattern, such as status = completed or a rolling order_date: inThePast 52 weeks window. A query then matches it only when it carries the same filter or a narrower one — the scope-compatibility rule above — expressed with the same filter operator.
See Filtered pre-aggregates for the definition syntax and a worked matching example.
Dimensions from joined tables
Pre-aggregates support dimensions from joined tables. Reference them by their full name (for example,customers.first_name) in the dimensions list.
Filtered pre-aggregates
Usefilters when you want a pre-aggregate to materialize only a subset of the source data.
For example, this pre-aggregate only stores data for the last 52 weeks:
- dbt v1.9 and earlier
- dbt v1.10+ and Fusion
- Lightdash YAML
How query matching works with filters
Filtered pre-aggregates are only used when the query filters are compatible with the pre-aggregate definition:- A query with the same or narrower filter can use the pre-aggregate
- A query without the filter, or with a broader or incompatible filter, falls back to another pre-aggregate or the warehouse
order_date inThePast 12 weekscan use the pre-aggregateorder_date inThePast 52 weekscan use the pre-aggregateorder_date inThePast 104 weekscannot use the pre-aggregateorder_date is 2026-01-15cannot use the pre-aggregate, even though the date falls inside the last 52 weeks (see the operator-matching note below)- no
order_datefilter: cannot use the pre-aggregate
Required filters and pre-aggregates
Models can declarerequired_filters that every query on the explore must apply. Pre-aggregates coexist with required filters, with a few rules on both sides.
How required filters are applied
Required filters are applied when a query reads from the pre-aggregate, not baked permanently into the materialized table. The materialization stores rows across every value of the required-filter field, and Lightdash re-applies the filter each time a query hits the rollup. This lets users override the required filter’s default value (where the model allows it) and still be served from the pre-aggregate — they don’t silently get an incomplete result from a materialization that only holds one value.Every required-filter field must be a pre-aggregate dimension
Because the filter is applied at query time, its target field has to exist as a column in the materialization. If anyrequired_filters target on the model isn’t listed in the pre-aggregate’s dimensions, the pre-aggregate is ineligible for that explore and Lightdash queries the warehouse instead. This applies to fields on the base table and on joined tables. Sibling time-dimension grains (for example, a required filter on created_at_week when the pre-aggregate’s time dimension is created_at at day grain) also need the underlying dimension in the pre-aggregate.
Only filters actually marked required: true count. Model filters marked as not required don’t need to be in the pre-aggregate.
Don’t duplicate required-filter targets in filters
The pre-aggregate’s own filters narrow the materialization at build time and can’t be overridden at query time. Setting an explicit pre-aggregate filter on the same field as a required_filters target creates a conflict — the required filter is meant to be overridable by the user, but the pre-aggregate filter isn’t. Lightdash treats these queries as a miss (pre_aggregate_filter_not_satisfied) rather than silently returning partial results.
Keep required-filter fields out of the pre-aggregate’s filters block. If you need to narrow the materialization on a required-filter field, split it into a separate pre-aggregate that doesn’t overlap.
Multiple pre-aggregates per model
You can define multiple pre-aggregates on the same model, each targeting different query patterns. It is better to have multiple small, focused pre-aggregates rather than a single one containing all metrics and dimensions. Including too many dimensions increases the number of unique combinations, which generates large materialization files — this defeats the purpose of pre-aggregates, since they are meant to be smaller and faster than querying the warehouse directly. For example, you might want a fine-grained daily pre-aggregate for detailed dashboards and a coarser monthly one for summary views:Scheduling refreshes
By default, pre-aggregates are materialized when your dbt project compiles. You can also schedule automatic refreshes using cron expressions, using your project’s configured timezone (defaults to UTC):Materialization triggers
Pre-aggregates can be materialized through four different triggers:Row limits
You can setmax_rows to cap the size of a materialization. If the aggregation produces more rows than the limit, the result is truncated.
Materialization sort order
Usesorts to control the order rows are written in the materialized table. Sorting the materialization on the dimensions you filter and group by most often can make downstream reads faster.
sorts is a list of entries. Each entry has:
fieldId— the canonical field ID of a dimension included in the pre-aggregate. Joined-table fields use thetable.fieldform.descending— boolean, required.truesorts high to low,falsesorts low to high.
sorts key accepts four shapes, each with a different meaning:
Every
fieldId in sorts must also appear in the pre-aggregate’s dimensions list. Metrics and time dimensions expanded from time_dimension + granularity use their canonical IDs (for example, orders_order_date_day).Materialization role
materialization_role is useful when access to the model depends on required_attributes or any_attributes.
For example, if a joined table is only available to users with region_access: emea, then materializing a pre-aggregate without a fixed access context could produce different results depending on who triggered the build.
Use materialization_role to make materialization run with a stable set of user attributes.
This is intended for access control fields such as:
- dbt v1.9 and earlier
- dbt v1.10+ and Fusion
Supported metric types
Pre-aggregates support two kinds of metrics. Re-aggregatable metrics work for any matching query, including ones at a coarser grain or on a subset of the pre-aggregate’s dimensions:sumcountminmaxaverage
count_distinctsum_distinctaverage_distinctmedianpercentile
non_additive_metric_requires_exact_match — the fix is to select exactly the pre-aggregate’s dimensions at exactly its granularity.
Non-additive metrics on exact matches
Non-additive metrics likecount_distinct, sum_distinct, average_distinct, median, and percentile normally can’t be re-aggregated from a rollup, because combining group-level values produces the wrong answer (see Metrics that need re-aggregation to combine). But on an exact match there is nothing to re-aggregate: each result row corresponds to exactly one materialization row, so the stored value is already the correct answer.
This is useful when a count_distinct (or another non-additive metric) is the slowest part of a query. Define a pre-aggregate whose dimensions and time-dimension granularity match how the metric is queried, and Lightdash serves those queries from the materialization instead of hitting the warehouse.
Current limitations
Pre-aggregates support a narrower subset of the Lightdash semantic layer than regular warehouse queries.Not supported
Pre-aggregates do not support:- Personal warehouse connections. Materialization always runs under a single user’s credentials, so warehouse-level access rules are not applied per viewer. If you rely on personal warehouse connections to enforce data access, use results caching instead.
- Parameters — parameter values are picked at query time, so they cannot be resolved during materialization. Queries that use parameters fall back to the warehouse.
- User attributes when referenced from SQL.
required_attributesandany_attributesare still supported throughmaterialization_role. - Custom metrics created in the Explorer
- Custom SQL dimensions created in the Explorer (Custom bin dimensions are supported)
- SQL table calculations (Formula table calculations are supported)
SQL compatibility
sql_filter (and its alias sql_where) runs both at materialization time and at query time on top of the materialized data.
- At materialization time, the filter is evaluated against your warehouse. If the SQL references Parameters or user attributes, the values injected come from the materialization context — you can pin this to a fixed identity or attribute set with
materialization_roleso the materialization captures the rows you need. - At query time, the same filter is re-applied against the materialized data, which is served by DuckDB. If the
sql_filterSQL uses warehouse-specific syntax that DuckDB doesn’t understand, the query will fail to run against the pre-aggregate and fall back to the warehouse.
Metrics that need re-aggregation to combine
Pre-aggregates do not support metric types that cannot be re-aggregated from pre-computed results. For example, considercount_distinct on a daily pre-aggregate. If the pre-aggregate stores “2 distinct customers on 2024-01-15” and “1 distinct customer on 2024-01-16”, you cannot sum those daily values to get the monthly distinct count, because the same customer can appear on multiple days.
Re-aggregating gives
2 + 1 = 3, but the correct monthly answer is 2 (Alice, Bob). The pre-aggregate no longer knows which customers were counted.
We’re investigating supporting count_distinct through approximation algorithms. Follow this issue for updates.
For similar reasons, the following metric types are also not supported:
sum_distinct,average_distinctmedian,percentilepercent_of_total,percent_of_previousrunning_total- Custom SQL / post-calculation metrics (including many
numbermetrics) — Follow this issue number,string,date,timestamp,boolean
Pre-aggregates vs results caching
Pre-aggregates and results caching are independent systems that speed up queries in different ways, and they work best together: pre-aggregates serve matching queries from materialized summary tables — no warehouse hit, even on the first query — while results caching stores the exact result of any query shape after its first run. A query that hits a pre-aggregate can also have its result cached, layering the two. For the full comparison — a feature-by-feature table and guidance on when to use each — see Results caching vs pre-aggregates.Complete example
Here’s a full model definition with a pre-aggregate, including joins, scheduling, and row limits:- dbt v1.9 and earlier
- dbt v1.10+ and Fusion
- Lightdash YAML
- Total order amount by status, grouped by day, week, month, or year
- Average order size by status, grouped by month
- Total order amount filtered to completed orders
- Order amount by customer country, grouped by quarter
- Queries grouped by a dimension not in the pre-aggregate (for example,
customer_id) - Queries with hourly granularity (finer than the pre-aggregate’s
day) - Queries without
status = completedor with a broaderstatusfilter - Queries with Parameters, user attributes inside SQL, or
sql_filter - Queries including a non-additive metric like
count_distinctunless they select exactly the pre-aggregate’s dimensions at exactly its granularity (see Exact match queries) - Queries with raw SQL table calculations