How it works
Usage analytics separates event capture, scheduled processing and interactive queries. A working query only verifies the data and storage access needed by that query.events/raw/, partitioned by organization, event stream and UTC date. Capture runs asynchronously and is best-effort: buffered events can be lost if a process stops or uploads exhaust their retries. Creating the analytics project is not required for capture.
The scheduler’s compactUsageEvents task runs the following stages in order on the usage-events-compaction queue:
The refresh follows the data freshness schedule. It processes closed UTC days; today’s raw events wait for a subsequent refresh. Inventory snapshots reflect the database when that stage runs.
Publication is separate for each output, not one transaction across the pipeline. A failed refresh can leave some datasets updated and others missing or showing their last complete version. People activity charts depend on user summaries; its population and returning/inactive-person charts also use the membership snapshot. Query timing charts can still read available query-event files while either People dependency is missing or stale.
For interactive queries, the backend checks the caller’s analytics permissions and selects files for the authorized organization. DuckDB reads those Parquet files using short-lived signed URLs and returns results through the normal Explore query flow. It does not query your business warehouse for this data.
Configure the deployment
- Upgrade Lightdash to 3.156.0 or later and keep the backend, scheduler and workers on the same release.
- Apply the enterprise license to all Lightdash components.
- Configure S3-compatible object storage. Usage storage inherits the base S3 configuration, not the pre-aggregate storage settings. To use a separate bucket or credentials, set the usage analytics storage variables. The base S3 configuration is still required.
- Ensure the backend, scheduler and any query workers can reach the storage endpoint. The capture and refresh processes need to list, read, write and delete usage objects; analytics queries need authenticated reads. Keep the bucket private and configure credentials through your deployment’s secret management.
- Keep the scheduler running to process captured data and publish inventory snapshots.
USAGE_EVENTS_S3_BUCKET to a bucket without one.
Enable processing
Usage analytics is available by default with a valid Enterprise license and configured usage storage. An organization administrator can create the analytics project without requesting feature enablement. Nightly processing selects organizations that have an analytics project. Creating the project starts background processing; organizations without one continue capturing events but are skipped by the nightly job. Raw event capture runs on deployments with a license key and working usage storage configuration. RudderStack telemetry is a separate system: disabling it does not disable usage analytics. Deleting the analytics project stops future nightly selection for that organization. It does not stop raw uploads, cancel queued or running jobs, or delete stored usage files. Retain or expire usage objects through your storage policy, including raw files for organizations without an analytics project.Verify setup
Create the analytics project and generate activity in that organization, then allow the scheduled data refresh to complete. Follow Check a processing run to verify the full pipeline. Successful project creation verifies access to storage, but does not prove that events have been collected or processed. Run a chart in the analytics project for a closed day with known activity. Check that the user, content and date match that activity. To update installed dashboards after an upgrade, follow Update models and dashboards. For retained database records from before capture began, use historical backfill. Usage files can contain user identities and content metadata. Restrict bucket access and choose a retention policy that matches your organization’s requirements.Troubleshooting processing
Start with scheduler logs for the affected refresh and identify which stage failed or left work pending. Use your deployed release when investigating: available models, log fields and monitoring signals depend on that release.Check a processing run
Search scheduler logs forUsage processing stage started and Usage processing stage finished. Correlate records with the same runId. Finished records include stage, durationSeconds and outcome; stage summaries can also include failed, remaining and limitReached.
Check the final record with stage="pipeline", which covers the full sequence:
A successful
compaction stage is not proof that inventory and user summaries finished. A completed queue job alone is also insufficient: reported compaction failures can mark the pipeline as failed even when the job handler returns normally. A start record without a matching finish can indicate an interrupted or still-running process; check worker health and elapsed time.
After a successful pipeline run, refresh a chart for a closed day with known activity. Pipeline success describes the work discovered by that run; it cannot prove that every event was captured.
Monitor refresh completion
If you enable Prometheus metrics, scrape the scheduler workers and use the full-pipeline success timestamp:lightdash_usage_processing_running and lightdash_usage_processing_last_started_timestamp_seconds show current stage execution. lightdash_usage_processing_work reports failed or remaining work and whether a stage reached its work limit. Counts have different units across stages, so do not add them together.
These metrics belong to individual processes and reset when workers are replaced. Use retained scrape history across scheduler replicas, keep deployments separate, and treat an absent success timestamp as unknown rather than healthy. Monitor failed and partial outcomes as well as the age of the last success; a recent success does not clear a later failure. Logs and worker-health checks are still needed for crashes between scrapes.
Investigate empty People charts while query charts work
Open the affected chart’s Explore from here action to identify its source. For User activity, inspect theusers stage. For People population and returning/inactive-person metrics, inspect the dimensions stage and its membership snapshot errors. Existing event files can serve latency charts even when these outputs are missing or stale.
Also check the chart’s date filters and capture coverage. A missing partition is unknown coverage, not proof that nobody was active. For metric definitions and supported historical data, use the product guide.
Resolve storage errors
Check the usage storage configuration on every backend, scheduler and worker replica. Confirm the endpoint is reachable and that the configured identity has the required list, read, write and delete access to usage objects. A successful read of an existing event file does not verify permission to publish snapshots or clean up raw inputs. After fixing credentials, permissions or networking, roll out any changed configuration and follow Recover a failed or partial run. Preserve raw files and compaction checkpoints underevents/compaction/; the pipeline uses them to resume cleanup without publishing the same inputs twice. Events that were never successfully uploaded cannot be recovered by rerunning processing.
Investigate a missing run
Check that the scheduler is running, can reach the application database and storage, and resolves the organization as eligible for processing. If you use scheduler task inclusion or exclusion, ensurecompactUsageEvents is allowed. Inspect the scheduler queue and worker logs for a run that is still active before arranging another run.
Recover a failed or partial run
Fix the reported cause, then check the next serialized run. Nightly jobs make one whole-job attempt; individual storage operations can retry within that attempt. Deferred work queues follow-up history-processing jobs, which have their own retries. The nightly job also reschedules exhausted history-processing jobs. Nightly processing prioritizes the previous seven closed UTC dates, excluding today. Follow-up history jobs process older retained raw files and rebuild user activity summaries from compacted events. This is separate from the database history backfill. Work limits can leave a run partial, so monitor subsequent outcomes until the backlog clears. Storage retention still determines which files remain available. If scheduled recovery stops making progress or you need an immediate rerun, contact the Lightdash team with your deployed version, affected UTC dates,runId, and sanitized stage errors. Coordinate the rerun through the scheduler’s serialized queue; do not run competing standalone compactors or enqueue duplicate jobs while a run is active. Do not include credentials, event payloads, or signed storage URLs in a support report.
For model and dashboard changes, use Sync content; it does not rerun this pipeline.