Skip to main content
🛠 This page is for engineering teams self-hosting their own Lightdash instance. For instance health metrics, see Prometheus metrics. Lightdash can export distributed traces using the OpenTelemetry SDK, so you can follow a request or scheduled job across the API server, scheduler, and warehouse queries in any OpenTelemetry-compatible backend (for example Grafana Tempo, Jaeger, Honeycomb, or Datadog). Tracing runs in one of two exclusive modes:
  • Sentry mode (default): spans are created and exported through Sentry, controlled by the Sentry environment variables.
  • OpenTelemetry mode: spans are created by the OpenTelemetry SDK and exported according to the standard OTEL_* environment variables. Sentry still captures errors, but receives no spans.

​
Enabling OpenTelemetry tracing

By default, Lightdash traces through Sentry. To switch to OpenTelemetry mode, set the following environment variable on every Lightdash container (API server and scheduler):
Then point the OTLP exporter at your collector:

​
Configuration options

OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are overridden by Lightdash. Control sampling with LIGHTDASH_OTEL_TRACES_SAMPLE_RATE instead.

​
Choosing an exporter

Lightdash defers exporter and protocol selection to the OpenTelemetry Node SDK’s standard environment variable handling:
  • OTEL_TRACES_EXPORTER supports otlp (default), console, zipkin, and none. You can combine exporters with a comma-separated list. If the list contains none, no traces are exported regardless of the other values.
  • With the otlp exporter, OTEL_EXPORTER_OTLP_PROTOCOL (or the traces-specific OTEL_EXPORTER_OTLP_TRACES_PROTOCOL) selects grpc, http/json, or http/protobuf (default).
  • Unsupported exporter or protocol values are ignored with a warning in the Lightdash logs at startup.
For example, to export traces over gRPC with an authentication header:

​
Sampling

LIGHTDASH_OTEL_TRACES_SAMPLE_RATE sets the head-sampling ratio for trace roots, from 0.0 (nothing) to 1.0 (everything, the default). Child spans follow their root’s decision, so a sampled request captures the whole waterfall. Two behaviours to be aware of:
  • AI agent traces are always sampled regardless of the ratio, so a broken agent run always has a trace to debug. Set LIGHTDASH_OTEL_ALWAYS_SAMPLE_AI_TRACES=false to make them follow the global ratio instead.
  • OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are overridden by Lightdash’s own sampler and have no effect.
Requests to health checks (/api/v1/health, livez), status polling endpoints, favicon.ico, and robots.txt are never traced.

​
Finding a request’s trace

Every API response carries an x-trace-id header with the trace ID of the request that produced it, so you can go from a request in your proxy or application logs to its trace in your backend. The same ID appears in the Lightdash logs for that request. The header is only set when the request was sampled. Requests dropped by LIGHTDASH_OTEL_TRACES_SAMPLE_RATE have no exported trace and no header. In OpenTelemetry mode the backend does not continue the sentry-trace and traceparent headers sent by the Lightdash web app, because the browser span is only exported to Sentry and would leave the trace without a root. A traceparent header set by your own ingress or middleware is continued as normal.

​
Database query tracing

Set LIGHTDASH_OTEL_DB_TRACES_ENABLED=true to add a span for each application database (Postgres) query, so you can see where a request spends time inside Lightdash’s own database:
  • Database spans only appear inside an existing trace, so they follow the sampling decision of their parent request or job.
  • Each span records the SQL statement, truncated to LIGHTDASH_OTEL_DB_TRACES_MAX_QUERY_LENGTH characters (default 1022). Invalid values fall back to the default with a warning.
  • This traces queries to Lightdash’s application database only, not queries sent to your data warehouse.

​
Troubleshooting

At startup, Lightdash logs a line confirming the tracing configuration, including the active exporters, OTLP protocol, and sampling ratio. This confirms configuration only; exporter connectivity is not validated at startup. If traces aren’t arriving in your backend, set OTEL_LOG_LEVEL=DEBUG to enable OpenTelemetry SDK and exporter diagnostics in the Lightdash logs. Diagnostic output is redacted before logging: URL credentials and query strings are stripped, values containing tokens or API keys are omitted, and span payloads are not printed.

​
Metrics

This page covers traces only. For instance health metrics (CPU, memory, event loop, query durations), see Prometheus metrics, which can also feed an OpenTelemetry backend through the collector’s Prometheus receiver.