Telemetry

The project uses OpenTelemetry to send trace samples to Azure Monitor (Azure Insights) for performance monitoring.

Accessing telemetry data

When Azure exported is used, data is sent to one of the following Azure Insights instances (depending which connection string is used):

Production: Collects data from all regions in Azure as well as Google Cloud production sites. Production data is also exposed in Site Performance and Monitoring dashboards.
Staging: Used for testing, collects data from staging sites (where enabled), and local environments (if enabled)

The above instances show per-view performance over time, with drill down and filtering options.

Configuration

Set Global variables to configure telemetry

Sample testing configuration to send all traces to Azure insights

INSIGHTS_CONNECTION_STRING: "InstrumentationKey=fdee2f99-54eb-4e1b-9049-60290ae7a5b6;IngestionEndpoint=https://westeurope-3.in.applicationinsights.azure.com/;LiveEndpoint=https://westeurope.livediagnostics.monitor.azure.com/"
INSIGHTS_SAMPLE_RATE: 1.0
REGION_NAME: Test Region
INSIGHTS_PSQL_INTEGRATION: 'True'
INSIGHTS_REDIS_INTEGRATION: 'True'
INSIGHTS_CELERY_INTEGRATION: 'True'

Exporting

Traces are exported to one of the configured endpoints.

Sampling

Use INSIGHTS_SAMPLE_RATE to set what portion of requests should be exported (0.0 to 1.0). Low value will affect site performance less. Use high value such as 1.0 for testing.

When set to less than 1.0, the itemCount reported in azure will be adjusted to show the number of items represented by each item. For example, sample rate of 0.1 means that 1 in 10 requests is sampled, so a single sample is representative of 10.

This means that regardless of the sample rate set, its variability by region or over time, the site traffic can be correctly estimated by summing itemCount value of each request.

For more information, visit Sampling in Application Insights and Task #30526.

Exporter

Only one exporter can be set at a time. The exporters are checked in the following order:

Azure Monitor: To use Azure monitor, set INSIGHTS_CONNECTION_STRING to the connection string. You can also enable live metrics by setting INSIGHTS_ENABLE_LIVE_METRICS.
SQLite: To save traces to a local SQLIte database, set TRACE_SQLITE_FILENAME to a path of the sqlite database file.
Log: Set TRACE_LOGGING_EXPORTER to save traces to logs (Python logging).

Instrumentors / integrations

The following integrations are available. Set the relevant environment variable to True to enable.

Variable	Default
`INSIGHTS_PSQL_INTEGRATION`	enabled
`INSIGHTS_REDIS_INTEGRATION`	enabled
`INSIGHTS_CELERY_INTEGRATION`	disabled

Utilities

See also

middleware.TraceExtrasMiddleware
logger.trace_span()
logger.trace_decorator()

telemetry.get_trace_attributes() → dict[str, Union[str, bool, int, float, Sequence[str], Sequence[bool], Sequence[int], Sequence[float]]]: Gets base trace attributes about this server such as version name and region name.

telemetry.init_celery_instrumentation(*args, **kwargs): Initializes instrumentation for Celery worker

telemetry.handle_celery_task(task_id, task, *args, **kwargs): Invoked before each celery task run to set-up telemetry details

telemetry.init_instrumentation(resource_name: str = 'web'): Initializes telemetry using OpenTelemetry. Use this while bootstrapping the project. Note that DJANGO_SETTINGS_MODULE must be set.

class utils.telemetry.ApiTokenTraceMixin

Mixin for API Token classes that adds user information to the current span.

Use this with classes That extend TokenAuthentication,

add_span_attrs(user, token)

Handles adding attributes to the span

The actual types of arguments may differ depending on Token Authentication implementation. User can be a user instance, or anonymous

utils.telemetry.update_span_attrs(span: Span | None = None, *, user: User)

Adds user and institution information to given span. If user is not logged in (AnonymousUser), the information will not be added to span.

Parameters:

span – the span to update, or use current span if null
user – The current user whose information is added to the span