.. _Telemetry:

==========
Telemetry
==========

The project uses `OpenTelemetry <https://opentelemetry.io/>`_ to send trace samples to Azure Monitor (Azure Insights) for performance monitoring.

Accessing telemetry data
=========================
When Azure exported is used, data is sent to one of the following Azure Insights instances (depending which connection string is used):

`Production <https://portal.azure.com/#@megsupporttools.com/resource/subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourceGroups/MEG/providers/microsoft.insights/components/Production/performance>`_
    Collects data from all regions in Azure as well as Google Cloud production sites.
    Production data is also exposed in `Site Performance <https://portal.azure.com/#@megsupporttools.com/dashboard/arm/subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourcegroups/dashboards/providers/microsoft.portal/dashboards/9a421baf-e806-4a98-a34d-77039be4b149>`_
    and `Monitoring <https://portal.azure.com/#@megsupporttools.com/dashboard/arm/subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourcegroups/dashboards/providers/microsoft.portal/dashboards/789d2416-56a9-4827-9aab-bcab5357c1ff>`_  dashboards.

`Staging <https://portal.azure.com/#@megsupporttools.com/resource/subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourceGroups/MatStaging/providers/microsoft.insights/components/staging-insights/performance>`_
    Used for testing, collects data from staging sites (where enabled), and local environments (if enabled)

The above instances show per-view performance over time, with drill down and filtering options.

Configuration
==============
Set :ref:`global environment variables` to configure telemetry

.. code-block:: yaml
    :caption: Sample testing configuration to send all traces to Azure insights

    INSIGHTS_CONNECTION_STRING: "InstrumentationKey=fdee2f99-54eb-4e1b-9049-60290ae7a5b6;IngestionEndpoint=https://westeurope-3.in.applicationinsights.azure.com/;LiveEndpoint=https://westeurope.livediagnostics.monitor.azure.com/"
    INSIGHTS_SAMPLE_RATE: 1.0
    REGION_NAME: Test Region
    INSIGHTS_PSQL_INTEGRATION: 'True'
    INSIGHTS_REDIS_INTEGRATION: 'True'
    INSIGHTS_CELERY_INTEGRATION: 'True'

Exporting
----------
Traces are exported to one of the configured endpoints.

.. _telemetry sampling:

Sampling
^^^^^^^^^
Use :envvar:`INSIGHTS_SAMPLE_RATE` to set what portion of requests should be exported (0.0 to 1.0).
Low value will affect site performance less. Use high value such as 1.0 for testing.

When set to less than ``1.0``, the ``itemCount`` reported in azure will be adjusted to show the number of items represented by each item.
For example, sample rate of ``0.1`` means that 1 in 10 requests is sampled, so a single sample is representative of 10.

This means that regardless of the sample rate set, its variability by region or over time, the site traffic can be correctly estimated by summing ``itemCount`` value of each request.

For more information, visit `Sampling in Application Insights <https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api>`_ and :task:`30526`.

Exporter
^^^^^^^^^^
Only one exporter can be set at a time. The exporters are checked in the following order:

Azure Monitor
    To use Azure monitor, set :envvar:`INSIGHTS_CONNECTION_STRING` to the connection string.
    You can also enable live metrics by setting :envvar:`INSIGHTS_ENABLE_LIVE_METRICS`.

SQLite
    To save traces to a local SQLIte database, set :envvar:`TRACE_SQLITE_FILENAME` to a path of the sqlite database file.

Log
    Set :envvar:`TRACE_LOGGING_EXPORTER` to save traces to logs (:ref:`python logging`).

Instrumentors / integrations
-----------------------------
The following integrations are available. Set the relevant environment variable to ``True`` to enable.

.. list-table::
    :header-rows: 1

    * - Variable
      - Default
    * - :envvar:`INSIGHTS_PSQL_INTEGRATION`
      - enabled
    * - :envvar:`INSIGHTS_REDIS_INTEGRATION`
      - enabled
    * - :envvar:`INSIGHTS_CELERY_INTEGRATION`
      - disabled

Utilities
----------
.. seealso::
    | :class:`middleware.TraceExtrasMiddleware`
    | :func:`logger.trace_span`
    | :func:`logger.trace_decorator`

.. automodule:: telemetry
    :members:
.. automodule:: utils.telemetry
    :members: