New instance

Instructions in this section document the process of deploying a new instance or region.

Note

In order to deploy a new region, you need the following utilities:

Azure CLI
Kubernetes client
Helm
CLI utilities: base64, openssl

If you are working with Cloud Shell these utilities are already available.

Create database

Edit and use the snippet below to deploy the database using az-cli:

LOCATION=centralus
PSQL_NAME=megforms-${LOCATION:?}

PSQL_PASSWORD=$(openssl rand -base64 20)
echo "${PSQL_NAME:?} database password: ${PSQL_PASSWORD:?}"

az postgres flexible-server create \
    --resource-group meg \
    --name ${PSQL_NAME:?} \
    --location ${LOCATION:?} \
    --database-name megforms \
    --admin-user megforms \
    --admin-password "${PSQL_PASSWORD:?}" \
    --storage-size 128 \
    --backup-retention 30 \
    --geo-redundant-backup Enabled \
    --tier Burstable \
    --sku-name Standard_B2s \
    --tags "purpose=production" "region=${LOCATION:?}" \
    --version 16

See also

To get a full list of locations, run az account list-locations -o table.

To learn more about az postgres flexible-server create, visit az-cli documentation

Test connection

You can use the output connection string to try connecting to the database:

psql "postgresql://megforms:${PSQL_PASSWORD}@${PSQL_NAME}.postgres.database.azure.com/postgres?sslmode=require"

Upgrading PostgreSQL Flexible Server

You can upgrade your Azure Database for PostgreSQL Flexible Server to a new major version using the following command:

VERSION=16

az postgres flexible-server upgrade \
  --resource-group meg \
  --name ${PSQL_NAME:?} \
  --version ${VERSION:?}

Important

After completing the major version upgrade, it is mandatory to run the ANALYZE command in the database.

postgres=> ANALYZE;

See the official Azure PostgreSQL Post upgrade documentation for details.

Performance optimization and tweaking

The new database will appear in Azure PostgreSQL flexible servers. You can edit the created database to optimize requirements as needed:

See also

Imposing hard time limit on SQL queries

Networking

Tick “Allow public access from any Azure service within Azure to this server”. You can later un-tick it and instead whitelist the kubernetes IP address.

To view the list of IP addresses, use az network public-ip list -o table.

Compute + storage

The above command creates a Burstable database tier, it is suitable for testing and demoing. For production workloads, consider upgrading to GeneralPurpose once clients in the region start using it full time.

Maintenance

Set desired maintenance window based on expected usage times. For example, schedule maintenance for Sunday morning.

Reservations

Create a reservation to commit to using the resource for a minumum time period and reduce cost of the database.

Monitoring

Add the database to the relevant widgets in the monitoring dashboard

Create Kubernetes cluster

Edit and use the following snippet to deploy a new kubernetes cluster:

LOCATION=centralus
NAME=${LOCATION}

kubernetes_version=$(az aks get-versions --query 'orchestrators[-1].orchestratorVersion' -o tsv)

# Create the cluster
az aks create \
    --resource-group meg \
    --name ${NAME} \
    --location ${LOCATION} \
    --load-balancer-sku standard \
    --tier standard \
    --enable-cluster-autoscaler \
    --min-count 2 \
    --max-count 5 \
    --network-plugin kubenet \
    --tags "purpose=production" "region=${LOCATION}" \
    --attach-acr MegForms \
    --auto-upgrade-channel patch \
    --node-vm-size Standard_D2as_v6 \
    --kubernetes-version ${kubernetes_version:?} \
    --nodepool-name meg

# Make it available to the kubectl command
az aks get-credentials \
    --name ${NAME} \
    --resource-group meg

The new kubernetes cluster will appear in Azure Kubernetes services

See also

To learn more about az aks create, visit documentation

Create a static IP address & domain name

Create a public static IP address for the cluster

Creates a static IP address and adds it to DNS

export DOMAIN_NAME="${LOCATION}.qms.megit.com"
export RESOURCE_GROUP=$(az aks show --resource-group meg --name ${NAME:?} --query 'nodeResourceGroup' -o tsv)
# Create the IP address
export IP_ADDRESS=$(az network public-ip create -g ${RESOURCE_GROUP:?} --name kubernetes-prod --sku Standard --allocation-method static --query publicIp.ipAddress -o tsv)
echo "Created IP ${IP_ADDRESS} in ${RESOURCE_GROUP}"
az network dns record-set a add-record --resource-group meg --zone-name qms.megit.com --record-set-name "${LOCATION}" --ipv4-address ${IP_ADDRESS}
echo "Record set added: ${DOMAIN_NAME}

Install Helm charts

Run the following snippet to install ingress controller and cert manager. Note that it relies on IP_ADDRESS and DOMAIN_NAME set in Create a static IP address & domain name, and NAME containing cluster name.

# Ensure you're installing charts on the correct cluster
kubectl config use-context ${NAME:?}

# Add helm repos
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add jetstack https://charts.jetstack.io

# Install charts
helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.replicaCount=3 --set controller.service.loadBalancerIP="${IP_ADDRESS:?}" --set controller.service.externalTrafficPolicy=Local -n ingress
helm install cert-manager jetstack/cert-manager --set installCRDs=true -n cert-manager

# Deploy Ingress - it is important that DOMAIN_NAME variable is exported so that envsubst can access it
cat kubernetes/deployment/template.yaml | envsubst | kubectl apply -f -

Note

In order to upgrade Helm charts, use the following command:

CERT_MANAGER_VERSION=updated-cert-manager-version
NGINX_VERSION=updated-nginx-version
helm upgrade --version ${CERT_MANAGER_VERSION:?} cert-manager jetstack/cert-manager --set installCRDs=true -n cert-manager
helm upgrade --version ${NGINX_VERSION:?} ingress-nginx ingress-nginx/ingress-nginx --set controller.replicaCount=3 --set controller.service.loadBalancerIP="${IP_ADDRESS:?}" --set controller.service.externalTrafficPolicy=Local -n ingress

NGINX Telemetry

To enable telemetry for the NGINX ingress controller using the OpenTelemetry collector, apply the following manifest. This allows exporting metrics and traces from NGINX to your observability backend.

kubectl apply -f ./kubernetes/nginx-telemetry/nginx-telemetry.yaml

Note

If the OpenTelemetry collector is not deployed when telemetry is enabled in NGINX, metrics and traces will be lost, and observability will be unavailable until the collector is running. NGINX itself is unlikely to crash or become unstable.

Enabling telemetry

By default, OpenTelemetry is disabled. To enable it, you can patch the nginx-ingress-controller ConfigMap using the following command:

kubectl patch configmap nginx-ingress-controller --patch '{"data": {"enable-opentelemetry": "true"}}'

Ensure the Ingress resource includes the nginx.ingress.kubernetes.io/enable-opentelemetry: "true" annotation, and the nginx-ingress-controller ConfigMap is configured with OpenTelemetry settings.

Sampling rate

To adjust sampling rate, set otel-sampler-ratio on the ingress config map:

kubectl patch configmap nginx-ingress-controller --patch '{"data": {"otel-sampler-ratio": "0.01"}}'

Configure Autoscaling (KEDA)

This section documents the process of installing and configuring KEDA (Kubernetes Event-driven Autoscaling) to scale the CMS pods based on traffic volume (Requests Per Second).

Install KEDA & Prometheus

Run the following snippet to install KEDA and a lightweight Prometheus instance. Prometheus is required to scrape metrics from the NGINX ingress controller.

# Add Helm repositories
helm repo add keda https://kedacore.github.io/charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install KEDA (The Autoscaling Brain)
helm install keda keda/keda --namespace keda --create-namespace

# Install Prometheus (The Metrics Collector)
# We disable heavy features (AlertManager, NodeExporter, Storage) as we only need it for scraping NGINX.
helm install prometheus-keda prometheus-community/prometheus \
  --namespace monitoring --create-namespace \
  --set alertmanager.enabled=false \
  --set kubeStateMetrics.enabled=false \
  --set prometheus-node-exporter.enabled=false \
  --set pushgateway.enabled=false \
  --set server.retention=1h \
  --set server.persistentVolume.enabled=false

Enable NGINX Metrics

The standard NGINX installation does not expose traffic metrics by default. We must upgrade the release to enable the metrics endpoint and open port 10254.

# Upgrade NGINX to enable metrics and open port 10254
# Note: Replace 'ingress-nginx' with your actual release name if different
helm upgrade --version ${NGINX_VERSION:?} ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress \
  --set controller.replicaCount=3 \
  --set controller.service.loadBalancerIP="${IP_ADDRESS:?}" \
  --set controller.service.externalTrafficPolicy=Local \
  --set controller.metrics.enabled=true \
  --set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true" \
  --set-string controller.metrics.service.annotations."prometheus\.io/port"="10254"

Deploy the scaler

Use envsubst to inject the correct namespace into the query before applying. This ensures the scaler listens to traffic for this specific environment.

# Set the namespace where your CMS is deployed (e.g., default, alpha)
export NAMESPACE=default

# Apply the configuration
envsubst < autoscale.yaml | kubectl apply -f - -n ${NAMESPACE:?}

Deploy the project into the cluster

Deploy base resources & configuration

Deploy yaml files located in /kubernetes/setup directory, and update secret containing database password. Note that PSQL_PASSWORD variable created in Create database is being used here as well as DOMAIN_NAME from Create a static IP address & domain name.

create base resources

kubectl apply -f kubernetes/setup/providers/azure.yaml -f kubernetes/setup/

Update configuration

POSTGRES_HOST=$(az postgres flexible-server show --resource-group meg --name ${PSQL_NAME:?} --query 'fullyQualifiedDomainName' -o tsv)
kubectl create secret generic database-password --from-literal="POSTGRES_PASSWORD=${PSQL_PASSWORD:?}" --dry-run=client -o yaml | kubectl apply -f -
kubectl patch configmap megforms --patch "{\"data\":{\"POSTGRES_HOST\":\"${POSTGRES_HOST:?}\", \"SITE_DOMAIN\": \"${DOMAIN_NAME:?}\", \"SAML_ENTITY_ID\": \"https://${DOMAIN_NAME:?}/\"}}"

Additional changes can be made to the ConfigMap by using kubectl edit configmap megforms

Add new deployment target to CI & Deploy

Important

Before proceeding with the following changes, you must assign the Contributor role to the Service Principal in Azure to grant GitLab the necessary permissions to make modifications in the new cluster. Run the following command:

az role assignment create --assignee 5ed4ddae-1272-478c-971e-e3a9a97716c3 --role "Contributor" --scope subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ContainerService/managedClusters/${NAME}

Add the following section to .gitlab-ci.deploy.yml:

Remember to replace CLUSTER_NAME with actual k8s cluster name

k8s:CLUSTER_NAME:deploy:production:
  extends: .k8s:deploy:production
  variables:
    cluster: CLUSTER_NAME
    DOMAIN_NAME: CLUSTER_NAME.qms.megit.com
  # Remove these after testing
  when: manual
  only: []

Commit and push this change, and trigger the job to deploy current version to the new cluster. Once deployed, you should start seeing the new pods in kubectl get pods.

Important

Remove when and only keys after initial deployment to allow the job to run only when new tags are being released.

Wait for database migrations to complete. Run kubectl.exe logs -f job/migration to observe progress.

Add new server to the Status Page

To ensure the new region is monitored on the status page, update the site list in index.js within the status-page repository.

Set up Azure Backup Media Files

The following instructions will guide you through the process of setting up a backup solution for media files stored in a persistent volume within an Azure Kubernetes Service (AKS) cluster.

Edit and use the snippets below to deploy the backup vault, create a backup policy, and register a file share with the backup vault using az-cli:

# Set variables
RESOURCE_GROUP=meg
LOCATION=westeurope
BACKUP_VAULT_NAME=west-backups
FILE_SHARE_NAME=file-share-name
STORAGE_ACCOUNT_NAME=storage-account-name
POLICY_NAME=west-daily-policy
SCHEDULE_RUN_TIME="2024-07-09T02:00:00+00:00"
RETENTION_DAYS=90

# Create a Backup Vault
az backup vault create \
    --resource-group ${RESOURCE_GROUP:?} \
    --name ${BACKUP_VAULT_NAME:?} \
    --location ${LOCATION:?}

# Create a Daily Backup Policy with 90 Days Retention
az backup policy create \
    --backup-management-type AzureStorage \
    --resource-group ${RESOURCE_GROUP:?} \
    --vault-name ${BACKUP_VAULT_NAME:?} \
    --name ${POLICY_NAME:?} \
    --policy '{
        "name": "'${POLICY_NAME:?}'",
        "properties": {
            "backupManagementType": "AzureStorage",
            "workLoadType": "AzureFileShare",
            "schedulePolicy": {
                "schedulePolicyType": "SimpleSchedulePolicy",
                "scheduleRunFrequency": "Daily",
                "scheduleRunTimes": ["'${SCHEDULE_RUN_TIME:?}'"]
            },
            "retentionPolicy": {
                "retentionPolicyType": "LongTermRetentionPolicy",
                "dailySchedule": {
                    "retentionTimes": ["'${SCHEDULE_RUN_TIME:?}'"],
                    "retentionDuration": {
                        "count": '${RETENTION_DAYS:?}',
                        "durationType": "Days"
                    }
                }
            }
        }
    }'

# Register the File Share with the Backup Vault Using the Daily Backup Policy
az backup protection enable-for-azurefileshare \
    --resource-group ${RESOURCE_GROUP:?} \
    --vault-name ${BACKUP_VAULT_NAME:?} \
    --storage-account ${STORAGE_ACCOUNT_NAME:?} \
    --azure-file-share ${FILE_SHARE_NAME:?} \
    --policy-name ${POLICY_NAME:?}

See also

To learn more about az backup, visit az-cli documentation for az backup

Site set-up

Visit the site created in Create a static IP address & domain name. Once migration is completed, you should be able to log in with credentials listed in Test user accounts.

Link to the EU site as a region 

Go to Region admin and add the new site.

Link to the EU permission groups

Go to the Region group description link admin page and add a link for each permission group you want to sync to the new server. You can then use the django admin action to sync the newly created links with the downstream server. Each time the permission group is updated in the EU server it will sync with the downstream permission groups it’s linked with. The group description slug is used as the unique identifier to match groups across regions. When a permission group is being synced from the EU site, if a matching group description slug doesn’t exist in the downstream server a new permission group be automatically be created.

Set-up observation e-mail

Task #26421

This section covers configuring a new subdomain so that MEG can send and receive observation e-mails on that server. It requires DNS changes, SendGrid configuration, a Kubernetes env var, and a Django admin step.

Examples below use sg.megit.com — replace it with the actual subdomain for the region being configured.

Note

DNS changes can take up to 24–48 hours to propagate globally, though in practice they are usually effective within minutes to an hour.

1. MX record — inbound mail routing

Add an MX record to GoDaddy DNS so that e-mails sent to @sg.megit.com are routed to SendGrid’s inbound parse service.

Type	Name	Value	Priority
MX	`sg`	`mx.sendgrid.net`	10

The Name field is the subdomain label only — enter sg, not sg.megit.com.

See also

SendGrid MX record guide

2. SPF record — outbound mail authorisation

Add a TXT record to authorise SendGrid to send e-mail on behalf of the subdomain using SPF. Without this, outgoing e-mails from MEG may be rejected or marked as spam by recipients.

Type	Name	Value
TXT	`sg`	`v=spf1 include:sendgrid.net -all`

Note

If a TXT/SPF record already exists for this subdomain, merge the include:sendgrid.net directive into the existing record rather than adding a second one. Since this is a new subdomain there should not be an existing record.

3. DKIM — SendGrid Sender Authentication

DKIM adds a cryptographic signature to outgoing e-mails so receiving servers can verify their authenticity. Together with SPF it ensures reliable e-mail delivery.

Go to SendGrid Sender Authentication and click Authenticate Your Domain.
Select GoDaddy as the DNS host and enter the subdomain (e.g. sg.megit.com).

SendGrid will generate CNAME records. Add all of them to GoDaddy DNS. They will look similar to:

Type

Name

Value

CNAME

em1234.sg.megit.com

u1234567.wl.sendgrid.net

CNAME

s1._domainkey.sg.megit.com

s1.domainkey.u1234567.wl.sendgrid.net

CNAME

s2._domainkey.sg.megit.com

s2.domainkey.u1234567.wl.sendgrid.net

The actual values are unique to the SendGrid account — use the ones SendGrid provides, not these examples.

Return to SendGrid and click Verify. Allow a few minutes for DNS to propagate if verification fails initially.

4. SendGrid Inbound Parse — webhook

This tells SendGrid to POST incoming e-mails received on the subdomain to MEG’s webhook endpoint.

Go to SendGrid Inbound Parse and click Add Host & URL.
Set Receiving Domain to the subdomain (e.g. sg.megit.com).
Set Destination URL to the server’s /emails/receive endpoint, e.g.:
```
https://audits.sg.megsupporttools.com/emails/receive
```

5. Set `INBOUND_MAIL_DOMAIN` env var

Set INBOUND_MAIL_DOMAIN to the subdomain created in step 1 in the megforms ConfigMap. See the INBOUND_MAIL_DOMAIN documentation for a full description of this variable.

6. Configure ObservationEmailConfig in Django admin

Once DNS has propagated and INBOUND_MAIL_DOMAIN is set, create the in-app configuration:

Go to django admin on the target server.
Set Sender e-mail address to an address on the configured subdomain, e.g. observations@sg.megit.com.
Set Institution to the relevant client institution.
Save — domain validation (MX + SPF checks) should now pass.

Set-up SSO with Google for meg staff

Add a new app in Google Admin in Apps ‣ Overview ‣ Web and mobile apps
Click Add app ‣ Add custom SAML app
Add new SAML identity provider django admin
Settings:
Link with existing user account: by e-mail

wantNameId: true

wantAttributeStatement: false

Client-side Entity ID: must match domain name (this is coming from env var SAML_ENTITY_ID)

New instance

Create database

Test connection

Upgrading PostgreSQL Flexible Server

Performance optimization and tweaking

Create Kubernetes cluster

Create a static IP address & domain name

Install Helm charts

NGINX Telemetry

Enabling telemetry

Sampling rate

Configure Autoscaling (KEDA)

Install KEDA & Prometheus

Enable NGINX Metrics

Deploy the scaler

Deploy the project into the cluster

Deploy base resources & configuration

Add new deployment target to CI & Deploy

Add new server to the Status Page

Set up Azure Backup Media Files

Restore Azure File Share Backup

Site set-up

Link to the EU site as a region

Link to the EU permission groups

Set-up observation e-mail

1. MX record — inbound mail routing

2. SPF record — outbound mail authorisation

3. DKIM — SendGrid Sender Authentication

4. SendGrid Inbound Parse — webhook

5. Set INBOUND_MAIL_DOMAIN env var

6. Configure ObservationEmailConfig in Django admin

Set-up SSO with Google for meg staff

Link to the EU site as a region 

5. Set `INBOUND_MAIL_DOMAIN` env var