Making releases

The project is hosted using kubernetes in Azure cloud. Kubernetes deployment and config files are located in kubernetes/ directory.

Deployments are managed by GitLab pipelines.
Currently deployed version is always rendered at the bottom of all pages

Versioning

Each live site update should be tagged in the repository with a version number as described in SOP 008. Increment version number by adding 0.0.1 for bug-fix releases, 0.1 for new features, and 1.0 for major changes (For instance when Django or Python is being upgraded, also Bootstrap, or major site redesign can warrant major version bump). Each version must be tagged in git repository and pushed using git push –tags.

Bug-fix releases / back-ports

To deploy a bug fix without deploying all changes from master branch:

Make a new branch off the release tag you intent to patch. For example, to fix release 1.0 you will create a new branch from it’s latest patch version: git checkout -b release-1.0 1.0.3, assuming that latest tagged version was 1.0.3.
Make required changes on that branch:
- create a new branch for each bugfix task by branching off release-1.0
- Make a new MR for each task to merge into release-1.0
Tag new version number (For example 1.0.1).
Deploy the branch to production by pushing the tag git push --tags
- if a newer tag is already released, cancel the automatic release job
Merge the branch release-1.0 to master

You can continue using

Building and deploying a release manually

In an event that gitlab runners are not available, it is necessary to build and deploy the docker images locally.

Pre-requisites:

Docker
Access to Azure Container Registry
Azure access, or ACR key to push images
az for logging in and retrieving ACR and AKS credentials
gcloud for logging in and retrieving GKS credentials
kubectl

Build & push docker image

In this step you will run commands normally handled by the docker:build:prod and docker:build:staticfiles gitlab jobs.

To authenticate with Container Registry, you can use an ACR Access Key as the password, or use az utility to get the password as in the example below (if you’re logged in to az-cli).

Important

Before building, make sure you’re on master branch and have pulled all the changes you want to deploy. Make sure there are no uncommitted changes.

export VERSION=# The version goes here
# use az-cli to get the ACR password, or replace with a literal
export DOCKER_REGISTRY_PASSWORD=$(az acr credential show --name MegForms --query "passwords[1].value" --output tsv)
export IMAGE=megforms.azurecr.io/megforms

# Tag the release
git tag $VERSION
git push --tags

# Make sure you're logged in to ACR before building so you can push the images
docker login megforms.azurecr.io -u MegForms -p ${DOCKER_REGISTRY_PASSWORD}

# Build & push the production docker image
docker build . -t ${IMAGE}:${VERSION}
docker push ${IMAGE}:${VERSION}

# Build & push the staticfiles docker image
docker build . --target deployment-static -t ${IMAGE}:${VERSION}-static
docker push ${IMAGE}:${VERSION}-static

Deploy the image to kubernetes clusters

Important

Before running any kubectl commands, make sure to retrieve the context for all kubernetes clusters in Azure and Google Cloud by using the az aks get-credentials and gcloud container clusters get-credentials commands.

For example:

az aks get-credentials --name production --resource-group MEG

gcloud container clusters get-credentials qatar --region me-central1 --project meg-qatar-434808
gcloud container clusters get-credentials ksa --region me-central2 --project meg-ksa

Deploy using shell script

You can use sh/deploy-k8s.sh script to deploy container to kubernetes. The script runs all necessary commands to deploy the docker image, run migrations, and can show output while running migrations.

Usage: sh/deploy-k8s.sh version cluster default --tail-migration

Important

When deploying to Google GKE manually using a shell script, the process must be performed separately from Azure AKS clusters, and you must export GCP_GKE env to true.

Example use of sh/deploy-k8s.sh to deploy to multiple regions using a loop

VERSION=# Version, or image tag if deploying untagged commit

# Regions to deploy to:
CLUSTERS="production uk uae au us"
# Optionally add your name here to indicate who made the deployment:
export GITLAB_USER_NAME=
# When deploying a tag, ref name and version will be the same
export CI_COMMIT_REF_NAME=${VERSION}
# When deploying to GKE
# export GCP_GKE=true

for cluster in $CLUSTERS
do
    sh/deploy-k8s.sh ${VERSION:?} ${cluster} default
done

Deploy manually

Once images are pushed to the registry, they can be deployed to the kubernetes clusters. These steps are a simplified version of the .k8s:deploy CI job, it does not update all resources as theses are usually not modified between versions.

Run the following snippet once for each region:

export VERSION=# The version goes here
kubectl config use-context # name of the cluster being updated: production, uk, au, etc..
# switch context to the kubernetes cluster being deployed to

# Deploy migration job to run database migrations
kubectl delete job migration --ignore-not-found=true
cat kubernetes/jobs/migration.yaml | envsubst | kubectl create -f -
# If running on Google Cloud, use this command instead:
# cat kubernetes/gke/jobs/migration.yaml | envsubst | kubectl create -f -

# Deploy the images
kubectl set image deployments,cronjobs *=megforms.azurecr.io/megforms:${VERSION:?} -l image=megforms
kubectl set image deployments *=megforms.azurecr.io/megforms:${VERSION:?}-static -l image=mat-cms-static

Important

After deploying the update, monitor the updated cluster to ensure the deployment was successful and the new version is live.

Rolling back releases

If a release introduces critical errors, it should be rolled back. Do this by re-deploying previous version using GitLab deploy jobs in the previous tag.

Important

Not all releases can be rolled back. You should check whether new migrations were introduced between the versions:

Releases that do not add migrations can generally be rolled back safely
If release has migrations, it is not always safe to roll back - investigate whether migration is backwards compatible
- Changes that add fields or make non-SQL changes (changes to verbose_name or choices) are generally safe
- Data migrations, removal, and renaming of database columns or tables and will cause system instability when rolled back.
  It is still possible to roll back by manually un-applying the migrations using manage.py migrate command. It may be risky to do this!
  
  Alternatively database can be restored to a backup before the migration - this will lead to loss of any data added or changed since migration was applied!

Reverting changes

In an event that a single change needs to be reverted, this can be achieved by using the “Revert” option in Gitlab.

Important

If reverted branch contains migrations, it cannot be directly reverted.

To revert a change with migrations:

Revert to a new branch

Delete the migrations added by the branch

re-create migrations using ./manage makemigrations

Test the revert before merging. Some migrations cannot be reverted (removals), and some need to be manually implemented (data migrations)

Warning

it is not recommended to revert branches with migrations. Consider making a new patch release if you can quickly address the regression.

Deployment environments

Tagged releases are automatically deployed to all production environments/regions after tests pass. EU is the main/central environment and redirects users to their relevant region at login. It also coordinates global uniqueness of usernames.

Address	Purpose
https://audits.megsupporttools.com/	EU production server (main)
https://audits.uk.megsupporttools.com/	UK production server
https://audits.au.megsupporttools.com/	AU production server
https://audits.uae.megsupporttools.com/	UAE production server
https://staging-alpha.azure.megsupporttools.com/	Staging / Alpha
https://staging-beta.azure.megsupporttools.com/	Staging / Beta
https://staging-gamma.azure.megsupporttools.com/	Staging / Gamma
https://staging-delta.azure.megsupporttools.com/	Staging / Delta
https://staging-epsilon.azure.megsupporttools.com/	Staging / Epsilon

For a more complete list of production deployments, visit Status page.

Staging sites are listed on a separate Staging Site Index page.

Production

The production websites are updated automatically whenever a tag is added to the repository (provided that the tests pass).
To manage deployments, visit see the relevant production environment.
Before rolling back to a previous update, make sure to un-apply any new migrations, or restore from database

Staging

This section has been moved to Staging Sites.