=============== Making releases =============== The project is hosted using kubernetes in Azure cloud. Kubernetes deployment and config files are located in ``kubernetes/`` directory. * Deployments are managed by `GitLab pipelines `_. * Currently deployed version is always rendered at the bottom of all pages Versioning ========== Each live site update should be tagged in the repository with a version number as described in `SOP 008 `_. Increment version number by adding `0.0.1` for bug-fix releases, `0.1` for new features, and `1.0` for major changes (For instance when Django or Python is being upgraded, also Bootstrap, or major site redesign can warrant major version bump). Each version must be tagged in git repository and pushed using `git push --tags`. Bug-fix releases / back-ports --------------------------------- To deploy a bug fix without deploying all changes from ``master`` branch: 1. Make a new branch off the release tag you intent to patch. For example, to fix release ``1.0`` you will create a new branch from it's latest patch version: :command:`git checkout -b release-1.0 1.0.3`, assuming that latest tagged version was ``1.0.3``. 2. Make required changes on that branch: * create a new branch for each bugfix task by branching off ``release-1.0`` * Make a new MR for each task to merge into ``release-1.0`` 3. Tag new version number (For example 1.0.1). 4. Deploy the branch to production by pushing the tag ``git push --tags`` * if a newer tag is already released, cancel the automatic release job 5. Merge the branch ``release-1.0`` to ``master`` You can continue using Building and deploying a release manually ============================================ In an event that gitlab runners are not available, it is necessary to build and deploy the docker images locally. Pre-requisites: * Docker * Access to Azure Container Registry * Azure access, or ACR key to push images * :command:`az` for logging in and retrieving ACR and AKS credentials * :command:`gcloud` for logging in and retrieving GKS credentials * :command:`kubectl` Build & push docker image -------------------------- In this step you will run commands normally handled by the ``docker:build:prod`` and ``docker:build:staticfiles`` gitlab jobs. To authenticate with Container Registry, you can use an `ACR Access Key `_ as the password, or use :command:`az` utility to get the password as in the example below (if you're logged in to az-cli). .. important:: Before building, make sure you're on master branch and have pulled all the changes you want to deploy. Make sure there are no uncommitted changes. .. code-block:: shell export VERSION=# The version goes here # use az-cli to get the ACR password, or replace with a literal export DOCKER_REGISTRY_PASSWORD=$(az acr credential show --name MegForms --query "passwords[1].value" --output tsv) export IMAGE=megforms.azurecr.io/megforms # Tag the release git tag $VERSION git push --tags # Make sure you're logged in to ACR before building so you can push the images docker login megforms.azurecr.io -u MegForms -p ${DOCKER_REGISTRY_PASSWORD} # Build & push the production docker image docker build . -t ${IMAGE}:${VERSION} docker push ${IMAGE}:${VERSION} # Build & push the staticfiles docker image docker build . --target deployment-static -t ${IMAGE}:${VERSION}-static docker push ${IMAGE}:${VERSION}-static Deploy the image to kubernetes clusters ----------------------------------------- .. important:: Before running any :command:`kubectl` commands, make sure to retrieve the context for all kubernetes clusters in Azure and Google Cloud by using the `az aks get-credentials `_ and `gcloud container clusters get-credentials `_ commands. For example: .. code-block:: shell az aks get-credentials --name production --resource-group MEG gcloud container clusters get-credentials qatar --region me-central1 --project meg-qatar-434808 gcloud container clusters get-credentials ksa --region me-central2 --project meg-ksa Deploy using shell script ^^^^^^^^^^^^^^^^^^^^^^^^^^ You can use :command:`sh/deploy-k8s.sh` script to deploy container to kubernetes. The script runs all necessary commands to deploy the docker image, run migrations, and can show output while running migrations. **Usage:** :samp:`sh/deploy-k8s.sh {version} {cluster} default --tail-migration` .. important:: When deploying to Google GKE manually using a shell script, the process must be performed separately from Azure AKS clusters, and you must export ``GCP_GKE`` env to `true`. .. code-block:: shell :caption: Example use of :command:`sh/deploy-k8s.sh` to deploy to multiple regions using a loop VERSION=# Version, or image tag if deploying untagged commit # Regions to deploy to: CLUSTERS="production uk uae au us" # Optionally add your name here to indicate who made the deployment: export GITLAB_USER_NAME= # When deploying a tag, ref name and version will be the same export CI_COMMIT_REF_NAME=${VERSION} # When deploying to GKE # export GCP_GKE=true for cluster in $CLUSTERS do sh/deploy-k8s.sh ${VERSION:?} ${cluster} default done Deploy manually ^^^^^^^^^^^^^^^^ Once images are pushed to the registry, they can be deployed to the kubernetes clusters. These steps are a simplified version of the ``.k8s:deploy`` CI job, it does not update all resources as theses are usually not modified between versions. Run the following snippet once for each region: .. code-block:: shell export VERSION=# The version goes here kubectl config use-context # name of the cluster being updated: production, uk, au, etc.. # switch context to the kubernetes cluster being deployed to # Deploy migration job to run database migrations kubectl delete job migration --ignore-not-found=true cat kubernetes/jobs/migration.yaml | envsubst | kubectl create -f - # If running on Google Cloud, use this command instead: # cat kubernetes/gke/jobs/migration.yaml | envsubst | kubectl create -f - # Deploy the images kubectl set image deployments,cronjobs *=megforms.azurecr.io/megforms:${VERSION:?} -l image=megforms kubectl set image deployments *=megforms.azurecr.io/megforms:${VERSION:?}-static -l image=mat-cms-static .. important:: After deploying the update, monitor the updated cluster to ensure the deployment was successful and the new version is live. Rolling back releases ====================== If a release introduces critical errors, it should be rolled back. Do this by re-deploying previous version using GitLab deploy jobs in the previous tag. .. important:: Not all releases can be rolled back. You should check whether new migrations were introduced between the versions: * Releases that do not add migrations can generally be rolled back safely * If release has migrations, it is not always safe to roll back - investigate whether migration is backwards compatible * Changes that add fields or make non-SQL changes (changes to ``verbose_name`` or ``choices``) are generally safe * Data migrations, removal, and renaming of database columns or tables and will cause system instability when rolled back. * It is still possible to roll back by manually un-applying the migrations using :command:`manage.py migrate` command. **It may be risky to do this!** * Alternatively database can be restored to a backup before the migration - **this will lead to loss of any data added or changed since migration was applied!** Reverting changes ====================== In an event that a single change needs to be reverted, this can be achieved by using the "Revert" option in Gitlab. .. important:: If reverted branch contains migrations, it cannot be directly reverted. To revert a change with migrations: * Revert to a new branch * Delete the migrations added by the branch * re-create migrations using ``./manage makemigrations`` * Test the revert before merging. Some migrations cannot be reverted (removals), and some need to be manually implemented (data migrations) .. warning:: it is not recommended to revert branches with migrations. Consider making a new patch release if you can quickly address the regression. Deployment environments ======================= Tagged releases are automatically deployed to all production environments/regions after tests pass. EU is the main/central environment and redirects users to their relevant region at login. It also coordinates global uniqueness of usernames. =================================================== ===================== Address Purpose =================================================== ===================== https://audits.megsupporttools.com/ EU production server (main) https://audits.uk.megsupporttools.com/ UK production server https://audits.au.megsupporttools.com/ AU production server https://audits.uae.megsupporttools.com/ UAE production server https://staging-alpha.azure.megsupporttools.com/ Staging / Alpha https://staging-beta.azure.megsupporttools.com/ Staging / Beta https://staging-gamma.azure.megsupporttools.com/ Staging / Gamma https://staging-delta.azure.megsupporttools.com/ Staging / Delta https://staging-epsilon.azure.megsupporttools.com/ Staging / Epsilon =================================================== ===================== For a more complete list of production deployments, visit :ref:`status page`. Staging sites are listed on a separate `Staging Site Index `_ page. .. seealso:: When setting up a new region, please refer to :ref:`new instance` documentation. Production ---------- * The production websites are updated automatically whenever a tag is added to the repository (provided that the tests pass). * To manage deployments, visit see the relevant `production environment `_. * Before rolling back to a previous update, make sure to un-apply any new migrations, or restore from database Staging ---------- This section has been moved to :ref:`staging_sites`.