.. _onprem:

=========
On-prem
=========

On premises deployment does not take advantage of Kubernetes. It can be hosted in a VM on a single machine

.. uml:: ../../../onprem.puml
    :caption: On-prem architecture diagram

.. glossary::

    VM
        Virtual Machine; a software emulating a computer inside another computer (host).
        It can run a different operating system than host.

    provisioning
        the process of installing software and configuring a :term:`VM`. In :program:`Vagrant`, provisioning is the first
        step after the VM is created using :command:`vagrant up`, but can be ran again if provisioning steps
        have changed to apply them using :command:`vagrant up --provision`, or :command:`vagrant provision`.

    vagrant box
    box
        a :term:`provisioned <provisioning>` and packaged :term:`vm` that can be transferred to another computer and ran.

    On-prem
    Onprem
        On-premises; as opposed to "in cloud" describes a setup where :term:`MEG QMS` is installed on client's own network.

    CA
        Certificate Authority

    CSR
        Certificate signing request; a template file used to create a certificate by :term:`CA` based off a trusted root certificate.

Vagrant box
--------------

The vagrant box contains the backend and client app with all components required to run it. It requires database on host machine to connect to.
These configuration files are located in :file:`/onprem/box` and are used to build the Box that can be deployed to client premises.

.. important:: Vagrant box definition is located in :file:`/onprem/box`. All commands in this section need to be ran in that directory.

:file:`Vagrantfile`
    `Vagrant <https://www.vagrantup.com/>`_ file containing VM definition and provisioning instructions.
    It is used to build a VM box that contains the MEG application and all the necessary files to run it.

:file:`docker-compose.yml`
    the compose file describing and configuring all components of meg.

:file:`env`
    Base environment variables for the :term:`QMS` and the :term:`client app`. It provides default values that can be overridden by `deployed instance <Deployment>`_.

:file:`megit.conf`
    Nginx server configuration to proxy incoming connections to the relevant docker containers.
    This file is symlinked to nginx, so any changes to it will affect :program:`nginx` server.
    Any changes to this file should be followed by :command:`sudo nginx -s reload` to reload :program:`nginx`.

.. uml:: vagrant-box.puml
    :caption: Vagrant box contents

Required environment variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following variables are required to build the Vagrant VM. Version number will be stored in the VM,
but :envvar:`ACR_KEY` is only required for :term:`provisioning`.

.. envvar:: ACR_KEY

    Valid Access key to `Azure Container Registry <https://portal.azure.com/#@megsupporttools.com/resource/subscriptions/0f14d0e0-e21b-4d40-912a-3111c3a445e2/resourceGroups/MEG/providers/Microsoft.ContainerRegistry/registries/MegForms/accessKey>`_.
    Required to pull docker images for the :term:`QMS` and its components.

.. envvar:: VERSION

    :term:`MEG QMS` version to be packaged in the :term:`box`.

.. envvar:: APP_VERSION

    MEG :term:`client app` version to be packaged in the :term:`box`.

.. envvar:: VM_NAME

    Name of the vagrant VM. Override the name if you need to make it unique, for instance when making concurrent builds.

    :default: megit-base

Run the :term:`VM`
^^^^^^^^^^^^^^^^^^^^

Vagrant VM can be built and ran locally for testing. Before running, you need to ensure that :program:`postgres` is running and accepting connections.
Edit :file:`env` with the correct :envvar:`POSTGRES_HOST` value pointing at your host machine.

.. code-block:: shell
    :caption: Run the :program:`vagrant` commands in :file:`onprem/box` directory

    vagrant up
    vagrant ssh -c "docker-compose up -d"

When the VM is running, the site is being served on port 80 (and 443 for HTTPS) on your localhost, so you can access the back-end easily via `localhost <http://localhost/>`_.
The app is only served at `app.localhost <http://app.localhost/>`_.
The mirth connect web view is served at `web.mirth.localhost <http://web.mirth.localhost/>`_.
The mirth connect admin endpoint is served at `admin.mirth.localhost <http://admin.mirth.localhost/>`_. There's no web view for this endpoint.

.. note:: You can edit your `hosts file <https://www.nublue.co.uk/guides/edit-hosts-file/>`_ to replicate domains on
    client's network. For example, append the following lines to :file:`Windows/System32/Drivers/etc/hosts`::

        127.0.0.1 eqms.kfmmcmed.net
        127.0.0.1 cqieqms.kfmmcmed.net
        127.0.0.1 web.mirth.kfmmcmed.net
        127.0.0.1 admin.mirth.kfmmcmed.net

When finished, you can delete the VM using :command:`vagrant destroy`, or just shut it down using :command:`vagrant halt`.
To reload any changes made locally to config files, run :command:`vagrant reload`.

.. _build vagrant box:

Build and package the :term:`Box`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To build a Vagrant :term:`box`, you need to :term:`provision <provisioning>` and package it. It is a good idea to delete any existing VM using :command:`vagrant destroy`
to ensure that the box is built on a clean box.

.. important:: The :envvar:`VERSION` variable must be a valid release version of the project already deployed to Azure Container Registry.
    Only invoke this command on the commit that has been tagged. If you're only testing the box, you can hardcode any version you want.
    Any changes to this value require that vagrant VM be re-provisioned.


.. important:: Ensure that the web app for this :envvar:`APP_VERSION` is built by triggering ``docker:build:web``
    job in the client app's `tag pipeline <https://gitlab.com/MedicalEguides/qms/meg-qms-client/-/pipelines?page=1&scope=tags>`_.
    This builds the ``${VERSION}-web`` image necessary to deploy the app as a docker container.


.. code-block:: shell
    :caption: Run the :program:`vagrant` commands in :file:`onprem/box` directory
    :emphasize-lines: 8-11

    # Version number of MEG back-end
    export VERSION=$(git describe --tags --abbrev=8)
    # Version number of MEG client app - use the latest stable release
    export APP_VERSION="3.45"
    # Password for Azure Docker Registry (you can extract it manually if you're logged in to Azure CLI)
    export ACR_KEY=$(az acr credential show --name MegForms --query "passwords[1].value" --output tsv)

    vagrant box update
    vagrant up
    vagrant halt
    vagrant package --output=meg-qms-v${VERSION}.box

.. note:: While running `vagrant up` you may run into the following error::

        A VirtualBox machine with the name 'megit-base' already exists.
        Please use another name or delete the machine with the existing
        name, and try again.

    To fix this, open virtualbox and remove the vm with that name, deleting all files.


The packaged :file:`.box` should be stored on a secure drive and transferred to the server.

.. _creating csr:

Generate a signed SSL certificate
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use key stored in :file:`onprem/box/ssl/megit.key.insecure` to generate a :term:`CSR`.

.. code-block:: powershell
    :caption: Run these commands in :file:`onprem/box/`. Replace domain name "localhost", "app.localhost", "web.mirth.localhost" and "admin.mirth.localhost" with the actual domain names

    openssl req -new -key ssl/megit.key.insecure -out megit.csr \
      -subj "/C=IE/ST=Co. Dublin/L=Dublin/O=MEG/OU=QMS/CN=localhost" \
      -addext "subjectAltName=DNS:localhost,DNS:app.localhost,DNS:web.mirth.localhost,DNS:admin.mirth.localhost"

Send the generated :file:`megit.csr` to :term:`CA` for signing. Once you have the signed certificate,
you can follow :ref:`these steps <add certificate>` to add it to the web server.


    .. code-block::
        :caption: You can use :program:`openssl` to verify the received certificate. Replace ``-inform der`` with "pem", or remove it altogether if certificate is not in "der" format.

        openssl x509 -text -inform der < ssl/megit.crt

A valid certificate will contain extensions listing all required subdomain names::

    X509v3 extensions:
            X509v3 Subject Key Identifier:
                35:A1:7D:6B:77:BA:17:29:43:18:11:FA:B3:89:EA:2F:10:74:56:7D
            X509v3 Authority Key Identifier:
                35:A1:7D:6B:77:BA:17:29:43:18:11:FA:B3:89:EA:2F:10:74:56:7D
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:app.localhost, DNS:web.mirth.localhost, DNS:admin.mirth.localhost

.. _troubleshooting:

Troubleshooting
^^^^^^^^^^^^^^^^^

.. _self-check:

Self-check
~~~~~~~~~~~~

The :command:`self-check` command checks various components of the project to see if they are running and available.
It is implemented in :file:`self-check.sh` script.

.. code-block:: powershell
    :caption: You can run self-check over ssh from Windows

    vagrant ssh -c self-check

.. note:: the :command:`self-check` may return false negatives, for example report directory as non-writeable while it is writeable.
    Verify any reported issues manually

    For example, create a file using :command:`touch media/test.txt` to verify that media folder can be written to.

HTTPS
~~~~~~
When testing HTTPS connections, :command:`self-check` uses ``localhost`` as the hostname. If certificate is bound to a different domain name,
the check will fail. Use :program:`curl` to verify the problem.

.. code-block:: shell
    :caption: :program:`curl` will give more detailed information about why

    export CURL_CA_BUNDLE=/etc/ssl/certs/megit.crt
    curl --head https://localhost/
    curl --head https://app.localhost/
    curl --head https://web.mirth.localhost/
    curl --head https://admin.mirth.localhost/

The output will tell you whether the issue is with untrusted (self-signed) certificate, which indicates that certificate
served by :program:`nginx` is different than :const:`CURL_CA_BUNDLE`, or the hostname does not match.

Commands
~~~~~~~~~~~~
When working with the Vagrant box, the following commands are essential::

    # Start Vagrant box
    vagrant up
    # Reboot running vagrant VM
    vagrant reload
    # Connect to the running vagrant VM
    vagrant ssh
    # Provision VM again (install/upgrade all software, including MEG and the client app)
    vagrant up --provision
    vagrant provision
    # Delete the VM to start from scratch
    vagrant destroy

    # Reload nginx configuration
    sudo nginx -s reload
    # Run the project
    docker-compose up -d

The box also contains utility packages:

    :program:`net-tools`
        Network utilities for diagnosing connectivity and issues such as :command:`ifconfig` and :command:`netstat`.

    :program:`postgres-client`
        Command-line PostgreSQL client utility; can be used to test database connectivity and access.
        Use :command:`psql` to connect the database and issue commands. For example:

        .. code-block:: shell

            psql -h $POSTGRES_HOST -U megforms megforms

Logs
~~~~~~

nginx logs
    The log files for nginx are located in :file:`/var/log/nginx`.

application logs
    MEG logs are stored in :file:`/home/vagrant/logs` directory. It should be mounted onto host machine.
    To follow docker logs, run :command:`docker-compose logs -f` inside Vagrant VM.

    .. seealso:: :ref:`onprem logs`

Migrations
~~~~~~~~~~~~

Migrations should run automatically when VM starts up. They are necessary for the site to function correctly.
They are only necessary when upgrading the project.

To view applied migrations::

    vagrant ssh -c "docker-compose run cms ./manage.py showmigrations"

To manually trigger migration service::

    vagrant ssh -c "docker-compose up database-migrate"

To view last migration logs::

    vagrant ssh -c "docker-compose logs -f database-migrate"

Common Errors
~~~~~~~~~~~~~~

VM isn't running
    Sometimes the :term:`VM` will be running in virtualbox, but vagrant think's the :term:`VM` is down:

    .. code-block:: shell

        $ vagrant ssh
        VM must be running to open SSH connection. Run `vagrant up`
        to start the virtual machine.

    To fix this, open virtualbox and stop the :term:`VM`. You should then be able to start the :term:`VM` using vagrant.

Virtualbox conflict
    Sometimes when starting the :term:`VM` via vagrant there will be a conflict with virtualbox:

    .. code-block:: shell

        $ vagrant up
        A VirtualBox machine with the name 'megit-base' already exists.
        Please use another name or delete the machine with the existing
        name, and try again.

    To fix this, open virtualbox and remove the :term:`VM` with that name, deleting all files. Then rebuild the :term:`VM` using vagrant.

Deployment
-------------

These files are deployed directly to client and can be modified to customized individual deployment.

.. important:: Deployment files are located in :file:`/onprem/deployment`. All :program:`vagrant` commands in this section need to be ran
    in the same folder as the deployment :file:`Vagrantfile`.

:file:`Vagrantfile`
    A Vagrant file based off the `box <Vagrant box files>`_
    and configures the specific VM instance such as memory and CPU allowance, directory paths etc.

:file:`env`
    Environment variables for MEG. To be copied during initial deployment and updated as-needed.
    Changes made to this file require :command:`vagrant reload`.

New Deployment
^^^^^^^^^^^^^^^^^^

.. note:: When setting up a new server, document any relevant usernames, passwords, and paths.

.. important:: These steps assume that you have already built and packaged the vagrant box

#. Ensure VT-x is enabled in BIOS for optimal performance
#. Install and configure software

    #. Install `VirtualBox <https://www.virtualbox.org/wiki/Downloads>`_.

        Requires `VC++ 2019 Redistributable <https://learn.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-170>`_

    #. Install `PostgreSQL <https://www.postgresql.org/download/windows/>`_

        * Add :file:`C:\\Program Files\\PostgreSQL\\12\\bin` to ``PATH`` in `environment variables <https://docs.oracle.com/en/database/oracle/machine-learning/oml4r/1.5.1/oread/creating-and-modifying-environment-variables-on-windows.html>`_.
        * Modify :file:`C:\\Program Files\\PostgreSQL\\12\\data\\postgresql.conf` to accept connections from inside VM::

            listen_addresses = '*'

        * Modify :file:`C:\\Program Files\\PostgreSQL\\12\\data\\pg_hba.conf`. Add the following line at the end of the file::

            host    all             all             0.0.0.0/0     md5

        * Restart :program:`postgresql` service
        * ensure that :program:`postgresql` is allowed connections from Public networks in Windows Firewall
        * Create database and user account for QMS::

            psql -U postgres -c "CREATE USER megforms WITH PASSWORD 'password'; ALTER USER megforms CREATEDB;"
            createdb -U postgres -O "megforms" --encoding=UTF8 "megforms"

        * Create database and user account for mirth connect::

            psql -U postgres -c "CREATE USER mirth WITH PASSWORD 'password'; ALTER USER mirth CREATEDB;"
            createdb -U postgres -O "mirth" --encoding=UTF8 "mirth"

        .. important:: Replace ``'password'`` with a secure password

    #. Install Vagrant

#. Deploy the Vagrant Box onto Windows Server.

    #. Make the necessary changes to the config files

        * Update :envvar:`POSTGRES_HOST` and :envvar:`POSTGRES_PASSWORD` in :file:`env` according to host's address and password set in the previous step
        * Set adequate CPU and memory settings in :file:`Vagrantfile` (e.g. ``vm.cpus`` set to maximum available cores, ``vm.memory`` set to half available memory)
        * Update folder mappings in :file:`Vagrantfile`

    #. Run the vagrant box

        .. code-block:: powershell
            :caption: Add base box and start the VM. ``{version}`` represents the version number of MEG back-end being deployed.

            vagrant box add meg-qms-{version}.box --name meg-qms-{version}
            vagrant up

        .. important:: After box is added, ensure that ``config.vm.box`` in :file:`Vagrantfile` matches the newly added box name (meg-qms-{version})

        .. note:: Append version number to ``vm.name`` in :file:`Vagrantfile` - this is necessary to make the vm name unique in the future when new update is deployed.
                It can be changed at any time the VM is deployed by restarting the VM using :command:`vagrant reload`.

    #. Ensure that :term:`migrations` have ran to completion

        .. code-block:: powershell
            :caption: Run this command to watch database migration progress

            vagrant ssh -c "docker-compose logs -f database-migrate"

    #. Create :term:`superuser` account

        .. code-block:: powershell
            :caption: this command can be invoked directly from Windows to create the initial superuser account that can be used to access :term:`django admin`.

            vagrant ssh -c "docker-compose run cms python manage.py createsuperuser"

#. Set-up VM to start automatically when system boots up

    #. Open :program:`Task Scheduler` in Windows
    #. Select "Create Basic Task"

        Enter the following details:

            Name
                "megit"

            When do you wan the task to start?
                "When the computer starts"

            Action
                "Start a program"

            Program
                "vagrant"

            Arguments
                "up"

            Start in:
                Paste the path to the folder where :file:`Vagrantfile` is located

    #. Edit the schedule and update it with the following

        General tab
            Select **"Run whether user is logged on or not"**

            **When running the task, use the following user account:** select the same user account as used to run the VM manually

            .. figure:: img/schedule.png

                Example schedule configuration dialog showing correct configuration for **General** tab

        Settings
            Tick "If the task fails, restart every 1 minute"

    .. seealso:: Supporting article: https://superuser.com/questions/1027410/how-do-you-configure-virtualbox-vm-to-autostart-after-windows-10-reboot

#. Replace SSL certificate with a signed one

    .. _add certificate:

    .. code-block:: ruby
        :caption: Add or un-comment the following line in the deployment :file:`Vagrantfile` to add the certificate to VM.

        config.vm.provision "file", source: "megit.crt", destination: "/home/vagrant/ssl/megit.crt", run: "always"

        # You will also need to reload nginx configuration to use the new cert
        config.vm.provision "shell", run: "always", inline: <<-SHELL
            nginx -s reload
        SHELL

    .. important:: This is assuming you have used the key provided in :file:`onprem/box/ssl/`. If for some reason you
        used a different key, you need to add that key to the VM as well

        .. code-block:: ruby

            config.vm.provision "file", source: "megit.key", destination: "/home/vagrant/ssl/megit.key", run: "always"

        If :program:`nginx` refuses to use the key due to file permissions, add :command:`chmod` and :command:`chown`
        commands to set owner and permissions as a part of provisioning step in :file:`Vagrantfile`.

    .. seealso:: Follow steps in :ref:`creating csr` :term:`CSR` and get a signed certificate

#. Log in to :term:`django admin` and update the :url:`Site </meg-admin/sites/site/1/change/>` object with correct domain name.

.. _onprem upgrade:

Updating on-prem installation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Deploy new update
~~~~~~~~~~~~~~~~~~
.. important:: Before update, take note of any relevant :ref:`release notes <release-notes>` and apply the necessary steps before/after update.

#. Transfer the box built in :ref:`build vagrant box` to the server
#. Use :command:`vagrant box add` to add the box on the server (Replace ``{version}`` with the actual version number being imported):

    .. code-block:: powershell
        :caption: import box into vagrant. Give the new box a unique name so it does not override previous version.

        vagrant box add meg-qms-{version}.box --name meg-qms-{version}

#. Take :ref:`backup <Onprem Backups>` of the data using :command:`backup-db` and :command:`backup-media` commands inside the VM

    .. code-block:: powershell
        :caption: Use :command:`vagrant ssh` to run commands inside the VM

        vagrant ssh -c "backup-db"
        vagrant ssh -c "backup-media"

    .. important:: Verify that backup was created and accessible on host machine. The new backup file size should be similar to other backups.

#. Update :file:`Vagrantfile`: change ``config.vm.box`` (line 4) to the newly imported box name (e.g. ``meg-qms-1.0``)

    .. code-block:: ruby
        :caption: Locations in :file:`Vagrantfile` to be updated
        :emphasize-lines: 4, 7
        :linenos:

        VAGRANTFILE_API_VERSION = "2"

        Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
            config.vm.box = "meg-qms-1.0"

            config.vm.provider "virtualbox" do |vm|
                vm.name = "megit-qms"
                # ...
              end

            # ...
        end

#. Remove the current version of MEG QMS

    .. code-block:: shell

        vagrant destroy

#. Start up the VM

    .. code-block:: powershell

        vagrant up

    .. note:: The console output should tell you that the new box is being imported::

        ==> default: Importing base box 'meg-qms-v1.0'...

#. Verify that site is working correctly

    * site is available at http://localhost/ and correct version is displayed in the footer
    * :term:`migrations` have ran successfully. Use the following command to view migration logs::

            vagrant ssh -c "docker-compose logs -f database-migrate"

    The log should not contain errors.

    Alternatively, run :command:`showmigrations` to view a list of migrations and ensure all boxes are ticked::

            vagrant ssh -c "docker-compose run cms ./manage.py showmigrations"

    * client app is available and version number is correct
    * Use :ref:`self-check` utility to quickly scan the VM

    Refer to :ref:`troubleshooting` section if anything is not working

#. After successful upgrade, old boxes can be removed.

    .. note:: This is an optional step and its only purpose is to conserve disk space.

    List all available boxes, and delete old versions, but keep 2 most recent versions

    .. code-block:: powershell

        vagrant box list
        vagrant box remove meg-qms-1.0

#. Copy :ref:`error logs <onprem logs>` onto a portable drive to investigate any crashes later.

Rolling-back update
~~~~~~~~~~~~~~~~~~~~

If updated version of box will not boot for some reason, you can revert to the previous version of the vm.

#. Revert box version in :file:`Vagrantfile` to the previous version
#. Run :command:`vagrant destroy`
#. Run :command:`vagrant up`
#. Restore backup using instructions from :ref:`Onprem Backups restore`

    .. note:: This step is only necessary if the new box booted up and started migrations

Maintenance
^^^^^^^^^^^^^

.. _onprem logs:

Application logs
~~~~~~~~~~~~~~~~~~
The application logs are stored in :file:`~/logs`. This directory should be mounted onto a drive shared with host machine.

:file:`~/logs/errors` contains error logs while remaining folder contain application logs by date.

Performance Monitoring
~~~~~~~~~~~~~~~~~~~~~~~~

The on-prem deployment can log performance data to an SQLite database. When enabled, it will write per-process
performance insights into :file:`~/logs/traces/` directory. To enable, set :envvar:`TRACE_SQLITE_EXPORTER` to a truthy
value, such as ``True``.

.. note:: Performance tracing itself may affect performance, so disable it when not needed, or set it to a low sampling rate when used over a long period. Use high sample rate (1.0) only when actively troubleshooting performance.

The trace files can be easily collected, sent, and opened using any SQLite utility, or PyCharm.

Recommended setup::

    # Enable tracing to SQLite db file
    TRACE_SQLITE_EXPORTER=True
    # Trace only 1% of requests
    INSIGHTS_SAMPLE_RATE=0.01
    # Don't trace every single SQL query
    INSIGHTS_PSQL_INTEGRATION=False
    # leave blank to create a separate database for each process
    TRACE_SQLITE_FILENAME=


.. seealso::
    Alternatively, to view data instantly in logs, use log exporter by setting :envvar:`TRACE_LOGGING_EXPORTER` to ``True``.
    This will print out tracing data into console in real time.

Backups & restoration
~~~~~~~~~~~~~~~~~~~~~~~~
Backups are stored in :file:`/home/vagrant/backups`. Nightly backups are scheduled using :program:`crontab`.
The CRON file :file:`onprem/box/database-utils/cron` is installed in `/etc/cron.d/backup` when box is built.

.. _Onprem Backups:

.. important:: The :file:`/home/vagrant/backups` folder must be mounted on the host machine so that backups persist when VM is re-built.

.. rubric:: Backup commands:

These commands can be ran inside the :term:`VM`. Use :command:`vagrant ssh` to shell into the VM.

:command:`backup-db`
    Dumps postgres database contents of the database :envvar:`POSTGRES_DB` at :envvar:`POSTGRES_HOST` into :file:`~/backups/db` using :command:`pg_dump`.
    The files are dated and have the following format: :file:`meg-backup-2023-04-24_132601.dump`.

:command:`backup-media`
    Synchronizes media files using :program:`rsync` into :file:`~/backups/media`.
    Unlike database backups, existing files are overwritten, as django will typically create a new filename every time a new file is uploaded.

:command:`restore-db [db-name] [file-path]`
    Loads data from backup located in ``[file-path]`` to a new database named ``[db-name]``.

:command:`backup-mirth`
    Dumps postgres database contents of the database :envvar:`MIRTH_DATABASE_NAME` at :envvar:`POSTGRES_HOST` into :file:`~/backups/mirth` using :command:`pg_dump`.
    The files are dated and have the following format: :file:`mirth-backup-2023-06-15_130813.dump`.

:command:`restore-mirth [db-name] [file-path]`
    Loads data from backup located in ``[file-path]`` to a new database named ``[db-name]``.
    Uses mirth connect specific credentials as defined in :envvar:`MIRTH_DATABASE_USERNAME` and :envvar:`MIRTH_DATABASE_PASSWORD`.


Rescheduling backups
#########################

Backups are scheduled to run at a specific time at night. To modify the schedule, you need to update the deployment :file:`Vagrantfile`
to write new cron file to `/etc/cron-d/backup` during provisioning. The file should be owned by ``root`` and have ``755`` permissions.

.. _Onprem Backups restore:

Restoring from backup
#######################

Use :command:`restore-db` and `restore-mirth` commands to restore the MEG and Mirth Connect databases from backup to a new database

.. code-block:: shell
    :caption: Example usage of :command:`restore-db` command

    restore-db megforms-restored backups/db/meg-backup-2023-04-24_132601.dump

.. code-block:: shell
    :caption: Example usage of :command:`restore-mirth` command

    restore-mirth mirth-restored /home/vagrant/backups/mirth/mirth-backup-2023-06-15_130813.dump

After the databases are restored, exit the VM, update the :envvar:`POSTGRES_DB` and :envvar:`MIRTH_DATABASE_NAME` variables in :file:`env`, and reload the VM using :command:`vagrant reload` to apply the change to the web server.

There's no dedicated command to restore media files, but you can just use :command:`cp` to restore them by copying from :file:`~/backups/media/` back to `~/media/`

.. code-block:: shell

    cp -r backups/media/* media/

.. seealso::
    Alternatively, :command:`pg_restore` command can be used directly load data into existing database.
    **This method will overwrite data in the existing database and can lead to data loss.**

    .. code-block:: shell
        :caption: Restoring database using :command:`pg_restore`

        pg_restore --dbname $POSTGRES_DB backups/db/meg-backup-2023-04-24_132601.dump

SSL Certificate Renewal
~~~~~~~~~~~~~~~~~~~~~~~~~~

Certificate can be re-created on-prem without updating the base :term:`vagrant box`.
To do this, :ref:`generate a CSR <creating csr>`, and :ref:`replace the SSL certificate <add certificate>`.

Rolling back changes
^^^^^^^^^^^^^^^^^^^^^^
In an event of unsuccessful upgrade, it is necessary to roll back the upgrade.
Rollback steps are similar to :ref:`upgrade <onprem upgrade>`.

.. warning:: These steps involve restoring database from backup.
    This may result in loss of any data stored after the last backup.

#. Change ``config.vm.box`` at the top of :file:`Vagrantfile` back to the previous version number
#. Rebuild the VM by destroying and re-creating it

    .. code-block:: powershell

        vagrant destroy
        vagrant up

#. ssh into the VM to restore the database

    This step is necessary to undo any applied :term:`migrations`.

    .. code-block:: shell
        :caption: Use :command:`vagrant ssh` to access the vm and run the following commands, then :command:`exit` the VM.
            The ``megforms-restored`` is a name for a new database, and the file name should point at the backup created during upgrade.

        restore-db megforms-restored backups/db/meg-backup-2023-04-24_132601.dump
        cp -r backups/media/* media/

    .. seealso:: :ref:`Onprem Backups restore`

#. Update/add :envvar:`POSTGRES_DB` in the :file:`env` file with database name used in the previous step
#. Restart the VM to apply change using :command:`vagrant reload`
#. Carry out the same verification steps as during :ref:`onprem upgrade`.

Mirth Connect
--------------

A mirth connect container is deployed as part of the on prem docker deployment.
It's configured to use the same postgres server as the QMS application.
Two services are exposed:
* `admin.mirth.localhost` the endpoint used to actually configure the channels
* `web.mirth.localhost` a web viewer of the configured channels in the mirth connect database

.. _Onprem SMS:

SMS
-----

When installing onprem the client may wish to send SMS messages, however in offline environments they can't leverage Twilio.
In offline environments Twilio should be disabled by setting :envvar:`TWILIO_ENABLED` to ``False``.
In this scenario :term:`MEG QMS` has the ability to store the messages to the database. They are stored in the ``sms_sms`` table.
As the client owns the database, they can implement their own application to query the database and extract messages that need to be sent.
Below are the steps required to implement this solution:

#. Enable storage of SMS messages in the :file:`env` by setting :envvar:`STORE_SMS` to ``True``.
#. Create a user in the database with access to only the SMS table.

    .. code-block:: shell

        psql -d megforms -U postgres -c "CREATE USER sms_admin WITH PASSWORD 'password' LOGIN; GRANT CONNECT ON DATABASE megforms TO sms_admin; REVOKE ALL ON SCHEMA public FROM sms_admin; GRANT USAGE ON SCHEMA public TO sms_admin; GRANT SELECT, UPDATE ON public.sms_sms TO sms_admin;"

    .. important:: Replace ``'password'`` with a secure password. Ensure the arg values for ``-d`` and ``-U`` are correct for the onprem installation you're working with.

#. To read ``pending`` messages from the SMS table, use the below query

    .. code-block:: postgresql

        SELECT * FROM sms_sms WHERE status = 'pending';

#. To update the table and mark messages as sent, use the below query

    .. code-block:: postgresql

        UPDATE sms_sms SET status = 'sent' WHERE id in (1, 2, 3);

    .. important:: Replace ``1, 2, 3`` with a list of the id's of the messages you'd like to mark as ``sent``.

#. To update the table and mark messages as failed, use the below query

    .. code-block:: postgresql

        UPDATE sms_sms SET status = 'failed' WHERE id in (1, 2, 3);

    .. important:: Replace ``1, 2, 3`` with a list of the id's of the messages you'd like to mark as ``failed``.