Connected On-Prem Backup and Restore Strategy

Updated 2 weeks ago by Michael Cretzman

Database backups are extremely important to protect Production Database data and prevent data loss. Harness On-Prem uses MongoDB and TimeScaleDB as its databases. This topic describes how to perform the necessary backups of Harness databases and ensure you are prepared for disaster recovery.

In this topic:

Harness Backup Strategy Overview

This section describes Harness' recommended backup strategy. The strategy proposed below provides two layers of protection.

Please schedule all of the following below backup methods to protect Harness On-Prem System from data loss:

  • Harness GitSync — Please use Harness GitSync to ensure all Harness entities (including configurations) are backed up incase of database loss. 
  • Full Backup Method — Daily full database backups using crontab.
Please test backup commands before using them in Production.

Full Backup Method

Harness recommends you perform full backups every 6 hours (or at a frequency you prefer) using the method given below.

Full Backup Process for 3-box Docker Connected On-Prem Setup

  1. Select the node where TimeScaleDB is running.
    1. You can verify this by running docker ps on the node and verifying that harness-timescaledb is running on the node. You can also check with Harness to verify the node.
  2. Derive the MONGO_URI from the Harness Manager by running the following command on the node: 

    docker inspect harnessManager | grep MONGO_URI
  3. Populate the value of MONGO_URI in the script below and save the script on the node as harness_dump.sh.
#!/usr/bin/env bash

set -e
set -x

MONGO_URI="<MONGO_URI>"

destination=${1:-.}
current_date=$(date "+%Y.%m.%d-%H.%M.%S")


mkdir -p $destination/$current_date/timescale
docker exec -i harness-timescaledb psql -U admin -d harness -c "\COPY (SELECT * FROM instance_stats) TO /var/lib/postgresql/data/instance_stats.csv DELIMITER ',' CSV"
docker exec -i harness-timescaledb psql -U admin -d harness -c "\COPY (SELECT * FROM deployment) TO /var/lib/postgresql/data/deployment.csv DELIMITER ',' CSV"
docker cp harness-timescaledb:/var/lib/postgresql/data/instance_stats.csv $destination/$current_date/timescale
docker cp harness-timescaledb:/var/lib/postgresql/data/deployment.csv $destination/$current_date/timescale
cd $destination/$current_date/ && tar -cvzf timescale.tar timescale && cd -
rm -rf $destination/$current_date/timescale


#Get mongoDB dump

mongo_uri=$(echo ${MONGO_URI} | sed 's/\\//g')

mkdir -p $destination/$current_date
docker exec -i mongoContainer mongodump --uri="${mongo_uri}" --out /data/db/backup/dump
docker cp -a mongoContainer:/data/db/backup/dump $destination/$current_date
cd $destination/$current_date/ && tar -cvzf mongo.tar dump && cd -
rm -rf $destination/$current_date/dump

The above script takes one optional argument: The backup folder where the files should be stored.If you do not provide the value, the default is the current directory where you are running the script.

  1. Test the script is working correctly by running the script as bash harness_dump.sh.

    If the script has worked successfully, then you should see a folder created at the current directory location with a name corresponding to the timestamp. For example: 2020.03.18-07.18.32.

    Inside the folder, there should be two TARs: mongo.tar and timescale.tar. Here is an example:
~/2020.03.18-07.18.32$ ls -l

total 76884

-rw-rw-r-- 1 demo demo 78715723 Mar 18 07:18 mongo.tar

-rw-rw-r-- 1 demo demo     5031 Mar 18 07:18 timescale.tar

  1. Set up a cron job for this script to be run every 6 hours (or at a frequency you prefer).

    Ensure you pass in the directory where the backups will be stored as an argument to the script. For example:
0 */6 * * * bash /home/harness.svc.account.user/harness_dump.sh /home/harness.svc.account.user/backup > /home/harness.svc.account.user/backup/output.log 2>&1
  1. Ensure that the backup directory has enough space for storing the backups. You can setup a cron job for deleting files older than a fixed duration to clean up old backups.

    For example, this cron job runs once a day at 1am and deletes files older than 30 days. This prevents the backup directory from filling up.
0 1 * * * find /home/harness.svc.account.user/backup/* -mtime +30 -exec rm -rf {} \; >> /home/harness.svc.account.user/backlog_folder/cleanup.log 2>&1
Important — Please ensure you perform the following:

  1. The backup folder should be copied into a redundant storage regularly. The storage should be available in case of a DC failure so it can be restored on another DC.
  2. The above script will only work if the TimeScaleDB and MongoDB containers are running on the node.

    If the containers are down, please bring them back up. This prevents the backups from failing.
  3. Please utilize the Harness Monitoring Service to ensure that the services are running on the system.

Restore Process

Perform the following steps below on the target cluster.

  1. Please copy the backup you are restoring to the target cluster node where TimeScaleDB is running.  You can check with Harness to verify the node.
  2. Ensure TimeScaleDB and MongoDB are running on the node:

    docker inspect harnessManager
    docker inspect harness-timescaledb
  3. Ensure all MongoDB nodes are running on all 3 nodes using the Harness Monitoring dashboard. 
    1. Run docker ps on each node to verify that mongoContainer is running.
  4. Derive the MONGO_URI from the Harness Manager by running the following command on the node:

    docker inspect harnessManager | grep MONGO_URI
  5. Stop the Harness Manager and Harness Verification service on all 3 nodes by running the following on all 3 nodes: 

    docker kill harnessManager verificationService
    You can run docker ps on each node to see if harnessManager and verificationService are not running.
  6. Populate the value of <MONGO_URI> in the script below and save it as harness_restore.sh:
#!/usr/bin/env bash
set -e
set -x
if [[ -z "${1}" ]]; then
echo "Target directory not specified, please pass in a target directory argument"
exit 1
fi
MONGO_URI="<MONGO_URI>"
source=$1
cd $source && tar -xvf timescale.tar && cd -
docker cp $source/timescale/instance_stats.csv harness-timescaledb:/var/lib/postgresql/data/instance_stats.csv
docker cp $source/timescale/deployment.csv harness-timescaledb:/var/lib/postgresql/data/deployment.csv
docker exec -it harness-timescaledb psql -U admin -d harness -c "DELETE FROM instance_stats;"
docker exec -it harness-timescaledb psql -U admin -d harness -c "DELETE FROM deployment;"
docker exec -it harness-timescaledb psql -U admin -d harness -c "\COPY instance_stats FROM /var/lib/postgresql/data/instance_stats.csv CSV"
docker exec -it harness-timescaledb psql -U admin -d harness -c "\COPY deployment FROM /var/lib/postgresql/data/deployment.csv CSV"
mongo_uri=$(echo ${MONGO_URI} | sed 's/\\//g')
cd $source && tar -xvf mongo.tar && cd -
echo "use harness" > drop_harness.sh
echo "db.dropDatabase();" >> drop_harness.sh
echo mongo_uri="$mongo_uri"
docker cp drop_harness.sh mongoContainer:/data/db/
docker cp $source/dump mongoContainer:/data/db/
docker exec -it mongoContainer bash -c "mongo \"${mongo_uri}\" < /data/db/drop_harness.sh"
docker exec -it mongoContainer mongorestore --uri="${mongo_uri}" --drop /data/db/dump
rm drop_harness.sh

  1. The above script takes one argument: the location of the backup.

    For example, if you copied the backup 2020.03.18-19.00.01 to the current directory, you can execute the script as bash harness_restore.sh 2020.03.18-19.00.01.
  2. Check the health of the MongoDB replicaset by running the following command. Replace <MONGO_URI> in the value below:

    docker exec -i mongoContainer mongo “<MONGO_URI>” --eval="rs.status();”
  3. Use the local start and stop scripts on the Ambassador to restore the system back to its healthy state.
  4. Reconfigure the Load Balancer to point to the nodes on the target cluster.


How did we do?