Repository navigation
Backups
This is a collection of notes about production backups.
We assume daily backups. They can be stored in /gemstorage/backups or somewhere (doesn't matter) and we can format the path with something like this. For sanity the directories need to be namespaced by date, but for additional sanity the backup filenames also need to be; and also need the release version encoded.
~$ RELEASE=20180718-2f8dfb75
~$ BACKUP_PATH=`date -u +%Y/%m/%d/%Y-%m-%d-$RELEASE.tgz`
~$ dirname $BACKUP_PATH
2018/07/20
~$ echo $BACKUP_PATH
2018/07/20/2018-07-20-20180718-2f8dfb75.tgzThe application container can include the cron job that triggers backups, which will let us avoid having to maintain a custom Postgres image.
So we need to add a crontab file to the mappings (like we did with the git history) and then we need to update the package stuff to install the crontab and turn on the cron service.
COPY crontab /etc/cron.d/cool-task -- do this via mappings
RUN chmod 0644 /etc/cron.d/cool-task
RUN service cron start
When we deploy a new version we have to shut down the old version, but we don't want this to happen if it's in the middle of a backup. Note the following:
-
docker stopissues aSIGTERMto the entry point process, and if it doesn't terminate within 10 seconds it follows up withSIGKILL. You can control this with--time <seconds>. - When Postgres gets a
SIGTERMit starts an orderly shutdown, which means it will stop accepting new connections, and will wait for all open connections to close before shutting down.
So this means we just need to ensure that we're doing docker stop --time <seconds> with a sufficiently large max to handle backups. Because this value will grow with time we'll need to keep an eye on it.