Nobody likes to lose data, hence a proper backup strategy and mechanism is good to have.
This post is part 1 of 2 on the backup strategy used. This post focusses on backing up data from (many) different hosts/applications. Part 2 will focus more on retention strategy and backing up the backups.
High level design
In this setup several hosts contain data which should be backed up. This is done on a central backup host. This backup host then creates a separate local backup as well as external backups. The mechanism is set up with flexibility in mind, adding another external site to backup to is easy.
The most high-level, simplified diagram would look something like the following. This diagram will be extended in the next paragraphs.
To explain how the entire setup works it will be split up in two, the lower part (hosts to central backup host) and the top part (inside the central backup host and external backups)
Tools used
The entire setup uses the following main tools:
- ansible - to setup and maintain the configuration
- rsync - to backup host data to the central backup host
- restic - to backup locally on the central backup host and to external locations
rsync setup
The lower part in the diagram uses rsync1 to backup files to the central backup host. rsync is used to synchronise files and directories but in a slightly more clever way than SCP2 for example. Where SCP simply copies all content, rsync will only transfer files that have changed. This means the first time it takes just as long as SCP, but thereafter it will take significantly less time if the content did not change too much.
Running rsync can be as easy as:
rsync -r <src> <dest>
OR
rsync -r <src> <user>@<host>:<path_on_host>
In this case, rsync needs to run automatically, triggered by a cronjob. This introduces some challenges regarding authentication. First of all, private/public keypairs will be used rather than password authentication. In the diagram shown above, it seems that all hosts are pushing data to the central backup host, meaning those hosts would need to authenticate towards the central backup host. This adds complexity on both the host and the central backup host and all cronjobs aren’t centrally managed/ran. Therefor the solution chosen here is to initiate connections from the central backup host.
rsync authentication
With security in mind, having one host with access to a lot of systems is a bit scary. This section outlines how authentication is setup from the central host towards the other hosts. Since it involves creating multiple unique keys (per host per app), this is automated using Ansible.
Keys
For each host and backed up application a new dedicated keypair is generated. If two hosts with one app are being backed up, and one additional host with two apps, four keys are needed:
ssh-keygen -t ed25519 -f id_ed25519_host1_app1 -c "Key to perform backup of host1 app1"
ssh-keygen -t ed25519 -f id_ed25519_host1_app2 -c "Key to perform backup of host1 app2"
ssh-keygen -t ed25519 -f id_ed25519_host2_app1 -c "Key to perform backup of host2 app1"
ssh-keygen -t ed25519 -f id_ed25519_host3_app1 -c "Key to perform backup of host3 app 1"
This ensures if one key would leak (for whatever reason), one does not get access to all systems, but only to the specific host.
Client host SSH config
On the client hosts, hosts 1, 2 and 3 in the example, the public keys for the backup host needs to be stored. First store the keys in separate files (host 1 requires the keys for app 1 and app 2).
echo $KEY1 > /root/.ssh/authorized_keys_host1_app1
echo $KEY2 > /root/.ssh/authorized_keys_host1_app2
Before going further it is good to understand that the authorized_keys usage can be limited.
Authorized keys limitation
You can limit what commands can be executed with a specific keypair, that would look something like the following:
command="<COMMAND>",<OPTIONS>,<PUBLIC_KEY>
This would translate into the following example; here we only allow this key to be used to execute the ps command. Also no PTY, agent forwarding or port forwarding is allowed using this key.
command="ps",no-pty,no-agent-forwarding,no-port-forwarding ssh-ed25519 AAAA....
You can see how this could help limiting access with the key. We know which command will be executed (rsync), so the string used is the following:
command="rsync --server --sender -vlogDtprze.iLsfxC . <BACKUP_PATH>",no-pty,no-agent-forwarding,no-port-forwarding <PUBLIC_KEY>}
Combining all keys
Now append all of these keys into one file:
cat /root/.ssh/authorized_keys_* >> /root/.ssh/authorized_keysbak
This creates a new file authorized_keysbak, which should be added to the ssh config in /etc/ssh/sshd_config. In that file, look for a line starting with AuthorizedKeysFile and append the path for the authorized_keysbak file in the same line, like this:
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys %h/.ssh/authorized_keysbak
Now SSH is configured to allow the backup host to login to the hosts. To make it more secure, the authorized_keys file can be adjust to only allow specific commands.
command="rsync --server --sender -vlogDtprze.iLsfxC . <PATH>",no-pty,no-agent-forwarding,no-port-forwarding ssh-ed25519 <KEY_DATA> Key to perform backup of host1 app1
Restic setup
The restic setup will be part 2 of this blogpost.