Using GitLab CI to deploy via SSH

It's been over a year since I last has a Tanuki Tuesday post. I normally look at specific features of GitLab when I do one of these - instead I want to cover some areas of using the CI/CD features for deployments via SSH which I kept having (all user error on my part, nothing bad about the product). It's something I've set up a few times in the past and therefore have some good reference tasks to work from, but I still had trouble in a few places.

In a world where software is built and deployed via containers to all kinds of clusters, it's easy to forget how to deploy via SSH in CI pipelines. GitLab have done a good job of documenting this with their docs on Using SSH keys with GitLab CI/CD, but it's still easy to overlook some finer points which come up.

SSH key from a variable

The first stage is to add a variable to the CI/CD variables. You can't mask the key as it doesn't meet the requirements to be masked. Unless you need to use the key on every branch, it's likely best to leave this to restricted branches and tags. That way, it's less likely someone will change a CI job to output the key in plain text to get access to it. Also set the type to "file".

When you run the pipeline which uses the variable, you may get the following error:

/usr/bin/bash: line 157: /root/.ssh/id_rsa: No such file or directory

If that happens, you'll have missed the part in step 3 of the GitLab guide which creates the .ssh directory. Add the following two lines to the before_script stage, or to the script stage prior to the cat $SSH_KEY > ~/.ssh/id_rsa part:

 - mkdir -p ~/.ssh
 - chmod 700 ~/.ssh

With that in place and running the pipeline, you may find an error just after the ssh-add... stage for

Error loading key "/root/.ssh/id_ionos_rsa": error in libcrypto

The most likely cause of that is the SSH key in the variable doesn't have a line break at the end. Go back to the CI variables part and check. Add a new, blank like at the end if needed.

Host key verification failed

Oh. So close, but no! You've got your key configured correctly, getting stored for the job, and being used as part of the SSH command. Now the host will not verify. Fear not, this is just another missed step which is part of the GitLab documents.

If you have a good known_hosts set of data for the server you are deploying to, then you can use the following steps as part of your before_script, or prior to the SSH connection:

- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

There's two other ways of dealing with this (either in place of the know_hosts file above):

1. Run a key-scan against each host you need to deploy to

- ssh-keyscan example.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

2. Turn of StrictHostChecking (though this isn't recommended as it may lead to a man-in-the-middle attack:

- ssh -o StrictHostKeyChecking=no "user@example.com"

Finally, a connection

With all of things done, the pipeline should connect to your host. You can then run your deployment task and have a pipeline that takes the repetitive deployment steps away from you. Connect it to other steps for a pipeline which will build, test, deploy, and verify the code!

It's worth noting that when I set up these deployments, I always do it from somewhere with a static IP address so SSH access is locked down to specific IPs. That may mean hosting your own GitLab Runner in your home or office (I had a Raspberry Pi running deployments from home for a while).