It's been over a year since I last has a Tanuki Tuesday post. I normally look at specific features of GitLab when I do one of these - instead I want to cover some areas of using the CI/CD features for deployments via SSH which I kept having (all user error on my part, nothing bad about the product). It's something I've set up a few times in the past and therefore have some good reference tasks to work from, but I still had trouble in a few places.
In a world where software is built and deployed via containers to all kinds of clusters, it's easy to forget how to deploy via SSH in CI pipelines. GitLab have done a good job of documenting this with their docs on Using SSH keys with GitLab CI/CD, but it's still easy to overlook some finer points which come up.
SSH key from a variable
The first stage is to add a variable to the CI/CD variables. You can't mask the key as it doesn't meet the requirements to be masked. Unless you need to use the key on every branch, it's likely best to leave this to restricted branches and tags. That way, it's less likely someone will change a CI job to output the key in plain text to get access to it. Also set the type to "file".
When you run the pipeline which uses the variable, you may get the following error:
/usr/bin/bash: line 157: /root/.ssh/id_rsa: No such file or directory
If that happens, you'll have missed the part in step 3 of the GitLab guide which creates the .ssh directory. Add the following two lines to the
before_script stage, or to the script stage prior to the
cat $SSH_KEY > ~/.ssh/id_rsa part:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
With that in place and running the pipeline, you may find an error just after the
ssh-add... stage for
Error loading key "/root/.ssh/id_ionos_rsa": error in libcrypto
The most likely cause of that is the SSH key in the variable doesn't have a line break at the end. Go back to the CI variables part and check. Add a new, blank like at the end if needed.
Host key verification failed
Oh. So close, but no! You've got your key configured correctly, getting stored for the job, and being used as part of the SSH command. Now the host will not verify. Fear not, this is just another missed step which is part of the GitLab documents.
If you have a good
known_hosts set of data for the server you are deploying to, then you can use the following steps as part of your
before_script, or prior to the SSH connection:
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
There's two other ways of dealing with this (either in place of the know_hosts file above):
- Run a key-scan against each host you need to deploy to
- ssh-keyscan example.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
2. Turn of StrictHostChecking (though this isn't recommended as it may lead to a man-in-the-middle attack:
- ssh -o StrictHostKeyChecking=no "firstname.lastname@example.org"
Finally, a connection
With all of things done, the pipeline should connect to your host. You can then run your deployment task and have a pipeline that takes the repetitive deployment steps away from you. Connect it to other steps for a pipeline which will build, test, deploy, and verify the code!
It's worth noting that when I set up these deployments, I always do it from somewhere with a static IP address so SSH access is locked down to specific IPs. That may mean hosting your own GitLab Runner in your home or office (I had a Raspberry Pi running deployments from home for a while).