How to configure your CI pipeline environments
This seems like a very simple and trivial thing, but it's something which gets easily overlooked. Setting up a CI pipeline environment is fairly trivial (unless you are using a custom docker image - but that's for another time); just tell the script to use a specific docker image, and the scripts to run. However, there's different ways of configuring environments to achieve the same results.
Consider a simple website/web application running on Linux. I'll assume it's running PHP, and requires some NodeJS elements to allow task runners such as Gulp to perform actions. There's nothing in that setup described so far which tells us which web server the application is on, nor how PHP is to be configured. Do we have an Apache web server, is it Nginx, or something else we haven't fully thought of? With PHP, is this the built in module, or will it run as FPM? Is the local environment configured for that?
Just those things alone lead to some combinations which are possible, probable, and likely. However, we will really only be running one of those in production (albeit on several servers). And though we've covered the application elements, what of the actual OS itself? With the myriad Linux distributions available unless developers are all developing on the same distro, there could be issues fairly easily.
Thinking in simple terms, the difference of users from Debian based Linux distributions to CentOS, and that family. For Apache web-server alone the user and groups Apache run as are different, so these may need configuring to be consistent, especially when factoring in file uploads and other aspects of an application which may need to access the file system.
Things also come unstuck somewhat if you only run certain parts of jobs for deployment, how will you know they whether they work until the deployment fails?
The lesson here is that your CI/CD environments should be configured the same as your production environments (run multiple pipelines if there's different environments to consider). You should also test your deployment tasks for each environment, too!