Continuous integration is one of the (if not the most) important component of a development workflow. Why? It will save your project. Probably numerous times a day.
As more “conventional” software development paradigms have been introduced into Drupal itself (primarily due to Drupal 8’s adoption of Symfony, Composer, and the advancement of the configuration management system) the need for continuous integration is more important now for Drupal sites and projects than ever. This article will cover the ten most critical steps for your Drupal continuous integration workflow.
What is continuous integration?
“Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process is comprised of automatic tools that assert the new code’s correctness before integration. A source code version control system is the crux of the CI process. The version control system is also supplemented with other checks like automated code quality tests, syntax style review tools, and more. ”
Put another way? CI represents an automation layer during your development workflow. By automating all the things you care about (e.g. all the things that could go wrong or break something) you ensure that you catch these things before they get deployed out to a webserver somewhere and cause trouble!
I once had a customer who got very upset about CI failures. “Why?” Was my question. Sure, it’s frustrating to see a build fail. Yes, it takes time to fix a problem and re-run a build. What’s the alternative though? I mean sure, yes obviously we want to catch problems before they fail a build. But the alternative I see is a project without CI at all that is constantly breaking environments in the cloud because bad code is constantly getting deployed. Failed builds are a small price to pay.
On Drupal projects the CI process should be directly tied to any pull requests your developers open. Additional manual steps (beyond development) such as code review, quality assurance (QA), and user acceptance testing (UAT) should not proceed until the CI process completes and passes. Furthermore, code should not be merged from the feature branches the developer(s) are working in until CI completes and passes.
My team typically uses either Acquia’s own Pipelines tool or a commercial product such as Travis CI, Gitlab Pipelines, or Microsoft Azure Pipelines in combination with our project’s Git repository, which is typically Github (but could be Bitbucket or Gitlab as well). Note that some Git repositories are more tightly tied to a particular CI tool and not all CI tools support all Git offerings. The important thing is that you have a CI tool (the particulars of which one you’re using is, IMHO, less critical).
Here’s an example of how I integrate CI into the development workflow on my projects:
Each developer creates a fork of the upstream repository
Developers create feature branches for their development on their forks
The developer opens a pull request when they are ready to complete the work from their fork’s branch into the upstream integration branch
CI executes (on the pull request)
Assuming CI passes, an architect or lead developer does code review
Assuming code review passes, the code is merged into the integration branch
CI executes again (on the integration branch)
Code is deployed into an integration environment
Manual QA is done in either the integration environment or a dedicated QA environment which will be manually deployed at an appropriate time
Recommended Steps to Include in a Drupal CI Build
Each of these steps helps build, validate, and/or test the code required to stand up your Drupal site. It is highly advised that you configure your build in such a way that if any of these steps fail, then the entire build fails (so you have to remedy issues prior to getting that much coveted green checkmark).
Configure Your Container
Before anything exciting happens, the first step is to make sure that you have configured the container for your build to match (as closely as possible) your production environment. I also assume that your local environment matches (as closely as possible) your production environment. The end result should be all your environments have the same versions of key services such as PHP, MySQL, etc. and are consistent throughout (e.g. if your prod environment uses Apache, you shouldn’t have a CI environment running Nginx).
During this step you also may need to create a database in your build container (or rename one), create a database user and give that user access to the database, etc. At the conclusion of this first step, your build container should be fully configured and ready for the remainder of the build!
Note: much of the “configuration” will likely occur in a configuration file for your build. This file should be committed in the project repository to ensure that all builds from all developers are consistent.Install Dependencies
I use composer on all of my projects at this point (and have for ~4 years). If you haven’t seen it, check out my 7 Composer Hacks For 2020 and Your Day To Day Workflow post from a few weeks ago!
One critical requirement for composer (or any dependency manager) is that you have to actually install the dependencies to use them. I recommend doing this right off the bat during your builds. I would suggest running the following two commands:
$ composer validate (to ensure that your composer.json and composer.lock file are valid)
$ composer install (to install all the dependencies from your composer.lock file)
Note: in most cases, you SHOULD commit the composer.lock file in your repository. There are some specific cases where you wouldn’t, but typically that’s when you’re building a Drupal module / profile / theme (and not a full project).
Depending on your specific project needs, you may also need a front end dependency manager such as Yarn or NPM. This is also the time to install these dependencies! Similarly you should commit the package.lock file for your front end dependency manager.Check Code Quality
There’s one major reason this is step 3 and not step 2 (you need stuff that composer manages to do this one—so you have to install composer’s dependencies before you can validate the code)!
Code quality should be used for as many languages as possible that are in use on your project. Typically on a Drupal project this includes: PHP (using PHPCS), Twig, SCSS/CSS, JS, and YAML (using YAML CLI). Linting PHP, Twig, and YAML can be run without the need to include front end tools on your project. If you want to lint your CSS / JS / SCSS you’ll need to setup Gulp or Webpack to execute these scripts. There are specific packages for each to aid in this!Install Drupal
At this point, we should have all of our dependencies and be fairly certain that all of the submitted code is up to our quality expectations. While it may seem logical to start running automated tests now, this can be challenging for many functional tests (since those need a valid installation of Drupal running on a webserver before they can execute). You are welcome to run code-based unit tests at this point if that’s all you have, but I typically wait until later in the process to run “all” of my automated tests at the same time!
So, instead of tests, step 4 for me is to actually install Drupal. I typically do this via the drush site-install command. There’s really nothing fancy about it. Obviously, make sure that you specify the correct installation profile that aligns with what’s on your sites in production, but otherwise, run the drush command and let it go!
Note: I do strongly advocate for doing a “clean” setup of Drupal on each CI build vs. syncing a database in from some other environment. Automated testing does not cope well with change, so relying on a database from an environment significantly raises the likely hood that your tests will just randomly start failing someday (due to upstream database changes). On the other hand, building a clean site on every build potentially requires work to prime the site with needed taxonomy, nodes, users, etc. but in doing so you ensure that you start fresh and put what you need in the site (so that, theoretically, your tests have less chance of failing because of changes outside the build process).Activate Config Split(s)
In fairness, not everyone uses Config Splits as part of their configuration management strategy. HOWEVER if you do, make sure that your codebase has some way of detecting that you are running in a CI container and activate your CI config split (if you have one). A quick example of the php code to enable the split(s) can be seen in BLT’s config.settings.php file. Here’s another example of how to detect an environment from my plugin for BLT and Microsoft Azure Pipelines.
If you’re not using config split or don’t plan on having a CI specific split, then you can skip this step. There are some real advantages to having a CI specific split however, such as default content or other materials that aid in testing that aren’t needed on any of the “real” versions of the site.Import Configuration
Once Drupal is installed, it’s time to import configuration from the codebase (assuming you do, in fact, have some configuration management strategy). I usually do this using drush config-import.
A few things to consider here:
First, if you are using config split, you actually need to import config twice. The first time will import the default config (which includes the definitions of your config splits themselves) and the second time will important any config stored in your active split(s).
Second, config imports can fail for a wide variety of reasons. This is part of the reason this step is so critical during your CI process! A couple of common example is config for a module that isn’t enabled in core.extensions.yml. TLDR carefully read the output of your build failure, Drush should tell you why the config-import failed.Validate Configuration
In all transparency, I sometimes disable this one. But it’s still a good one to know about (and try to adhere to)!
Once you’ve imported your configuration into your site, I recommend running the drush config-status command. This will do a quick diff between the config on disk (YAML files) and the active config in the database. If you’re doing a complete config sync, it’s important to know when your database is out of sync with the file system (and this step will tell you just that). Sometimes though, it can take a few tries to capture all the config and get it back in sync. I wrote a Knowledge Base Article over on Acquia’s website talking about this in more detail if you want to dig deeper.Compile Front End
If you are using Gulp / Webpack to automate such tasks as compiling your SCSS into CSS, minimizing your CSS / JS, etc. then this is the time to do it! While I won’t go into this piece in detail, it’s an important one to consider before running your tests (especially if you are doing any automated accessibility, JavaScript, or visual regression testing that need your front end intact before running).
Run Automated Tests
Let’s be honest; most of what we’ve been doing up to this point has been to enable this step. Obviously if your dependencies don’t resolve, Drupal doesn’t install, config doesn’t import, or code doesn’t validate these are great things to know! BUT what we really want to know is: do the automated tests still pass.
Now, perhaps I am making an unfair assumption here. I’m assuming you have some automated tests to run. If you don’t, you should prioritize that. Like immediately. There are some really easy ones to incorporate into your CI process that won’t take a ton of effort and will really pay off. Here are a few examples!
Use Behat (or PHPUnit) to visit the webserver and make sure that Drupral can serve a 200 status. Here’s an example of this. Is this a fancy test? Absolutely not. Does it further confirm that all the things we’ve done up to this point worked? Absolutely!
Use Drush and composer to test for outstanding security updates for packages you’re running. Drush provides this out of the box with the pm:security command. Senseio Labs provides a security checker for Symfony composer components as well.
Use the Drupal Spec Tool to map out your content model, workflow, etc. and then literally copy out a whole bunch of Behat tests to validate that your config created these properly.
Other common testing tools include Behat (for functional testing), Pa11y (for automated accessibility testing), BackstopJS (for visual regression testing), PHPUnit (for unit / functional testing), and Nightwatch (for JS testing).
At the end of the day, the specific testing framework(s) that you are using on your project are “less important” than having testing frameworks. If you don’t have test coverage today, focus first on the features on your platform that might get you sued if they went down. Then move on to the things that would embarrass you if they went down. Work backward from there!
Also, keep in mind it’s much easier to build tests as you develop your site / platform (as opposed to going back and jacking them in well after the fact). Anything you can do to build up tests as you go is strongly advised for new projects.Build Artifact
The final step of any build (assuming we’ve made it this far) is to generate a hosting artifact that is ready for use. Why is this important? A development repository that uses dependency managers and compiles code (e.g. SCSS => CSS) requires some “work” to get everything ready for use on a webserver. This is why we had to install composer, compile the code, etc. during the build process. Typically you can’t just take the code directly out of your Git repository and “use it.”
So, the last step here should take everything that you’ve built out (other than the database), strip out anything that might identify the versions of the stuff you’re using, and push out the necessary files for running your website. If your CI tool can automatically deploy the result, so much the better (although, you should only auto-deploy AFTER you’ve merged a pull request at the conclusion of an integration build).
In conclusion
That’s it! Do these ten things on every single build and I guarantee that the chances of you deploying broken code will go down drastically (especially if you follow my advice on #9 and actually expand your automated test coverage). I’m not saying things won’t slip through—I’m saying fewer things will.
For those of you playing along at home (or for those of you that are familiar with Acquia BLT) this list shouldn’t look that surprising to you. This is a pretty standard workflow and it’s something that BLT largely automates and “just does” for you on any BLT project. So, you can certainly build this all out yourself (and I’ve seen more than one success story at this) or you can check out BLT and start a BLT project. Either way, automate all the things. And whatever you do, make sure that you use the same automation everywhere. Your developers should be running the same commands that your CI tool is running. Where possible your deployments should do the same!
Yes, Devops represents a significant investment of time and effort. But the better your Devops process (and more rigorously you adhere to a CI workflow like the one above) then the less likely you are to break your site. It’s worth it to spend the time on this. You won’t be sorry you did!
Photo by Joe deSousa from StockSnap
Related Content
An overview of all the things I’ve been trying / testing to get ready for PHP 8.1.