Integrating Composer into CI/CD Pipelines

I spend a lot of time talking to folks about Composer. I LOVE Composer and it’s an increasingly important component in the Drupal ecosystem. However, using Composer (and using it effectively) requires more than just knowledge of Composer. Why? Let’s get into it.

Managing Dependencies with Composer

A decade ago I was using Git, but I wasn’t using a dependency manager. That means that every project repository contained ALL the stuff I needed for Drupal, its modules, etc. Of course, in Drupal 6 and 7 I didn’t have much of a choice since there wasn’t a dependency manager.

Fast forward to today and I’m entirely managing my dependencies on my projects with Composer. BUT the dependencies managed by composer are not in my project git repositories. If you check out https://github.com/Drupal4Gov/Drupal-GovCon-2017 there is no vendor directory at all in the repo (but I have one locally):

Repo:

Drupal GovCon codebase in Github (without Composer) https://github.com/Drupal4Gov/Drupal-GovCon-2017

Local:

Drupal GovCon codebase in PHPStorm (after composer install)

I’ll talk a bit more about how this works practically and how we set it up in the next section of the article.

But first, to give you some perspective, the Govcon project is a Drupal 9 project. The docroot/core directory (which contains all of the Drupal 9 files) contains something like 16,000 files!

$ find . -type f -print | wc -l

16140

The contributed modules directory in docroot/modules/contrib contains an additional roughly 9,000 files.

$ find . -type f -print | wc -l

8949

And of course, looking at the vendor directory (which houses any non-Drupal projects) is an additional roughly 11,000 files.

$ find . -type f -print | wc -l

11373

For those of you counting along at home, that’s nearly 35k files! If I were to commit all of the files required for Drupal (so, docroot/core, the contrib modules, profiles, and themes (if any), vendor, etc.) that would add all 35,000 files to my Github repository. That means anytime I need to do an update to <whatever> then my Git will track those changes.

You might be thinking to yourself… well don’t we want Git to track changes? Yes, we do! But we don’t necessarily need to track them at that granularity.

With composer anytime you update a dependency we see that dependency change in the composer files. We can see the version change. The dependency has a Git repository… somewhere and you should then be able to go see what changed down in the depths of that dependency’s files on their repo. Also tracking the dependency’s file changes in your own repo is redundant at that point!

So, adding tens of thousands of extra files to your repo:

• is redundant

• makes it harder to track the history of your project (due to the sheer volume of files)

• slows down actions in the repo (given the increased size and quantity of files)

Hopefully then, you’re coming to the conclusion that if you’re using Composer, you don’t want to put the dependencies in your Git repo. Good, great. There’s a problem though: to actually host a Drupal website so it serves websites to the internet you need all of that stuff there. Obviously, it’s there in my local! But if it’s not in the Github repository… how do we get it to the webserver for hosting?

The Role of Continuous Integration when using Composer

This is where continuous integration comes into play. Obviously there are a lot of benefits of using CI on the project but integration with a dependency manager is a big one.

If you have a look at the GovCon CI file (for Acquia Pipelines) one of the very first things we do is run composer install:

https://github.com/Drupal4Gov/Drupal-GovCon-2017/blob/develop/acquia-pipelines.yml#L25

You might also spot that we’re doing an npm install as well (because NPM is also a dependency manager, just for front end dependencies.)

Let’s quickly examine the configuration up to this point:

• The project relies on composer

• The gitignore is configured to exclude all of the dependencies managed by composer

• Continuous integration executes and installs our dependencies so they are present in the CI container

At this point, the CI process can run “all the things” and install Drupal, run tests, etc. with all of your files in place. Good!

The next problem we face is ensuring that the files you need (those managed by composer) land in your hosting environment.

Hosting Websites from Composer Projects

Remember all that stuff I said earlier about not wanting an extra 35,000 files in your Github repository? When you’re actually hosting a website, you “need” all that stuff (or a lot of it, anyway). So we now face our final challenge: how do we get all that stuff we “need” without cluttering up our development repository?

Thankfully, continuous integration comes to the rescue (again). The goal during the CI step is to transform a composer-managed developer repository into an artifact that can be used to host your website. This is, by the way, exactly the same thing you do locally when you run composer install and we just talked about during the CI process when composer install runs. The difference is that we need to automate this and ensure all the right files get bundled up and transported to the hosting environment.

Acquia’s Build and Launch Tools (BLT) is a tool that aids in this, if you use it. The blt deploy command essentially handles it. But, even if you aren’t running BLT, the TLDR is:

• hosting a website requires all of the composer dependencies to be present on disk

• consider sanitizing the artifact to remove things (like change logs and read me files) that could identify versions.

In Conclusion

So often when we talk about using a dependency manager as a best practice we overlook all of the pesky details of the intervening tasks required to effectively implement the strategy. Continuous integration can be incredibly challenging to adopt and integrate into your workflow, but the benefits far outweigh the costs. If you aren’t currently using continuous integration, I would strongly recommend you spend some time investigating and trying it out. Getting it in place should allow you to do a lot of new, automated things (like integrating composer).

Good luck!

Related Content