Release Cadence: Keeping the Wheels Rolling

One of the most common problems I come across with customers has to do with their release cadence. This may not sound like a problem… but bear with me. It really is.

Imagine this scenario: development ends and a website launches. The development team scales down in size (perhaps because a partner was brought in to help with the big development push). As a result, the number of folks working to maintain the platform shrinks. The amount of work that can be done, also necessarily shrinks. Due to the smaller dev capacity, the team gets focused on the highest priority requests from their stakeholders. This may be bugs, it may be features. But the end result is the same: a lot of technical debt and maintenance slips through the cracks.

Now, fast forward a year or two into the future. The technical debt has continued to build. Something significant happens (maybe a security update, or a major end of life event like Drupal 8 going end of life this year). What should be a totally normal, incremental security update now becomes an “oh crap we have a year or more of updates to do” if we want to address this security update! Worse, dealing with something like an end of life event becomes even more challenging because there are potentially years of history blocking the update.

So, how do you deal with this problem? Not everyone can afford a bigger team. And, in fairness, that’s not always the answer. Let’s dig into what your team can (and should) do to make sure that you aren’t so focused on features and bugs that you miss out on all the other important stuff.

Planning for Releases

In general, there are two types of releases:

Security: these releases should take priority over everything else
Feature: these releases may include new functionality, bug fixes, etc.

The frequency of security updates will change depending on the underlying technology you are using. For instance, the Drupal security team has a weekly security release window on Wednesdays. Does this mean that they will drop a critical security update for Drupal itself every week? I mean, probably not (but they could). The point is, they could. So, how should you handle that?

You should plan for them to drop a critical security update every week.
You should plan to do a production deployment (of the security updates) every week.
You should have a plan to do “other” feature deployments at other times outside of the security releases.

Best case scenario, there’s no security updates and you can skip a release. Worst cast scenario, there are security updates but you’ve already got time planned for that release and it won’t impact your existing feature schedule.

This is one of the fatal flaws I see time and again. There is no release plan to handle security updates. So there is no capacity to handle security updates. So there has to be an approval to drop feature work to handle security updates and a lot of organizations don’t give that approval. So the security work falls farther and farther behind. So the work to do security becomes larger and more complex with each new release, and it gets kicked further down the road because it will take more and more time away from features.

Release Governance

A lot of this comes down to the governance within an organization. Specifically, what are we prioritizing, what are we tasking our team to focus on, etc. This is part of the reason I said earlier that just getting a bigger team doesn’t necessarily “help.” You could have a really big team, but still have terrible governance model (meaning the release cadence might still be off kilter).

Therefore, a critical first step is helping your stakeholders understand how important security (and other) releases become to the health and maintenance of a platform.

Ideally, you should have a plan sort of like this:

Our work is done in <x increments> (e.g. 2 weeks). We will deploy new feature work every two weeks (a couple of days after we complete a sprint). This means we will do roughly 2 production deployments for features each month.
We anticipate weekly security releases (which will be applied directly into the most recently committed tag / branch / etc.). This means we will do a maximum of 4 production deployments for features each month.

Note: you could potentially combine security and feature releases if the timing lines up. However, this does add to your testing load between the time the security update comes out and the release goes.

Your initial reaction to the above might well be… “six deployments a month?!” And I tend to agree. That’s a lot. So maybe back up and think about how you can make that plan work for your organization. And remember, that you may not actually have weekly security deployments (you’re just planning on them).

The takeaway here should be that if you don’t have the capacity on your team (due to team size and/or workload) to maintain your application (which means doing the necessary updates to stay current and deployments to stay secure) then that’s a problem. You may need to grow your team, you may need to change the workflow and/or work load of that team, etc.

I’m not going to lie… this may require some uncomfortable conversations with your stakeholders. One of the hardest thing about my job as an architect is telling people that they can’t have something they want. Or that they have to pick and choose between things that they want. But at the end of the day, many stakeholders make requests / set priorities in a vacuum that focuses on what they want to see happen. They may not be making an informed decision about the impacts of letting a site get out of date.

You don’t have to result to scare tactics, but you should help them make informed decisions. Help them understand that not applying security updates could actually result in the entire platform going down (or perhaps worse, being compromised). I’ve seen hacked websites before, and the amount of spam, porn, and other defaced content can be quite damaging to a brand. It’s absolutely critical (regardless of if you’re using Drupal, Wordpress, or anything else) that you stay up to date!

In Conclusion

Most updates are incremental and small. But, as with nearly any chore, if you let a lot of small, incremental, simple tasks build up you end up with something big and unmanageable. Imagine if you let your dirty dishes pile up in your kitchen for two years without ever once doing dishes. That would be a real big mess. Doing security updates should be a regular part of an applications lifecycle. Deploying those updates should be a regular part of your team’s business on a monthly basis. If they aren’t right now, you aren’t alone! But identifying a problem is the first step to fixing it. And this is on that I strongly urge that you prioritize a fix.

Seriously, these updates are not that bad if you stay on top of them. Please do!

Photo by Kristin Hardwick from StockSnap