Road Rules: Why we choose process over pressure

Our CTO Hamish Friedlander shares an insight into our internal deployment guidelines and how they've shaped product development for SilverStripe Platform... 

Rules, in general, are annoying. They limit us and constrain us. They do not listen to context or reason or excuse. Remembering them adds cognitive load. Forgetting them leads to surprises and missed deadlines when we run up against them.

Worse, they often seem to do all this needlessly.

Take, for example, the road rules. At most intersections, most of the time, we could simply ignore the give way rules and everything would work out fine. Most of the time ignoring the speed limit wouldn’t cause any problems. Most of the time if our horn didn’t quite work, if our lights didn’t quite turn on, as long as only drove during the day, what harm? If we know the roads, if everything goes as we expect, and if there are no surprises, we don’t need rules at all.

But surprises occur, and not everything goes as we expect. We share the roads with other motorists, and cyclists, and pedestrians, and not all of them are equally skilled at navigation. The roads themselve change - roadworks, potholes and obstructions may have appeared since we last travelled this way. Surprises are rare (that’s why they’re surprising), but without rules, when they did occur they would spell disaster. Without rules, surprises are accidents.

We need rules. We need them to remind us that not everything is predictable. They’re not there to limit us, but rather, to keep us (and others) safe.

When working on a website, the same is true. When you are working on a server by yourself, when everything is as you expect, when there are no surprises, rules seem unnecessary. Rules are not agile, and they are not lean. Nonetheless, surprises do occur.

Some of the disasters I have seen because people did not have (or did not follow) rules:

• A code change was released directly to production from a developer’s machine that immediately brought the site down. The developer did not notice and went home. No one else knew what had been deployed.
• A site needed an urgent change, but the only developer who knew how to deploy to production was away on holiday and uncontactable, leading to a massive delay getting the fix out.
• A hotfix for a critical performance issue that took multiple days to develop was made directly to the live site. Next time a regular deployment occurred, this hotfix was overwritten, immediately taking the site down.
• A support staff member logged into a live server to perform diagnostics, and accidentally deleted the database
• Non-standard changes were made to a website’s infrastructure. When the server hosting the website’s infrastructure died, no one knew what those changes were.

These were all excellent developers, working diligently, paying attention. They knew that things can and do go wrong, for whatever reason -- client or timeline pressure, their own internal drive, or just simple overconfidence -- they forgot this momentarily, or just thought “it won’t happen this time.”

All it would take to protect against this is a few simple guidelines saying “these things aren’t optional - do them every time, no exceptions”. These guidelines may include:

• Code must be stored in a git repository so that everyone can see what changes have been made
• Deployments must be easy -- single button easy -- and reliable.
• Deployments are always logged, and deployments to production have to have the approval of a senior team member, to ensure that everyone is aware of what has been deployed and when
• Deployments to production have to be a version of code that has previously successfully been deployed to UAT, to minimise the chance that an environment-specific issue occurs
• No changes are ever allowed to be made directly on the server

So that’s what we do. These are all standard rules at SilverStripe. They’re not in place to limit us, but rather, to keep us safe. They protect us when emergencies, clients, or our own sense of purpose and drive pressure us to do things we know are risky. We think they’re so useful, and have saved us so often, that when we built SilverStripe Platform, we encoded these rules right into the way it’s designed.