How we use Continuous Integration at SilverStripe

SilverStripe is a modular content management system and framework, with a plethora of additional modules which can be plugged in easily with Composer. At SilverStripe we maintain around 140 of the some 2,400 total SilverStripe modules and themes available for public use, and with this number of individual modules comes the need for consistency using automated tools.

What is Continuous Integration?

CI (continuous integration) is a term coined by Grady Booch in the early 90s for "a development practice that requires developers to integrate code into a shared repository several times a day." The key part of this however is that "each check-in is then verified by an automated build, allowing teams to detect problems early."

In my team at SilverStripe - the Creative Commoners - we'll often contribute to a variety of different open source repositories throughout a given day, but since there's so many of them we'll potentially only hit each repository once a month.

Having automated checks in place is an invaluable failsafe that helps us save time while making these contributions, as we can tell immediately if an existing automated test covering a certain piece of functionality starts to break with a change we've made. This in turn allows us to keep the stability of our software as high as possible throughout a development cycle.

The contribution life cycle

At SilverStripe we contribute to the SilverStripe open source ecosystem in the same way as the community does. Typically this involves checking out a project with Git, making a change, reviewing and committing that change, making a pull request, then finally having that pull request merged and released so everyone can benefit from it.

Continuous Integration kicks in at the stage where you make a pull request to a repository on GitHub. Here's what that looks like:

In the obviously-not-copied-from-an-Agile-workflow-diagram iteration loop above you see the three primary CI services that we use at SilverStripe:

• Travis CI: runs our automated test suites - PHPUnit, jest, Behat, PHP CodeSniffer
• Scrutinizer CI: runs static code analysis to detect bugs, standards violations and security vulnerabilities
• Codecov: consolidates and processes code coverage reports into beautiful graphs

It's a circle because the process will continue until the automated checks pass (or someone is happy with the changes anyway). If a contribution you've made passes all three of these checks comfortably enough, you'll see something like this on your GitHub pull request:

Now any of the module maintainers will see that your change is at the very least not breaking any existing functionality that is covered by existing tests across multiple PHP and SilverStripe versions and database drivers. They could also see that the changes you made have new or existing tests that cover them, and that you haven't introduced any major security vulnerabilities - ideal!

Of course, this isn't everything - but it gives a pretty healthy baseline to any contribution, and the more tests a repository has, the more this toolset will help going forward.

Gaming the system

For giggles, let's try and game this a bit. If we dig into the code coverage reports for the SilverStripe versioned module we can find a part of the code that isn't already covered by a test. Here's a relatively mundane example:

The red indicates that a particular line hasn't been executed while running the PHPUnit test suite.

Theoretically, we could change this line to return 'monkey'; and people might see "monkey" on their screen a few times when viewing history of an unsaved object in the CMS. Sure - pointless example, but in this case the pull request wouldn't fail its tests - it may however fail its code coverage check because we didn't add a new test covering this line, so the patch coverage would be 0%.

A change like this would also require the module maintainers to merge your pull request with their eyes closed, or while asleep. It's fun to use for an example though.

How you can benefit from CI

Scrutinizer CI is the easiest CI system to implement, since all you need to do is copy and paste your repository URL into their system and it'll automatically configure all the webhooks from GitHub to ensure pull requests trigger static analysis runs. You can then add a badge to your readme file to indicate how awesome your code is.

Travis CI can be implemented fairly painlessly into any SilverStripe project or module. If you're using SilverStripe 3 then you can use the silverstripe-travis-support module, or if using SilverStripe 4 you can more or less copy a simple Travis configuration file from another repository that already has one. Adding it to your project is similar to in Scrutinizer - plug and play. The only requirement here is that you've got some PHPUnit tests to run.

If you're using a project it'll be the same configuration, only with a couple of paths tweaked.

"What about private repositories though?" I hear you say. Well, while most CI providers are free for open source projects, they'll generally all cost a bit for private ones. Travis CI has paid plans for private repositories. You could also use an alternative CI provider like CircleCI which offers a modest quota of free builds for private repositories.

The idea here is that Travis CI (or whatever provider you choose) will install your code in a real-life scenario then run your tests for you.

Lastly - configuring Codecov. This is possibly the easiest of them all (assuming you already have functioning Travis CI or CircleCI unit test builds). All you need to do is add a code coverage run to one (or all) of your build matrix steps and add their one-line code snippet to post the report off to them at the end. They'll automatically create your repository account and you can access it with your GitHub account permissions.

Cool things you can do now

Ok, so you've got everything set up. Now you want to do something extra cool with the data to soak up all that free time that you've got.

Here at SilverStripe, our Lead Solution Architect Ingo Schommer whipped up a contributors dashboard which shows who the most active contributors are to our software, as well as some statistics around open issues and pull requests, and the CI build status of various branches on some of our more important repositories. This is cool, because we can see where builds are broken across multiple repositories:

You could also hook it up to a Nerf gun to shoot anyone who breaks a build, introduce a Siren of Shame into your team's workspace or some cheeky build lights for your laptops.

These providers all have APIs you can consume, so the world is your oyster. We'll also be using these metrics to help us to rate SilverStripe modules for our addons site.

Do you want to fire foam darts at your teammates when they push up syntax errors? Personally I'd recommend going for a beer if you've collectively improved your code coverage level by 20% in a week, but each to their own. Jokes aside, we should all encourage participation as much as possible rather than crucify people for doing something wrong. 

Adding continuous integration to your projects or modules is totally worth the effort. It'll help you to maintain them, help your boss to know that your software maintains stability, and help people looking for new, stable modules to add to their project.

If you would like to learn more about our development practices here at SilverStripe and think we could help with your project, please send us a message!