I’ve got a secret, but I won’t tell, I can't

When we were looking to add HTTPS support to SilverStripe Platform (as recently announced), we had a problem. Where do we put the HTTPS certificates and keys?

If you get hold of these certificates and keys, you can decrypt the site’s traffic, launch man-in-the-middle attacks, and generally misbehave in all the ways HTTPS is designed to prevent. So these certificates and keys are sensitive information—important secrets.

In a traditional, non-cloud based hosting provider, your web servers would be pretty stable. You might have the occasional hardware failure, but otherwise the same servers would be running the site permanently. So you could put the certificates and keys on the web server (with an encrypted backup somewhere).

But there aren’t really any permanent servers like this in SilverStripe Platform. Each EC2 instance (the name of a single virtual server in AWS terminology) has a brief, fleeting lifespan—spun up whenever load increases, and discarded just as carelessly. All persistent data needs to be stored somewhere else. If you want to store secrets, you need a place specifically to put them—a secret store.

Enter FriSP— Short for the Friedlander Secrets Pit (I initially named the tool “vault” but was rightly derided for using such a generic term). It’s a secret store, specifically designed for the needs of SilverStripe Platform.

We’ve open sourced FriSP, but before you go trusting your secrets to it, you might want to know a little bit more about its design processes and architecture.

Room of requirements

Anybody contemplating creating a secret store will come up with a similar naive design initially. Each store will have its own public/private keypair. The public key will be known to the people allowed to add secrets to the store. The private key will only be known to the web servers allowed to read secrets out of the store. By the magic of encryption, secrets encrypted with the public key can only be decrypted by using the private key; and because only the web server has the private key, only it can read those secrets.

But there’re a couple of problems with that naive design:

1. We’re back to the same problem. We still have to store something on the web server— in this case, the private key for the store.

2. In SilverStripe Platform, each website needs its own secret store. These websites are automatically generated using the platform tool. If the platform tool was the one creating the keypair, it could sneakily keep a copy of the private key and read out the secrets later.

AWS—The S stands for Sssssh

Fortunately we’re able to leverage several AWS services to solve these issues. EC2 roles, S3 with bucket policies, and Lambda.

Each EC2 instance in AWS can be assigned a “role” on creation. A role is just a name, but you can grant or deny access to various other resources based on that name.

For instance: you can grant and deny access to various areas of an S3 bucket. S3 is a distributed file storage system provided by AWS. To use S3, you create a bucket—a name for a set of files—then read and write files into that bucket. On each of these buckets, you can attach Policies that say who can access those files. We can specify that certain areas can be read by some role, say ReaderRole and written to by a different role (WriterRole—you get the idea).

So strictly we don’t need to encrypt the secrets at all. If we set up S3 with a bucket policy like the above, make sure only the web server EC2 instances are able to act as ReaderRole, then we can store secrets in that S3 bucket, and only the web servers can read them out again.

Of course there’s a catch: any service that can add bucket policies can remove them again. So if our platform tool is the one creating the bucket policies, it can remove them too, and read out the secrets. We also would prefer to encrypt the secrets as well, just in case we get our bucket policy wrong. However, nothing in EC2 roles or bucket policies allows us to prevent the platform tool from keeping a copy of any keys it creates.

Get deputised

FriSP’s solution is to have a separate service that manages sensitive operations. We’ve deputised this service as the only code allowed to read or write private keys, and to set or change bucket policies. It’s responsible for checking that the requests for new stores are valid, and from an authorised source.

The platform tool is allowed to request new secret stores through the deputy, but crucially isn’t allowed to adjust or replace the deputy, or do any of the work itself. And because everything the deputy does is private, the platform tool can’t eavesdrop and get a copy of the private key or bucket policy.

Of course, this deputy needs to be durable against outages. Also, since most of the time it won’t be doing anything (creating new secret stores is fairly rare), we didn’t want to have to have dedicated EC2 instances for it.

So we leveraged Lambda, AWS serverless code solution. By writing the deputy in Node.js, we’re able to have a highly available service in the cloud with truly elastic scaling with almost no infrastructure setup.

Behold, the pit where I store my shame

Thanks to the power of AWS services and security tools, we’ve been able to create a secret store tailor-made to SilverStripe Platform’s needs. A store that can securely and dynamically have new web servers added to it. A store that needs no extra infrastructure or ongoing maintenance. A pit, which the platform tool can throw secrets into, but from which no secret can ever escape.

As I mentioned, the code is open source, so I encourage anyone interested in the deep details to check out the code. Or leave a comment below.

Header image by Shawn Rossi.