Cloudbleed Security Incident

Earlier today, Cloudflare announced a very serious security incident relating to their service. It's been dubbed Cloudbleed, and you can read their very detailed analysis of the cause and their remediation on their site.

The short version for anybody who doesn't want to read the post is that any data that has travelled through a Cloudflare server may have been inadvertently inserted into another random HTTP response for a completely different site. And that response may have been cached by Google or any number of intermediary proxy service.

Impact for Voltos customers

We use Cloudflare as a means to provide TLS termination for all of our HTTPS requests, as well as a way to mitigate any potential DDoS attacks we may become the target of. As a result all traffic to and from the voltos.io domains passes through Cloudflare.

This means in theory any request you've made to Voltos, including setting and receiving your credentials, may have been exposed to a 3rd party. Cloudflare estimate the probability of this as being very low (it affected 0.00003% of all requests at its peak), but it still remains theoretically possible at any point between 22nd September 2016 and last week.

Remediation and moving forward

Change your password

Please change your Voltos password immediately. By now you all customers should have received an email with instructions on how to reset their password. Within the next 24hrs we will block web and CLI access for anybody who hasn't reset their password, and require you to go through the password reset process to set a new one. You can save yourself the extra email dance by doing it now.

Issue new tokens in your clients

Again we will expire all tokens in the next 24hrs. To issue new ones simply run:

Roll credentials within bundles

We strongly recommend that everyone issue new credentials for anything you'd consider sensitive, as soon as possible. While it's a low probably exposure, it's also impossible to quantify what may have been leaked and we don't feel complacency is worth the potential risk.

Feature enablement for all customers

We've already enabled full API auditing for all accounts. This will allow you to use the voltos audit command to see all API requests using your credentials. If your Voltos credentials themselves have been exposed you'll be able to identify unauthorized access.

This will not however help identify if Cloudflare has leaked other credentials elsewhere. You should still rotate your credentials (see previous point).

We've also a beta feature that will proactively notify you of credentials that need to be change, which I'd be happy to extend to anyone who needs it to address this issue. I didn't want to be presumptuous and flood inboxes though. If you'd like this enabled on your account email me personally (glenn @ voltos.io) and I'll get it turned on.

Additions to product roadmap

We're already deep into development of version 2 of the API. We've long debated the merits of both client-side encryption and zero-knowledge encryption. We deferred both in the belief that HTTPS/TLS communication covered the vast majority of the use cases. We're no longer willing to make that assumption. Version 2 of the API will be adjusted to include client-side encryption and decryption of all communications. The downside is that any supported use of the API in the future will need to be via one of the official API libraries we produce and not simply via curl.

Zero-knowledge encryption is still a longer-term objective, but is unlikely to arrive in the coming weeks.

Final words

We understand how disruptive this incident will be to you and your business. Part of the value we hope to bring is to shield you from these issues as much as possible. Where that isn't possible we hope to give you a clear way to quickly minimize your risk exposure.

There are a lot of lessons to be learned here and a number of immediate actions to be taken. If you feel we're fallen short I would really value the opportunity to talk with you further about it.

Glenn Gillen

Co-founder of Voltos. I'm also an advisor to, and investor in, early-stage tech startups such as StackShare, Stamplay, GrapheneDB, Fossa, and Polybit. Ex-Heroku, ran Heroku Add-ons & Ecosystem.