Earlier today, Cloudflare announced a very serious security incident relating to their service. It's been dubbed Cloudbleed, and you can read their very detailed analysis of the cause and their remediation on their site.
The short version for anybody who doesn't want to read the post is that any data that has travelled through a Cloudflare server may have been inadvertently inserted into another random HTTP response for a completely different site. And that response may have been cached by Google or any number of intermediary proxy service.
Impact for Voltos customers
We use Cloudflare as a means to provide TLS termination for all of our HTTPS requests, as well as a way to mitigate any potential DDoS attacks we may become the target of. As a result all traffic to and from the voltos.io domains passes through Cloudflare.
This means in theory any request you've made to Voltos, including setting and receiving your credentials, may have been exposed to a 3rd party. Cloudflare estimate the probability of this as being very low (it affected 0.00003% of all requests at its peak), but it still remains theoretically possible at any point between 22nd September 2016 and last week.
Remediation and moving forward
Change your password
Please change your Voltos password immediately. By now you all customers should have received an email with instructions on how to reset their password. Within the next 24hrs we will block web and CLI access for anybody who hasn't reset their password, and require you to go through the password reset process to set a new one. You can save yourself the extra email dance by doing it now.
Issue new tokens in your clients
Again we will expire all tokens in the next 24hrs. To issue new ones simply run:
voltos auth: which will log you back in on the CLI and give you access to account-wide commands again.
voltos use [bundle]: which will issue you a new development token to locally access credentials within a bundle.
Roll credentials within bundles
We strongly recommend that everyone issue new credentials for anything you'd consider sensitive, as soon as possible. While it's a low probably exposure, it's also impossible to quantify what may have been leaked and we don't feel complacency is worth the potential risk.
Feature enablement for all customers
We've already enabled full API auditing for all accounts. This will allow you to use the
voltos audit command to see all API requests using your credentials. If your Voltos
credentials themselves have been exposed you'll be able to identify unauthorized access.
This will not however help identify if Cloudflare has leaked other credentials elsewhere. You should still rotate your credentials (see previous point).
We've also a beta feature that will proactively notify you of credentials that need to be change, which I'd be happy to extend to anyone who needs it to address this issue. I didn't want to be presumptuous and flood inboxes though. If you'd like this enabled on your account email me personally (glenn @ voltos.io) and I'll get it turned on.
Additions to product roadmap
We're already deep into development of version 2 of the API. We've long debated the merits of
both client-side encryption and zero-knowledge encryption. We deferred both in the belief that
HTTPS/TLS communication covered the vast majority of the use cases. We're no longer willing
to make that assumption. Version 2 of the API will be adjusted to include client-side
encryption and decryption of all communications. The downside is that any supported use of
the API in the future will need to be via one of the official API libraries we produce and
not simply via
Zero-knowledge encryption is still a longer-term objective, but is unlikely to arrive in the coming weeks.
We understand how disruptive this incident will be to you and your business. Part of the value we hope to bring is to shield you from these issues as much as possible. Where that isn't possible we hope to give you a clear way to quickly minimize your risk exposure.
There are a lot of lessons to be learned here and a number of immediate actions to be taken. If you feel we're fallen short I would really value the opportunity to talk with you further about it.