As part of our commitment to security and support, we periodically upgrade the stack image, so that we can install updated package versions, address security vulnerabilities, and add new packages to the stack. Recently we had an incident during which some applications running on the Cedar-14 stack image experienced higher than normal rates of segmentation faults and other “hard” crashes for about five hours. Our engineers tracked down the cause of the error to corrupted dyno filesystems caused by a failed stack upgrade. The sequence of events leading up to this failure, and the technical details of the failure, are unique, and worth exploring.

Background

Heroku runs application...

On Friday January 18, security researcher Benjamin Manns notified Heroku of a security vulnerability related to our add-ons program. At a high level, the vulnerability could have resulted in disclosing our Cross-Site Request Forgery tokens (these tokens are used to prevent browser hijacking) to third parties.

We quickly addressed the vulnerability and on Sunday, we deployed a patch to remediate the issue. We also reviewed our code for related vulnerabilities and conducted a review of our audit logs to determine the impact of the vulnerability. We found no instances of this issue being exploited.

We wish to thank Mr. Manns for his work and commitment to responsible disclosure. You can...

A serious security vulnerability has been found in the Ruby on Rails framework. This exploit affects nearly all applications running Rails and a patch has been made available.

Rails developers can get a full list of all your affected Heroku applications by following instructions here. Please address this security vulnerability by immediately upgrading your affected apps to any of the safe versions of Rails listed below. The following Rails versions have been patched and deemed safe from this exploit:

• 3.2.11
• 3.1.10
• 3.0.19
• 2.3.15

If you do not upgrade, an attacker can trivially gain access to your application, its data, and run arbitrary code or commands. Heroku recommends upgrading to a...

Heroku recently learned of and resolved a security vulnerability. We want to report this to you, describe how we responded to the incident, and reiterate our commitment to constantly improving the security and integrity of your data and source code.

On December 19, 2012, security researcher Stephen Sclafani notified us of an issue in our account creation system. Using a maliciously-crafted HTTP request, an attacker could change the password of a pre-existing Heroku user account, and thus gain control of it. This attack would not disclose the pre-existing password to the attacker (those are stored internally as non-recoverable bcrypt hashes).

Upon receiving notification, our engineering...