There have been thousands of reported security vulnerabilities in 2013 alone, often with language that leaves it unclear if you're affected. Heroku's job is to ensure you can focus on building your functionality, as part of that we take responsibility for the security of your app as much as we're able. On Friday, November 22nd a security vulnerability was disclosed in Ruby (MRI): CVE-2013-4164 . Our team moved quickly to identify the risk to anyone using the Heroku platform and push out a fix.
The vulnerability
The disclosed Ruby vulnerability contains a denial-of-service vector with the possibility of arbitrary code execution as it involves a heap overflow. In a denial-of-service attack, a malicious individual sends your app requests that either causes the system to lock up or become unresponsive. When multiple people or systems do this, it becomes a distributed denial-of-service attack (or DDoS).
A denial-of-service attack can vary in damage depending on how crucial uptime is to your app. For example if your app brings in a significant amount of money every hour, an attacker could cost you a large sum by bringing your service down.
In this case, the denial-of-service was particularly easy to execute, making this a potentially devastating attack to some users. In addition, because this attack triggers a heap overflow, there is also a slim theoretical possibility of a much more serious vulnerability, an arbitrary code execution. We could not rule out the possibility of arbitrary code execution even though there is no known way to achieve it through this vulnerability.
The fix
When the vulnerability was announced, patched versions of Ruby 1.9.3, 2.0.0, and 2.1.0 were made available from Ruby core. Heroku’s Ruby Task Force pulled in the latest versions, compiled them and after running tests to confirm their compatibility with the platform released updated versions of 1.9.3, 2.0.0, and 2.1.0. Any new pushes to the platform receive these patched versions.
Some have asked why we didn’t automatically force-update all Ruby apps. First, Heroku will not re-deploy a user’s app without their direct knowledge or action. Additionally, since some upstream dependencies may have changed since an app’s last deploy, we also want to maintain erosion resistance by only changing components on an explicit push.
In addition to building and deploying the fixed versions of Ruby, we:
- Released a Changelog entry.
- Sent out an email to all Ruby users.
We also did something unprecedented in the history of Heroku: patched and released two unmaintained language versions, Ruby 1.8.7 and 1.9.2.
Unmaintained Ruby versions
The Ruby 1.8.7 version has been at the End of Life for months, which means that the Ruby Core team will no longer issue bug or security fixes for the language. However, developers are still using it actively in production. Ruby 1.9.2 is currently unmaintained, though this status has not been formally announced.
While security patches were not made available for Ruby 1.8.7 and 1.9.2, Ruby engineer Terence Lee discovered that he could cleanly apply the security fix to both versions. He then built, tested, and released these versions on Heroku’s Cedar stack. Terence recently got commit access to Ruby core and is currently working on pushing the changes upstream even though the two versions are technically unmaintained.
Terence’s actions give you a longer runway but developers using Ruby 1.8.7 or 1.9.2 must upgrade as soon as possible. Heroku recommends upgrading to Ruby 2.0.0 or at very minimum 1.9.3. Matz has stated that support for 1.9.3 will likely be dropped within a year. Heroku’s Bamboo stack runs Ruby Enterprise Edition which has been at end of life since early 2012 and was not patched. Bamboo users should upgrade to the Cedar stack to stay secure.
Staying on a current version is crucial to being able to iterate quickly and respond to vulnerabilities. For example, the Chrome web browser auto updates in the background, and recently Apple software has moved their Mac and iOS app stores to this model. Using updated software provides benefits to a maintainer: less fragmentation and less time spent supporting legacy versions. This means more time for features, performance, and compatibility.
The Rails web framework only supports Ruby 1.9.3+ for their 4.0.0 release, and it is rumored that they will be supporting only 2.0.0+ for their 4.1.0 release. Without dropping support for older syntaxes, developers cannot utilize new ones. As a language user, staying current means you have access to the latest features, latest security updates, and most active language support. For all these reasons, we want to encourage Heroku users to regularly upgrade and stay up-to-date.
The ending of support for Ruby 1.8.7 and 1.9.2 are interesting events for Heroku: they represent the first time a technology used on the platform became unmaintained by its core developers. While we have extended the period you are secure on these unmaintained versions, we have not made a commitment to maintain either Ruby 1.8.7 or 1.9.2 indefinitely. We know that these environments are still in use, and want to make sure customers have ample time to upgrade. We’re working to make Heroku’s Ruby version support commitments and timelines explicit, and will publish documentation to that effect.
Security Matters
For Heroku’s security team, communication is as much a concern as technical fixes. When communication breaks down, so does security. This year we’ve seen several large vulnerabilities that required notifications for languages, frameworks and tools including a Postgres patch and the Rails YAML vulnerability. We’re working on better ways to notify affected application owners. In this incident, we sent notifications to all Ruby application owners even if they were running JRuby or Rubinius, runtimes that were not affected. In the future we aim to be able to dial up our signal-to-noise ratio on security notifications, and even be able to provide app-specific information quickly and securely. If this sounds like fun, our Security team is hiring.
If you’re using Ruby on the platform, be sure to take advantage of the fix as soon as possible. If you’re depending on an older, unmaintained version of Ruby, upgrade as soon as possible. If you’re maintaining your own versions of Ruby, make sure you update and re-compile.
If you’re running on Heroku, sleep well knowing that we care about your security.