Password Hijacking Security Vulnerability and Response

Heroku recently learned of and resolved a security vulnerability. We want to report this to you, describe how we responded to the incident, and reiterate our commitment to constantly improving the security and integrity of your data and source code.

On December 19, 2012, security researcher Stephen Sclafani notified us of an issue in our account creation system. Using a maliciously-crafted HTTP request, an attacker could change the password of a pre-existing Heroku user account, and thus gain control of it. This attack would not disclose the pre-existing password to the attacker (those are stored internally as non-recoverable bcrypt hashes).

Upon receiving notification, our engineering and security staff engaged with Mr. Sclafani. We developed and deployed a preliminary patch to production on December 20. While we were deploying the patch, Mr. Sclafani also discovered a related issue in the password reset flow that could be used to reset the passwords of a certain subset of users at random. A preliminary patch for this was also developed and deployed on December 20.

After deploying these patches, we conducted a thorough and comprehensive audit of our internal logs. We found no evidence that these vulnerabilities were exploited prior to Mr. Sclafani’s research on December 19, either by him or any other third parties. Due to the nature of the vulnerability, any customer whose account was compromised would have found both their existing password and API key invalidated, and would have had to initiate a password reset.

While both Mr. Sclafani and Heroku endeavoured to use test accounts exclusively, a very small number of customer account passwords were reset during the incident. We have contacted the impacted customers and advised them to reset their passwords and credentials.

We would like to thank Mr. Sclafani for notifying us of this vulnerability, and giving us ample opportunity to fix it. His description is available at http://stephensclafani.com/2013/01/09/vulnerabilities-in-heroku/. We are extremely grateful to both him and all external security researchers who practice responsible disclosure.

We are confident in the steps we have taken to protect our customers from this vulnerability and will continue to improve our internal processes in order to provide our customers with a trusted cloud platform. We would also like to reaffirm our commitment to the security and integrity of our customers’ data and code. Nothing is more important to us.

Oren Teich, Chief Operating Officer

Run JRuby on Heroku Right Now

Over a year ago Heroku launched the Cedar stack and the ability to run Java on our platform. Java is known as a powerful language - capable of performing at large scale. Much of this potential comes from the JVM that Java runs on. The JVM is the stable, optimized, cross-platform virtual machine that also powers other languages including Scala and Clojure. Starting today you can leverage the power of the JVM in your Ruby applications without learning a new language, by using JRuby on Heroku.

After a beta process with several large production applications, we are pleased to move JRuby support into general availability immediately. One of these companies Travis CI which provides free CI testing to open source repositories, and a pro plan for private projects, was a JRuby beta tester. Josh Kalderimis of the Travis team had this to say about using JRuby on Heroku:

We love JRuby, everything from the threading support to having the power of the JVM at our finger tips. But what we love most is that we can set up a JRuby app in seconds, the same way as all of our other Heroku apps. Git push and it's live, no matter what the language.

We've been working with the JRuby team to make sure that the experience using the language on Heroku is going to be everything you've come to expect from using our platform. So why should you be interested in running JRuby?

Why JRuby

If you're coming from a Java background and want to use a more dynamic language, JRuby allows you to leverage the syntax of Ruby with the the ability to run JVM based libraries. If you're a Ruby developer already on Heroku, the JRuby implementation has several unique features that you can leverage. The most prevalent difference between running code on JRuby and MRI, or cRuby, is JRuby's lack of a Global Virtual Machine Lock. This means you can run multiple threads of JRuby code within the same process. While cRuby does allow you to perform IO and other non-ruby commands in parallel threads, running Ruby code concurrently can only be done in multiple processes. The second difference is the JVM ecosystem. JRuby can use Java libraries such as JDBC based database drivers. Many of these libraries have been heavily optimized and can offer speed upgrades.

JRuby on Heroku

JRuby on Heroku lowers the barrier of entry to both learning and running a new language in production. The interface of JRuby with the Heroku platform is the same as our other languages: you push your code to us and we do the rest. You don't need to think about all of the details of running a new language. The result is you get to focus on adding features, not on your how to deploy and keep your systems up.

We have been working with the JRuby community together to make sure the experience is a good one. Charles Nutter, the co-lead of JRuby, is excited about the future of running JRuby on Heroku:

One of the most frequently-requested features for JRuby isn't a JRuby feature at all...it's support for JRuby on Heroku. We're very excited that Heroku now officially supports JRuby, and we're looking forward to working with and supporting Heroku users trying out JRuby on their cloud of choice.

By normalizing the interface to deployment across implementations, we hope to ease the process of trying new intrepreters within the Ruby community. We are excited to see a new class of applications, by running Ruby on the JVM, deployed and supported on Heroku. With all these options, which Ruby should you use in production?

Which Ruby to Use?

Heroku supports many languages, and we have a long and happy history of supporting Ruby. We are continuing to invest in the exciting future of MRI we are also excited about the ability for you to run your code on the interpreter of your choice. This can open up new possibilities such as taking advantage of different VM optimizations or concurrent Ruby processing.

As you're trying JRuby, remember that it may behave slightly differently than you're used to with MRI. If you're interested in trying JRuby out on an existing Heroku app, you can read more about converting an existing Rails app to use JRuby.

Every app is different and every project has different requirements. Having the ability to quickly and easily run your app in production on multiple Ruby VMs gives you the power to choose.

Try it Today

If you have an existing Rails app you can deploy your existing app on JRuby. If you're just starting from scratch, running JRuby on Heroku is a breeze. All you need to do is specify the version of Ruby you want to run, the engine, and the engine version in your Gemfile:

ruby '1.9.3', engine: 'jruby', engine_version: '1.7.1'

You'll need to run bundle install with JRuby locally, then commit the results and push to Heroku:

$ git add .
$ git commit -m "trying JRuby on Heroku"
$ git push heroku master
Counting objects: 692, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (662/662), done.
Writing objects: 100% (692/692), 141.01 KiB, done.
Total 692 (delta 379), reused 0 (delta 0)

-----> Heroku receiving push
-----> Ruby/Rails app detected
-----> Using Ruby version: ruby-1.9.3-jruby-1.7.1
-----> Installing JVM: openjdk7-latest
-----> Installing dependencies using Bundler version 1.2.1
# ...

That should be all you need to do to run JRuby on Heroku. If you're converting an existing Rails application, please read moving an existing Rails app to run on JRuby.

While you're trying JRuby out on Heroku you can also try out Ruby 2.0.0 preview, before it is released in February.

Conclusion

With the release of JRuby on Heroku now you can to run your code on multiple VMs, and leverage concurrent Ruby code in production. You've got a standard interface to deployments and the power to choose the right tool for the right job. Special thanks to all of the customers who tried the JRuby beta, and to the JRuby team for being available for technical support. Give JRuby a try and let us know what you think.

Postgres 9.2 – The Database You Helped Build

Hosting your data on one of the largest fleets of databases in the world comes with certain advantages. One of those benefits is that we can aggregate the collective pain points that face our users and work within the Postgres community to help find solutions to them.

In the previous year we worked very closely with the broader Postgres community to build features, fix bugs, and resolve pain points. You've already seen some of the results of that work in the form of extension support on Heroku and query cancellation. With the 9.2 release we're delighted to say that with your help, we've been able to bring you a whole host of new power and simplicity in your database.

Effective immediately, we're moving Postgres 9.2 into GA, which will become the new default shortly after. Postgres 9.2 is full of simplifications and new features that will make your life better, including:

  • Expressive new datatypes
  • New tools for getting deep insights into your database's performance
  • User interface improvements.

You can request a version 9.2 database from the command line like this:

heroku addons:add heroku-postgresql:dev --version=9.2

Get started by provisioning one today or read more about the many great features now available in Postgres 9.2 over on the Heroku Postgres blog

Postgres 9.2 – The Database You Helped Build

Hosting your data on one of the largest fleets of databases in the world comes with certain advantages. One of those benefits is that we can aggregate the collective pain points that face our users and work within the Postgres community to help find solutions to them.

In the previous year we worked very closely with the broader Postgres community to build features, fix bugs, and resolve pain points. You've already seen some of the results of that work in the form of extension support on Heroku and query cancellation. With the 9.2 release we're delighted to say that with your help, we've been able to bring you a whole host of new power and simplicity in your database.

Effective immediately, we're moving Postgres 9.2 support into GA, which will become the new default shortly after. Postgres 9.2 is full of simplifications and new features that will make your life better, including expressive new datatypes, new tools for getting deep insights into your database's performance, and even some simple user interface improvements. Oh, and it's much, much faster for the most common kind of write performance pattern we see in our fleet.

You can request a version 9.2 database from the command line like this:

heroku addons:add heroku-postgresql:dev --version=9.2

Let's dig in a bit further with the new features this version brings.

Visibility

Visibility into your data has long been a problem for many application developers. Thanks to Peter Geoghegan, and the many involved in reviewing/testing, in the new version of Postgres all queries are normalized and data about them is recorded. This allows you to gain insight such as:

  • How often a query is run
  • How much time is spent running the query
  • How much data is returned

Each of these key pieces of data are critical when it comes to being able to effectively optimize your database's performance. The old way of drudging through logs is no longer needed to gain this insight. Now your database contains what it needs in order to help you improve performance within an un-forked Postgres database. Ensuring such functionality is committed back to the Postgres core is very important as it prevents lock-in and creates a better ecosystem for the community as a whole.

Let's take a look at how we can begin using some of this. First turn on the tracking of pg_stat_statements with CREATE EXTENSION pg_stat_statements; Then run the query below and you'll receive all of your top run queries:

SELECT 
    count(*),
    query 
FROM
  pg_stat_statements 
GROUP BY 2 
ORDER BY 1 DESC 
LIMIT 10;

We're very excited about the visibility you can now gain into your database. We've begun exploring the powerful new ways we can show what's occurring with your database and look forward to seeing how we and our users can further expand the power of the improved visibility within Postgres 9.2.

URLs

All Postgres tools and libraries now support URLs natively. No more need for heroku pg:credentials -- just use the URL with any Postgres project tool.

JSON Support

Developers are always looking for more extensibility and power when working with and storing their data. Earlier this year we announced our support for hstore, a powerful key/value store within Postgres, which you can easily use within Rails, Django, and Java Spring.

With Postgres 9.2 there's even more robust support for NoSQL within your SQL database, thanks to Andrew Dunstan, in the form of JSON. By using the JSON datatype your JSON is validated that it's proper JSON before it's allowed to be committed.

Beyond the datatype itself there are several new functions available – record_to_json, row_to_json, and array_to_json. Using these functions we can turn a row immediately into JSON to be used within an application or returned via an API:

$ heroku pg:psql
=> SELECT row_to_json(row('foo','bar', 1, 2));
     row_to_json     
---------------------
 {"f1":"foo","f2":"bar", "f3": 1, "f4": 2}
(1 row)

Range Type Support

The range datatype, thanks to Jeff Davis, is another example of powerful data flexibility. The range datatype is a single column consisting of a to and from value. Your range can exist as a range of timestamps, alpha-numeric, or numeric range and can even have constraints placed on it to enforce common range conditions.

For example, this schema ensures that in creating a class schedule we can't have two classes at the same time:

CREATE TABLE schedule (class int, during tsrange);
ALTER TABLE schedule ADD EXCLUDE USING gist (during WITH &&);

Then attempting to add data we would receive an error:

INSERT INTO schedule VALUES (3, '[2012-09-24 13:00, 2012-09-24 13:50)');
INSERT INTO schedule VALUES
(1108, '[2012-09-24 13:30, 2012-09-24 14:00)');
ERROR:  conflicting key value violates exclusion constraint "schedule_during_excl"

Performance

Of course, any new release of a database wouldn't be complete without some focus on performance. Postgres 9.2, as expected, has delivered here in a big way including up to 4X improvements in speed on read queries and up to 20X improvements on data warehousing queries. In particular index-only scans can offer much faster queries because they no longer need to access disk to ensure correct results.

Summary

Heroku Postgres provides reliability and safety when working with your data. At Heroku Postgres, we were very excited to be able to fund core Postgres features for the first time and work with the community more closely to make Postgres an even better cloud database. The support of Postgres 9.2, now in general availability, makes power, flexibility and insight available to all Heroku Postgres users. Whether you’re looking to have NoSQL in your SQL database, better understand visibility, or receive a performance boost, this version should help you. Get started by provisioning one today from the Heroku CLI:

heroku addons:add heroku-postgresql:dev --version=9.2

Presenting the New Add-ons Site

Heroku Add-ons make it easy for developers to extend their applications with new features and functionality. The Add-on Provider Program has enabled cloud service providers with key business tools, including billing, single sign-on, and an integrated end-user support experience. Since the launch of the Heroku Add-ons site over two years ago, the marketplace has grown to nearly 100 add-ons. As the add-ons ecosystem has grown, we've learned a lot about how cloud service providers structure their businesses and how users interact with them.

Today we're happy to announce the launch of the updated Heroku Add-ons site.

The goal of the new site is to make it even easier to find, compare, purchase, and use add-ons. In addition to categorization, tagging, search, and an add-on showcase, we've made it easier to understand the benefits of each add-on, distinguish between plans, access documentation, and provision add-ons from the web or the command line. Here are some highlights of the new design:

Showcase

We're now featuring add-ons on the homepage in an active rotation based on three criteria: newness, popularity, and staff picks.

Add-ons Showcase

Categories

We've introduced categories to help you make more informed decisions about which add-ons are right for your use case, like which database to use.

Add-ons Categories

Search

The home page now features a lightning-fast search field. Each search result includes the CLI command to install the add-on, so if you know the add-on you're looking for you can be on and off the site in a matter of seconds.

Add-ons Search

The search tool also has some handy vim-inspired keyboard shortcuts:

  • / focuses the search field.
  • esc clears the search field.
  • j (or down arrow) moves you down in the results.
  • k (or up arrow) moves you up in the results.
  • o (or enter) opens the currently selected search result.
  • y selects the CLI command so you can copy it.

Emphasis on Productivity

In the new marketplace, we've encouraged add-on providers to highlight the ways in which their add-on will improve developers' lives. Rather than emphasizing technical commodities like megabytes of cache or number of allowed requests, benefits highlight the high-level value of each service, such as ease of integration, time saved, and higher productivity.

Add-on Benefits: CloudAMQP

Clear differentiation of Plans

The new plan interface makes it easier to distinguish how an add-on's offerings change across plans.

Add-on Plans

Dev Center Documentation

We've added tighter integration with Dev Center for easy access to each add-on's documentation.

Add-on Documentation

Looking Forward

As of today, the new Add-ons Marketplace is the default for everyone on the platform. Watch closely for updates and new features. To stay up to date as new add-ons enter the marketplace, check out the new add-ons changelog and subscribe to the feed or follow our new twitter account, @HerokuAddons.


Heroku is hiring

Browse the blog archives, subscribe to the full-text feed, or visit the engineering blog.