Buildpacks: Heroku for Everything

Last summer, Heroku became a polyglot platform, with official support for Ruby, Node.js, Clojure, Java, Python, and Scala. Building a platform that works equally well for such a wide variety of programming languages was a unique technical design challenge.

siloed products would be a non-scalable design

We knew from the outset that maintaining siloed, language-specific products – a Heroku for Ruby, a Heroku for Node.js, a Heroku for Clojure, and so on – wouldn't be scalable over the long-term.

Instead, we created Cedar: a single, general-purpose stack with no native support for any language. Adding support for any language is a matter of layering on a build-time adapter that can compile an app written in a particular language or framework into an executable that can run on the universal runtime provided by Cedar. We call this adapter a buildpack.

build-time language adapters support a single runtime stack

An Example: Ruby on Rails

If you've deployed any app to the Cedar stack, then you've already used at least one buildpack, since the buildpack is what executes during git push heroku master. Let's explore the Ruby buildpack by looking at the terminal output that results when deploying a Rails 3.2 app:

$ git push heroku master
Counting objects: 67, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (53/53), done.
Writing objects: 100% (67/67), 26.33 KiB, done.
Total 67 (delta 5), reused 0 (delta 0)

-----> Heroku receiving push
-----> Ruby/Rails app detected
-----> Installing dependencies using Bundler version 1.2.0.pre
       Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment
       Fetching gem metadata from https://rubygems.org/.......
       Installing rake (0.9.2.2)
       ...
       Your bundle is complete! It was installed into ./vendor/bundle
-----> Writing config/database.yml to read from DATABASE_URL
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       Asset precompilation completed (16.16s)
-----> Rails plugin injection
       Injecting rails_log_stdout
       Injecting rails3_serve_static_assets
-----> Discovering process types
       Procfile declares types      -> (none)
       Default types for Ruby/Rails -> console, rake, web, worker
-----> Compiled slug size is 9.6MB
-----> Launching... done, v4
       http://chutoriaru.herokuapp.com deployed to Heroku

To git@heroku.com:chutoriaru.git
 * [new branch]      master -> master

Everything that happens between Heroku receiving push and Compiled slug size is 9.6MB is part of the buildpack. In order:

The slug that results from this Rails-specific build process can now be booted on our language-agnostic dyno manifold alongside Python, Java, and many other types of applications.

Using a Custom Buildpack

In the example above, the appropriate buildpack was automatically detected from our list of Heroku-maintained defaults.

However, you can also specify your desired buildpack using arguments to the heroku create command or by setting the BUILDPACK_URL config variable. This enables the use of custom buildpacks. If you want to run your Rails app on JRuby, for example, specify the buildpack created by the JRuby team at app creation time:

$ heroku create --buildpack https://github.com/jruby/heroku-buildpack-jruby

Arbitrary Language Support

Since language support can be completely contained inside a buildpack, it is possible to deploy an app written in nearly any language to Heroku. Indeed, there are a variety of third-party buildpacks already available:

See the full list of third party buildpacks in the Dev Center.

Customizing the Build Process

In addition to enabling new language support, the ability to select a buildpack allows you to modify the previously closed Heroku build process for popular languages.

For example, consider a Ruby app that needs to generate static files using Jekyll. Before buildpacks, the only solutions would have been to 1) generate the files before deployment and commit them to the repository or 2) generate the files on-the-fly at runtime. Neither of these solutions are ideal as they violate the strict separation that should be maintained between the codebase, the build stage, and the run stage.

By forking the official Ruby buildpack, you could add a site generation step to your build process, putting file generation in the build stage where it belongs.

All of the default buildpacks are open source, available for you to inspect, and fork to modify for your own purposes. And if you make a change that you think would be useful to others, please submit an upstream pull request!

Adding Binary Support

Your app might depend on binaries such as language VMs or extensions that are not present in the default runtime. If this is the case, these dependencies can be packaged into the buildpack. A good example is this fork of the default Ruby buildpack which adds library support for the couchbase gem. Vulcan is a tool to help you build binaries compatible with the 64-bit Linux architecture which dynos run on.

Buildpacks Beyond Heroku

Buildpacks are potentially useful in any environment, and we'd love to see their usage spread beyond the Heroku platform. Minimizing lock-in and maximizing transparency is an ongoing goal for Heroku.

Using buildpacks can be a convenient way to leverage existing, open-source code to add new language and framework support to your own platform. Stackato, a platform-as-a-service by ActiveState, recently announced support for Heroku buildpacks.

You can also run buildpacks on your local workstation or in a traditional server-based environment with Mason.

Conclusion

Get started hacking buildpacks today by forking the Hello Buildpack! Read up on the implementation specifics laid out in the Buildpack API documentation, and join the public Buildpacks Google Group. If you make a buildpack you think would be useful and that you intend to maintain, send us an email at buildpacks@heroku.com and for potential inclusion on the third-party buildpacks page.

Heroku Postgres Basic Plan and Row Limits

Today, the Heroku Postgres team released into beta the new basic plan, $9 / month version of the free dev plan.

Accompanying this announcement is the implementation of a 10,000 row limit on the dev plan. This row limit was designed to correspond to the 5mb limit on the existing free shared plan.

Please note that these plans are still beta, and Heroku Postgres has not yet announced a migration schedule from the shared plan. However you can start using these plans today.

Read more about the new plan, and the mechanics of the row limits on the Heroku Postgres Blog.

Ten Million Rows for Under Ten Bucks

Six weeks ago we launched into beta the Heroku Postgres dev plan, a free, postgres 9.1 plan that offers many of the features of our production tier service. Over 3,000 of these dev databases are in active use, and it has been operating exceptionally well.

When we launched the dev plan, we wrote that the plan would be limited based on rows rather than physical byte size. Today we are implementing a 10,000 row limit for the dev plan. This limit was chosen to correspond to the 5mb limit on the existing, shared database service. Over 98% of the active shared databases that are under 5mb are also under the 10,000 row limit.

Introducing the Basic Plan

If you need more than 10,000 rows, you can upgrade to the production tier crane plan ($50 / mo), or upgrade to the new new basic plan, available today.

Basic is identical to dev except that it has a 10 million row limit. The basic plan will be available for $9 / month when it exits beta. We expect the beta period for basic to be brief, so it should be provisioned only if you intend to purchase it.

Understanding Row Limits

The dev and basic database plans are both row limited. In order to ensure that the row limits do not disrupt application operation, we have developed the following mechanism for enforcement:

  1. When your database is at 70% of its row capacity, the owner receives a warning e-mail.
  2. When the database exceeds its row capacity, the owner will receive an additional notification. At this point, the database will receive a 24-hour grace period to either reduce the number of records, or migrate to another plan.
  3. If the number of rows still exceeds the plan capacity after 24 hours, INSERT privileges will be revoked on the database. Data can still be read, updated or deleted from database. This ensures that users still have the ability to bring their database into compliance, and retain access to their data.
  4. Once the number of rows is again in compliance with the plan limit, INSERT privileges are automatically restored to the database. Note that the database sizes are checked asynchronously, so it may take a few minutes for the privileges to be restored.

The Future

The dev and basic plans are a big leap forward from the current shared database offering. By leveraging the infrastructure of our production tier plans, we've built a powerful, low-cost database service that is accesible to a wide audience of developers.

We encourage all of our users to start using the dev plan by default for all new apps. Simply enable the Heroku Labs feature flag:

$ heroku labs:enable default-heroku-postgresql-dev

Please read more about this flag on this Dev Center article.

You can also provision a database through the Heroku add-ons catalog or standalone service today.

Codon Security Issue and Response

Heroku learned of and resolved a security vulnerability last week. We want to report this to you, describe how we responded to the incident, and reiterate our commitment to constantly improving the security and integrity of your data and source code.

On Tuesday, June 26, Jonathan Rudenberg notified us about an issue in our Codon build system. The Codon build system is responsible for receiving application code from Git and preparing it for execution on the Aspen and Cedar stacks. This vulnerability exposed a number of sensitive credentials which could be used to obtain data and source code of customer applications. Upon receiving notification we rolled the most sensitive credentials. An initial patch was in place within 24 hours. The final patch was deployed to production after thorough testing the morning of Friday, June 29. That same morning all relevant credentials were rotated.

Subsequent to this patch, we conducted a thorough and comprehensive audit of our internal logs. We found no evidence that these credentials were used to obtain customer data or credentials, either by Jonathan or any third parties.

We would like to thank Jonathan for notifying us of this vulnerability last week, and giving us ample opportunity to fix it. He provides his description of events on his blog at http://titanous.com/posts/vulnerabilities-in-heroku-build-system

We are confident in the steps we took to protect our customers from this vulnerability and are redoubling our efforts to provide you with the most secure cloud platform available. We would also like to reaffirm our commitment to the security and integrity of our customer's data and code. Nothing is more important to us.

Cedar is the Default Heroku Stack

The Heroku Cedar stack went public beta last year with a series of blog posts. Since then, over 80,000 developers have deployed over 4.5 million times, to apps written in dozens of different programming languages and frameworks. Today, over 75 percent of Heroku app development activity is on the Cedar stack. Production apps like Banjo, Rapportive, PageLever, do.com, and Project Zebra run on Cedar; some of these serve hundreds of millions or even billions of requests per month.

Cedar features a streamlined HTTP stack allowing for advanced HTTP capabilities, heroku run for execution of arbitrary one-off dynos, Procfile and the process model for execution of any type of worker process. Most importantly, Cedar is a polyglot platform with official support for Clojure, Java, Node.js, Python, Ruby, and Scala, and extensibility for unlimited others via buildpacks.

You can still create applications on one of our other stacks using heroku create --stack, but we recommend Cedar for all new apps. If you have applications under active development running on Aspen or Bamboo, we recommend migrating to Cedar.

Browse the blog archives or subscribe to the full-text feed.