Buildpacks: Heroku for Everything

Last summer, Heroku became a polyglot platform, with official support for Ruby, Node.js, Clojure, Java, Python, and Scala. Building a platform that works equally well for such a wide variety of programming languages was a unique technical design challenge.

siloed products would be a non-scalable design

We knew from the outset that maintaining siloed, language-specific products – a Heroku for Ruby, a Heroku for Node.js, a Heroku for Clojure, and so on – wouldn't be scalable over the long-term.

Instead, we created Cedar: a single, general-purpose stack with no native support for any language. Adding support for any language is a matter of layering on a build-time adapter that can compile an app written in a particular language or framework into an executable that can run on the universal runtime provided by Cedar. We call this adapter a buildpack.

build-time language adapters support a single runtime stack

An Example: Ruby on Rails

If you've deployed any app to the Cedar stack, then you've already used at least one buildpack, since the buildpack is what executes during git push heroku master. Let's explore the Ruby buildpack by looking at the terminal output that results when deploying a Rails 3.2 app:

$ git push heroku master
Counting objects: 67, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (53/53), done.
Writing objects: 100% (67/67), 26.33 KiB, done.
Total 67 (delta 5), reused 0 (delta 0)

-----> Heroku receiving push
-----> Ruby/Rails app detected
-----> Installing dependencies using Bundler version 1.2.0.pre
       Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment
       Fetching gem metadata from https://rubygems.org/.......
       Installing rake (0.9.2.2)
       ...
       Your bundle is complete! It was installed into ./vendor/bundle
-----> Writing config/database.yml to read from DATABASE_URL
-----> Preparing app for Rails asset pipeline
       Running: rake assets:precompile
       Asset precompilation completed (16.16s)
-----> Rails plugin injection
       Injecting rails_log_stdout
       Injecting rails3_serve_static_assets
-----> Discovering process types
       Procfile declares types      -> (none)
       Default types for Ruby/Rails -> console, rake, web, worker
-----> Compiled slug size is 9.6MB
-----> Launching... done, v4
       http://chutoriaru.herokuapp.com deployed to Heroku

To git@heroku.com:chutoriaru.git
 * [new branch]      master -> master

Everything that happens between Heroku receiving push and Compiled slug size is 9.6MB is part of the buildpack. In order:

The slug that results from this Rails-specific build process can now be booted on our language-agnostic dyno manifold alongside Python, Java, and many other types of applications.

Using a Custom Buildpack

In the example above, the appropriate buildpack was automatically detected from our list of Heroku-maintained defaults.

However, you can also specify your desired buildpack using arguments to the heroku create command or by setting the BUILDPACK_URL config variable. This enables the use of custom buildpacks. If you want to run your Rails app on JRuby, for example, specify the buildpack created by the JRuby team at app creation time:

$ heroku create --buildpack https://github.com/jruby/heroku-buildpack-jruby

Arbitrary Language Support

Since language support can be completely contained inside a buildpack, it is possible to deploy an app written in nearly any language to Heroku. Indeed, there are a variety of third-party buildpacks already available:

See the full list of third party buildpacks in the Dev Center.

Customizing the Build Process

In addition to enabling new language support, the ability to select a buildpack allows you to modify the previously closed Heroku build process for popular languages.

For example, consider a Ruby app that needs to generate static files using Jekyll. Before buildpacks, the only solutions would have been to 1) generate the files before deployment and commit them to the repository or 2) generate the files on-the-fly at runtime. Neither of these solutions are ideal as they violate the strict separation that should be maintained between the codebase, the build stage, and the run stage.

By forking the official Ruby buildpack, you could add a site generation step to your build process, putting file generation in the build stage where it belongs.

All of the default buildpacks are open source, available for you to inspect, and fork to modify for your own purposes. And if you make a change that you think would be useful to others, please submit an upstream pull request!

Adding Binary Support

Your app might depend on binaries such as language VMs or extensions that are not present in the default runtime. If this is the case, these dependencies can be packaged into the buildpack. A good example is this fork of the default Ruby buildpack which adds library support for the couchbase gem. Vulcan is a tool to help you build binaries compatible with the 64-bit Linux architecture which dynos run on.

Buildpacks Beyond Heroku

Buildpacks are potentially useful in any environment, and we'd love to see their usage spread beyond the Heroku platform. Minimizing lock-in and maximizing transparency is an ongoing goal for Heroku.

Using buildpacks can be a convenient way to leverage existing, open-source code to add new language and framework support to your own platform. Stackato, a platform-as-a-service by ActiveState, recently announced support for Heroku buildpacks.

You can also run buildpacks on your local workstation or in a traditional server-based environment with Mason.

Conclusion

Get started hacking buildpacks today by forking the Hello Buildpack! Read up on the implementation specifics laid out in the Buildpack API documentation, and join the public Buildpacks Google Group. If you make a buildpack you think would be useful and that you intend to maintain, send us an email at buildpacks@heroku.com and for potential inclusion on the third-party buildpacks page.

Browse the blog archives or subscribe to the full-text feed.