Buildpacks: Heroku for Everything
July 17, 2012 by Matthew Manning
Last summer, Heroku became a polyglot platform, with official support for Ruby, Node.js, Clojure, Java, Python, and Scala. Building a platform that works equally well for such a wide variety of programming languages was a unique technical design challenge.
We knew from the outset that maintaining siloed, language-specific products – a Heroku for Ruby, a Heroku for Node.js, a Heroku for Clojure, and so on – wouldn't be scalable over the long-term.
Instead, we created Cedar: a single, general-purpose stack with no native support for any language. Adding support for any language is a matter of layering on a build-time adapter that can compile an app written in a particular language or framework into an executable that can run on the universal runtime provided by Cedar. We call this adapter a buildpack.
If you've deployed any app to the Cedar stack, then you've already used at least one buildpack, since the buildpack is what executes during
git push heroku master. Let's explore the Ruby buildpack by looking at the terminal output that results when deploying a Rails 3.2 app:
$ git push heroku master Counting objects: 67, done. Delta compression using up to 4 threads. Compressing objects: 100% (53/53), done. Writing objects: 100% (67/67), 26.33 KiB, done. Total 67 (delta 5), reused 0 (delta 0) -----> Heroku receiving push -----> Ruby/Rails app detected -----> Installing dependencies using Bundler version 1.2.0.pre Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment Fetching gem metadata from https://rubygems.org/....... Installing rake (0.9.2.2) ... Your bundle is complete! It was installed into ./vendor/bundle -----> Writing config/database.yml to read from DATABASE_URL -----> Preparing app for Rails asset pipeline Running: rake assets:precompile Asset precompilation completed (16.16s) -----> Rails plugin injection Injecting rails_log_stdout Injecting rails3_serve_static_assets -----> Discovering process types Procfile declares types -> (none) Default types for Ruby/Rails -> console, rake, web, worker -----> Compiled slug size is 9.6MB -----> Launching... done, v4 http://chutoriaru.herokuapp.com deployed to Heroku To firstname.lastname@example.org:chutoriaru.git * [new branch] master -> master
Everything that happens between
Heroku receiving push and
Compiled slug size is 9.6MB is part of the buildpack. In order:
- installing Ruby,
- installing and running Bundler to manage gem dependencies,
- injecting database configuration,
- compiling Rails assets,
- and installing Heroku-specific plugins for logging and serving static assets.
In the example above, the appropriate buildpack was automatically detected from our list of Heroku-maintained defaults.
However, you can also specify your desired buildpack using arguments to the
heroku create command or by setting the
BUILDPACK_URL config variable. This enables the use of custom buildpacks. If you want to run your Rails app on JRuby, for example, specify the buildpack created by the JRuby team at app creation time:
$ heroku create --buildpack https://github.com/jruby/heroku-buildpack-jruby
Since language support can be completely contained inside a buildpack, it is possible to deploy an app written in nearly any language to Heroku. Indeed, there are a variety of third-party buildpacks already available:
- Perl by Lincoln Stoll
- Common Lisp by Mike Travers
- Go by Keith Rarick
- Dart by Ilya Grigorik
- Null by Ryan Smith
See the full list of third party buildpacks in the Dev Center.
In addition to enabling new language support, the ability to select a buildpack allows you to modify the previously closed Heroku build process for popular languages.
For example, consider a Ruby app that needs to generate static files using Jekyll. Before buildpacks, the only solutions would have been to 1) generate the files before deployment and commit them to the repository or 2) generate the files on-the-fly at runtime. Neither of these solutions are ideal as they violate the strict separation that should be maintained between the codebase, the build stage, and the run stage.
All of the default buildpacks are open source, available for you to inspect, and fork to modify for your own purposes. And if you make a change that you think would be useful to others, please submit an upstream pull request!
Your app might depend on binaries such as language VMs or extensions that are not present in the default runtime. If this is the case, these dependencies can be packaged into the buildpack. A good example is this fork of the default Ruby buildpack which adds library support for the couchbase gem. Vulcan is a tool to help you build binaries compatible with the 64-bit Linux architecture which dynos run on.
Buildpacks are potentially useful in any environment, and we'd love to see their usage spread beyond the Heroku platform. Minimizing lock-in and maximizing transparency is an ongoing goal for Heroku.
Using buildpacks can be a convenient way to leverage existing, open-source code to add new language and framework support to your own platform. Stackato, a platform-as-a-service by ActiveState, recently announced support for Heroku buildpacks.
You can also run buildpacks on your local workstation or in a traditional server-based environment with Mason.
Get started hacking buildpacks today by forking the Hello Buildpack! Read up on the implementation specifics laid out in the Buildpack API documentation, and join the public Buildpacks Google Group. If you make a buildpack you think would be useful and that you intend to maintain, send us an email at email@example.com and for potential inclusion on the third-party buildpacks page.