Hacking Buildpacks

Buildpacks are an extremely powerful tool for specifying the ecosystem of tools and dependencies packaged with your Heroku application and controlling the way the application is built from code to a deployed app.

In the post announcing the release of buildpacks we illustrated this point, explaining how buildpacks provide the mechanism by which Heroku supports a variety of languages and frameworks, not just Ruby and Rails. We also briefly covered some of the end-user customizations that can be achieved with custom buildpacks, such as adding binary support and modifying the build process.

Today we'll examine the basic structure of buildpacks and study some example customizations to better understand how they can be used to extend the capabilities of the Heroku defaults.

The Anatomy of a Buildpack

At its core, a buildpack is a collection of 3 Bash scripts stored in the bin directory. These scripts are called detect, compile, and release. We'll take a quick look at how each of these scripts contributes to supporting a specific language or framework.

An excellent skeleton buildpack with all of these minimal components is Ryan Smith's null-buildpack. If you're creating a buildpack from scratch, forking null-buildpack is a good place to start.

detect

The bin/detect script is important to Heroku's default buildpacks. When an app is deployed, the detect script is used to figure out which buildpack is appropriate for the project.

Most buildpacks detect frameworks by searching for certain config files. For example, the Ruby buildpack looks for a Gemfile. The Node.js buildpack looks for packages.json, and the Python buildpack looks for requirements.txt.

If a buildpack matches, it returns an exit code of 0 and prints the language/framework name to STDOUT. The aforementioned null-buildpack shows what an absolutely minimal detect script would look like.

In the case of custom buildpacks you'll be specifying the buildpack directly, so detection isn't as important, and a minimal detect script is usually sufficient.

compile

The bin/compile script is where most of the magic happens. This script takes 2 arguments, BUILD_DIR and CACHE_DIR.

BUILD_DIR gives you a handle for the root directory of the app, so you can read and write files into the slug. This is where binaries are installed, Heroku-specific config files are written, dependencies are resolved and installed, and static files are built.

CACHE_DIR gives you a location to persist build artifacts between deployments.

We'll take a closer look at modifying the slug and caching build artifacts in the examples below.

release

Whereas the bin/compile script modifies the slug, the bin/release script modifies the release. Instead of modifying files, this script returns a YAML-formatted hash to define any default config variables, Add-ons, or default process types needed by the buildpack.

Now that we understand the basic structure of buildpack scripts and their roles, lets take a look at some example hacks.

Example 1: Adding Binaries for Language Support

The ability to install binaries is critical for most language support. Richard Schneeman's mruby buildpack article over on RubySource provides an excellent example of installing custom binaries to support a new language.

Building the binary files with Vulcan

The first step for getting new binaries onto Heroku is building them in such a way that they can be run on Heroku dynos (64-bit Linux virtual machines). It turns out that building these binaries directly on Heroku is the easiest method, since the operating system of a Heroku dyno contains common Linux development tools.

Heroku user Jonathan Hoyt discovered these tools early on and blogged about the process of building xpdf for Heroku using:

  • heroku run bash to boot a new dyno and get a bash session on it
  • curl to download the source
  • make to build the project
  • scp to copy the build artifacts to a local machine

Although you could certainly copy Jon's procedure, Heroku now provides a tool called Vulcan to make this process much easier. The vulcan create command deploys a custom app to Heroku under your account. Once the app is created, you use the vulcan build command to upload source, build the project, and download the results all in one step.

For more info on Vulcan, see its README and the Heroku Dev Center article on packaging binaries.

Hosting the built files

Once Vulcan has completed and the build artifacts have been downloaded, you'll need to host them somewhere on the web so that Heroku's build servers will be able to download them. Heroku's default language binaries are stored on AmazonS3, for example. Make sure the location you use is publicly readable.

Modifying the compile script

Next you need to make the buildpack copy the binary files down into your project. This is done in the buildpack's bin/compile script. We can again refer to the mruby buildpack for a straightforward example of how the files are copied down. The steps used are as follows:

  1. Change directories into the build directory. (This directory will be the root of any apps deployed with this buildpack.)

  2. Fetch the archive of the binary files.

  3. Make a directory under /vendor to store the binaries.

  4. Extract the archive.

Modifying the PATH

Finally, you'll need to add the location of your binaries to the PATH environment variable so that they can be called from anywhere. As discussed earlier, default environment variables are defined by the YAML string returned by the bin/release script. Here you can see PATH being set for the mruby binaries.

Example 2: Using the Build Cache to Speed Up Deployments

The default Ruby buildpack provides support for the Rails asset pipeline by running rake assets:precompile, building shorthand source like coffeescript and sass files into static, browser-consumable javascript and css files. As long as the correct conditions are met, this task will be run on every deploy. Unfortunately, assets:precompile can be very slow, especially on large projects with lots of assets.

To address this slowness, Nathan Broadbent released the turbo-sprockets-rails3 gem. TurboSprockets only compiles assets whose source files have changed, making the asset compilation step much faster after the initial run. This is a nifty enhancement, but it depends on the ability to cache assets between builds. This is where the CACHE_DIR argument to the compile script comes in handy.

(NOTE: The Ruby buildpack is a bit different in that it's not totally Bash. The bin/compile script invokes a Ruby script. The Ruby code provides some convenience methods for manipulating files in and out of the build cache.)

Nathan forked and extended the default Ruby buildpack to take advantage of turbo-sprockets-rails3's abilities. This buildpack modifies the default behavior by loading cached files from CACHE_DIR/public/assets into BUILD_DIR/public/assets. The assets:precompile task then runs as usual, but since turbo-sprockets-rails3 is installed, any unmodified assets won't be rebuilt. The script then runs a custom asset expiration task, storing assets back to CACHE_DIR/public/assets if it's successful, and clearing CACHE_DIR/public/assets if it fails.

Example 3: Installing Framework Tools

Many popular languages have several competing frameworks for web apps; Rails and Sinatra for Ruby and Django and Pylons for Python are some well-known examples. Usually, these frameworks are contained within a few libraries, so if you want to support a new framework it makes sense to modify existing language buildpacks instead of starting from scratch. James Ward took this approach when he wanted to support Revel, a web framework for the Go programming language.

First he forked the existing Go buildpack.

Next, he modified the bin/detect script to look for a Revel-specific file and announce that it's a Revel buildpack.

Then he added some functionality to the end of the bin/compile script to fetch and build Revel.

A new line in the bin/release script defines a default web process in the Procfile for running Revel.

Conclusion

I hope this tour into hacking buildpacks has been informative, and you've gained some insight into how buildpacks work and how they can be extended to meet your needs. Whether you want to host apps in a new language, or tweak the tools for an existing one, buildpacks are a step towards always answering "yes" to the question, "Does it run on Heroku?"

For more reference information, please check out the buildpack articles available in our Dev Center.

Fork Your Application's Data (Not Just Your Code) with Heroku Postgres

In the same way Git and Github revolutionized software development by letting you Fork your code with a single click, Heroku Postgres is changing the way you work with database by letting you Fork your data.

The ability to Fork your data is generally available as of today. This exciting technology which is simple, safe (placing no load on your primary database), and robust makes available some exciting new use cases:

  • Analytics reporting
  • Worry-free migrations
  • Trivial cloning of data to development/staging

Read more about our Fork announcement or get started using Fork today and then let us know about your exciting uses at dod-feedback@heroku.com

Hacking mruby onto Heroku

If you're in the Ruby world, you've likely heard about mruby, Matz's latest experimental Ruby implementation. What I bet you didn't know is that you can run mruby on Heroku right now. As a matter of fact you can run just anything on Heroku, as long as it can compile it into a binary on a Linux box.

If you're new to mruby, or to compiling binaries take a look at my last article Try mruby Today. I cover getting mruby up and running on your local machine. If you are already up to speed then follow along as we use vulcan to package mruby as binary, wrap it up in a custom buildpack and then launch an app to use mruby on the Heroku cloud.

Continue Reading ...

Yesterday

If you missed it yesterday we announced official support for Ruby 2.0.0 Preview1, and announced the dates for our developer conference, Waza 2013, including the Waza call for Speakers.

Waza 2013

Waza Returns to San Francisco in February 2013

Heroku's developer event, Waza, returns on Thursday, February 28th, 2013 to the Concourse in San Francisco. Sign up to be notified when tickets are available.

What is Waza?

Waza (技) is the Japanese word for art and technique. At Heroku, we believe that software development is a craft. Building modern technologies that engage and inspire is an art, with techniques shared, passed on, and honed in the process of creation.

Waza is an event where developers can find inspiration – about what’s happening in technology, what’s happening at Heroku, how people are thinking about the future, and how the landscape of technology is changing. Through technical sessions and non-technical, artistic, and interactive happenings, Waza celebrates what it means to embrace art, technique and the creative process of software development.

Held in early 2012, the inaugural Waza featured moments that included large-format photography, architectural origami, hacking, discussions, musical performances, and technical sessions on a range of topics. Building upon the successes and lessons learned from Waza 2012, the next Waza will be equally unconventional, more ambitious, and even more focused on technical developer content.

Call For Speakers

At the core of this year's developer gathering is exceptional content. We have opened a Call For Speakers for Waza 2013 through November 30th. We are looking for interesting stories, innovative projects, emerging philosophies, and technical craftsmanship. We're hunting for topics ranging from technical deep-dives about cutting-edge application architectures and stellar & advanced uses of the Heroku platform, to non-technical topics such as the philosophy of application design or building modern company culture. If you have something to share in the realm of art and technique that could fit well at Waza, send some information to waza@heroku.com and start the conversation.

Ruby 2.0 Preview Available on Heroku

When Heroku first launched you could only use one version of Ruby: 1.8.6. As the Ruby implementation matured and improved, so did Heroku. We recently announced the ability to specify your ruby version on Heroku, and we are happy to announce the first preview-build of Ruby available: starting today you can use Ruby 2.0 preview1 on Heroku.

Ruby 2.0

The Ruby core team has been hard at work on Ruby 2.0, which has a host of new features and boasts performance improvements. You can get a list of the major new features on the official Ruby 2.0.0 Preview1 announcement.

Heroku has been committed to the Ruby project by sponsoring the work of Yukihiro "Matz" Matsumoto, Koichi Sasada and Nobuyoshi Nakada on MRI Ruby. We have been very pleased with Ruby Core's progress, and look forward to the full release. You can take an in-depth look at the new Ruby 2.0 features available. But don't just read about it, try it out on Heroku today:

Get Ruby 2.0 Running on Heroku

Start by making a Gemfile that specifies Ruby 2.0.0:

$ echo 'source "https://rubygems.org"'  > Gemfile
$ bundle install
$ echo 'ruby "2.0.0"' >> Gemfile

Add the files to a Git repository:

$ git init
$ git add .
$ git commit -m "Ruby 2.0.0 preview on Heroku"

Then create a new heroku app and deploy:

$ heroku create
$ git push heroku master
-----> Heroku receiving push
-----> Ruby app detected
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using Bundler version 1.2.1
       Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment
       The Gemfile specifies no dependencies
       Your bundle is complete! It was installed into ./vendor/bundle
       Cleaning up the bundler cache.
-----> Discovering process types
       Procfile declares types -> (none)
       Default types for Ruby  -> console, rake
-----> Compiled slug size: 23.3MB
-----> Launching... done, v4
       http://safe-earth-3679.herokuapp.com deployed to Heroku

To git@heroku.com:safe-earth-3679.git
 * [new branch]      master -> master

Now you can run Ruby 2.0 beta on Heroku!

$ heroku run bash
Running `bash` attached to terminal... up, run.1
~ $ ruby --version
ruby 2.0.0dev (2012-11-01 trunk 37411) [x86_64-linux]
~ $ ruby -e "puts 'hello world'"
hello world

Why Run 2.0 Preview1?

Ruby 2.0 will ship on February 24th, 2013 - which is four days before our developer conference Waza. Heroku's own Matz has announced that Rails 3.2 apps should work with Ruby 2.0 if they work on Ruby 1.9. Help us find any incompatibilities by checking your existing applications on Heroku. If you run into any bugs in the Ruby implementation, please open a bug in the Ruby bug tracker and let the Ruby team know what didn't work with the implementation.

Troubleshooting

If you are updating an existing app you will likely need to update your config vars manually. New apps should not experience any problems.

Thanks

Thanks to the entire Ruby Core Team for their hard work on this release of Ruby, and special thanks to Terence Lee. Terence is Heroku's Ruby buildpack maintainer who did the leg work to enable Ruby 2.0.0 Preview1 on Heroku. Please try out this Ruby preview today, to help the Ruby core team and to help our community. Try it today.

Browse the blog archives or subscribe to the full-text feed.