Presenting the New Add-ons Site

Heroku Add-ons make it easy for developers to extend their applications with new features and functionality. The Add-on Provider Program has enabled cloud service providers with key business tools, including billing, single sign-on, and an integrated end-user support experience. Since the launch of the Heroku Add-ons site over two years ago, the marketplace has grown to nearly 100 add-ons. As the add-ons ecosystem has grown, we've learned a lot about how cloud service providers structure their businesses and how users interact with them.

Today we're happy to announce the launch of the updated Heroku Add-ons site.

The goal of the new site is to make it even easier to find, compare, purchase, and use add-ons. In addition to categorization, tagging, search, and an add-on showcase, we've made it easier to understand the benefits of each add-on, distinguish between plans, access documentation, and provision add-ons from the web or the command line. Here are some highlights of the new design:

Showcase

We're now featuring add-ons on the homepage in an active rotation based on three criteria: newness, popularity, and staff picks.

Add-ons Showcase

Categories

We've introduced categories to help you make more informed decisions about which add-ons are right for your use case, like which database to use.

Add-ons Categories

Search

The home page now features a lightning-fast search field. Each search result includes the CLI command to install the add-on, so if you know the add-on you're looking for you can be on and off the site in a matter of seconds.

Add-ons Search

The search tool also has some handy vim-inspired keyboard shortcuts:

  • / focuses the search field.
  • esc clears the search field.
  • j (or down arrow) moves you down in the results.
  • k (or up arrow) moves you up in the results.
  • o (or enter) opens the currently selected search result.
  • y selects the CLI command so you can copy it.

Emphasis on Productivity

In the new marketplace, we've encouraged add-on providers to highlight the ways in which their add-on will improve developers' lives. Rather than emphasizing technical commodities like megabytes of cache or number of allowed requests, benefits highlight the high-level value of each service, such as ease of integration, time saved, and higher productivity.

Add-on Benefits: CloudAMQP

Clear differentiation of Plans

The new plan interface makes it easier to distinguish how an add-on's offerings change across plans.

Add-on Plans

Dev Center Documentation

We've added tighter integration with Dev Center for easy access to each add-on's documentation.

Add-on Documentation

Looking Forward

As of today, the new Add-ons Marketplace is the default for everyone on the platform. Watch closely for updates and new features. To stay up to date as new add-ons enter the marketplace, check out the new add-ons changelog and subscribe to the feed or follow our new twitter account, @HerokuAddons.


Heroku is hiring

Hacking Buildpacks

Buildpacks are an extremely powerful tool for specifying the ecosystem of tools and dependencies packaged with your Heroku application and controlling the way the application is built from code to a deployed app.

In the post announcing the release of buildpacks we illustrated this point, explaining how buildpacks provide the mechanism by which Heroku supports a variety of languages and frameworks, not just Ruby and Rails. We also briefly covered some of the end-user customizations that can be achieved with custom buildpacks, such as adding binary support and modifying the build process.

Today we'll examine the basic structure of buildpacks and study some example customizations to better understand how they can be used to extend the capabilities of the Heroku defaults.

The Anatomy of a Buildpack

At its core, a buildpack is a collection of 3 Bash scripts stored in the bin directory. These scripts are called detect, compile, and release. We'll take a quick look at how each of these scripts contributes to supporting a specific language or framework.

An excellent skeleton buildpack with all of these minimal components is Ryan Smith's null-buildpack. If you're creating a buildpack from scratch, forking null-buildpack is a good place to start.

detect

The bin/detect script is important to Heroku's default buildpacks. When an app is deployed, the detect script is used to figure out which buildpack is appropriate for the project.

Most buildpacks detect frameworks by searching for certain config files. For example, the Ruby buildpack looks for a Gemfile. The Node.js buildpack looks for packages.json, and the Python buildpack looks for requirements.txt.

If a buildpack matches, it returns an exit code of 0 and prints the language/framework name to STDOUT. The aforementioned null-buildpack shows what an absolutely minimal detect script would look like.

In the case of custom buildpacks you'll be specifying the buildpack directly, so detection isn't as important, and a minimal detect script is usually sufficient.

compile

The bin/compile script is where most of the magic happens. This script takes 2 arguments, BUILD_DIR and CACHE_DIR.

BUILD_DIR gives you a handle for the root directory of the app, so you can read and write files into the slug. This is where binaries are installed, Heroku-specific config files are written, dependencies are resolved and installed, and static files are built.

CACHE_DIR gives you a location to persist build artifacts between deployments.

We'll take a closer look at modifying the slug and caching build artifacts in the examples below.

release

Whereas the bin/compile script modifies the slug, the bin/release script modifies the release. Instead of modifying files, this script returns a YAML-formatted hash to define any default config variables, Add-ons, or default process types needed by the buildpack.

Now that we understand the basic structure of buildpack scripts and their roles, lets take a look at some example hacks.

Example 1: Adding Binaries for Language Support

The ability to install binaries is critical for most language support. Richard Schneeman's mruby buildpack article over on RubySource provides an excellent example of installing custom binaries to support a new language.

Building the binary files with Vulcan

The first step for getting new binaries onto Heroku is building them in such a way that they can be run on Heroku dynos (64-bit Linux virtual machines). It turns out that building these binaries directly on Heroku is the easiest method, since the operating system of a Heroku dyno contains common Linux development tools.

Heroku user Jonathan Hoyt discovered these tools early on and blogged about the process of building xpdf for Heroku using:

  • heroku run bash to boot a new dyno and get a bash session on it
  • curl to download the source
  • make to build the project
  • scp to copy the build artifacts to a local machine

Although you could certainly copy Jon's procedure, Heroku now provides a tool called Vulcan to make this process much easier. The vulcan create command deploys a custom app to Heroku under your account. Once the app is created, you use the vulcan build command to upload source, build the project, and download the results all in one step.

For more info on Vulcan, see its README and the Heroku Dev Center article on packaging binaries.

Hosting the built files

Once Vulcan has completed and the build artifacts have been downloaded, you'll need to host them somewhere on the web so that Heroku's build servers will be able to download them. Heroku's default language binaries are stored on AmazonS3, for example. Make sure the location you use is publicly readable.

Modifying the compile script

Next you need to make the buildpack copy the binary files down into your project. This is done in the buildpack's bin/compile script. We can again refer to the mruby buildpack for a straightforward example of how the files are copied down. The steps used are as follows:

  1. Change directories into the build directory. (This directory will be the root of any apps deployed with this buildpack.)

  2. Fetch the archive of the binary files.

  3. Make a directory under /vendor to store the binaries.

  4. Extract the archive.

Modifying the PATH

Finally, you'll need to add the location of your binaries to the PATH environment variable so that they can be called from anywhere. As discussed earlier, default environment variables are defined by the YAML string returned by the bin/release script. Here you can see PATH being set for the mruby binaries.

Example 2: Using the Build Cache to Speed Up Deployments

The default Ruby buildpack provides support for the Rails asset pipeline by running rake assets:precompile, building shorthand source like coffeescript and sass files into static, browser-consumable javascript and css files. As long as the correct conditions are met, this task will be run on every deploy. Unfortunately, assets:precompile can be very slow, especially on large projects with lots of assets.

To address this slowness, Nathan Broadbent released the turbo-sprockets-rails3 gem. TurboSprockets only compiles assets whose source files have changed, making the asset compilation step much faster after the initial run. This is a nifty enhancement, but it depends on the ability to cache assets between builds. This is where the CACHE_DIR argument to the compile script comes in handy.

(NOTE: The Ruby buildpack is a bit different in that it's not totally Bash. The bin/compile script invokes a Ruby script. The Ruby code provides some convenience methods for manipulating files in and out of the build cache.)

Nathan forked and extended the default Ruby buildpack to take advantage of turbo-sprockets-rails3's abilities. This buildpack modifies the default behavior by loading cached files from CACHE_DIR/public/assets into BUILD_DIR/public/assets. The assets:precompile task then runs as usual, but since turbo-sprockets-rails3 is installed, any unmodified assets won't be rebuilt. The script then runs a custom asset expiration task, storing assets back to CACHE_DIR/public/assets if it's successful, and clearing CACHE_DIR/public/assets if it fails.

Example 3: Installing Framework Tools

Many popular languages have several competing frameworks for web apps; Rails and Sinatra for Ruby and Django and Pylons for Python are some well-known examples. Usually, these frameworks are contained within a few libraries, so if you want to support a new framework it makes sense to modify existing language buildpacks instead of starting from scratch. James Ward took this approach when he wanted to support Revel, a web framework for the Go programming language.

First he forked the existing Go buildpack.

Next, he modified the bin/detect script to look for a Revel-specific file and announce that it's a Revel buildpack.

Then he added some functionality to the end of the bin/compile script to fetch and build Revel.

A new line in the bin/release script defines a default web process in the Procfile for running Revel.

Conclusion

I hope this tour into hacking buildpacks has been informative, and you've gained some insight into how buildpacks work and how they can be extended to meet your needs. Whether you want to host apps in a new language, or tweak the tools for an existing one, buildpacks are a step towards always answering "yes" to the question, "Does it run on Heroku?"

For more reference information, please check out the buildpack articles available in our Dev Center.

Fork Your Application's Data (Not Just Your Code) with Heroku Postgres

In the same way Git and Github revolutionized software development by letting you Fork your code with a single click, Heroku Postgres is changing the way you work with database by letting you Fork your data.

The ability to Fork your data is generally available as of today. This exciting technology which is simple, safe (placing no load on your primary database), and robust makes available some exciting new use cases:

  • Analytics reporting
  • Worry-free migrations
  • Trivial cloning of data to development/staging

Read more about our Fork announcement or get started using Fork today and then let us know about your exciting uses at dod-feedback@heroku.com

Fork Your Application's Data (Not Just Your Code)

Git and Github revolutionized software development by letting you fork your source repository with a single click. Wouldn't it be great to be able to do the same thing with your database?

In the same way you can fork your code you can now fork your data. Fork changes the way you can work with your data, making it a snap to provision a clone of your production database. The technology is simple, safe, and robust, and thanks to Heroku Postgres' cloud architecture, places no load on your primary database. Today, we’re announcing the release of this functionality into General Availability.

Forking your data opens up whole new ways of developing software. We've seen our users finding all kinds of use-cases including:

  • Easier load testing
  • Worry-free migrations
  • Trivial cloning of data to development/staging

How many times have you run a migration that went wrong in production but worked perfectly against your testing data? With Heroku Postgres, you can simply fork your production database and test the migration against real data. After it finishes, if the result looks good, you can promote it, fix it, or throw it away. If there's a problem, you still have your original database.

This kind of fearless interaction is the product of a new way of thinking about data. By shifting the focus away from database servers and towards your data, Heroku Postgres gives you tools to work risk-free throughout the entire software lifecycle: in development, staging, and production.

Creating forks is supported from both the CLI and through the Heroku Postgres dashboard. From within the Heroku Postgres dashboard, you can make a fork with one click:

Fork

You can also use a single command with the Heroku CLI to fork your data:

$ heroku addons:add heroku-postgresql:crane --fork HEROKU_POSTGRESQL_LAVENDER
Adding heroku-postgresql:crane on sushi... done, v113 ($50/mo)
Attached as HEROKU_POSTGRESQL_VIOLET_URL
...

Heroku Postgres features like fork empower you to focus on what really matters – building your application and business. By choosing Heroku Postgres as your database-as-a-service provider you get the benefit of constant improvement, and new functionality which gives you the ability to build better software faster and more safely.

Fork is available immediately on every Heroku Postgres production database. To learn more about forking your data, you can read the dev center article, go to the Heroku Postgres dashboard and click the fork button on one of your existing databases, or simply create one from the command-line. As always with Heroku, you only pay for what you use, so give it a try today and see how easy it is and then let us know how you're using it at dod-feedback@heroku.com.

Hacking mruby onto Heroku

If you're in the Ruby world, you've likely heard about mruby, Matz's latest experimental Ruby implementation. What I bet you didn't know is that you can run mruby on Heroku right now. As a matter of fact you can run just anything on Heroku, as long as it can compile it into a binary on a Linux box.

If you're new to mruby, or to compiling binaries take a look at my last article Try mruby Today. I cover getting mruby up and running on your local machine. If you are already up to speed then follow along as we use vulcan to package mruby as binary, wrap it up in a custom buildpack and then launch an app to use mruby on the Heroku cloud.

Continue Reading ...

Yesterday

If you missed it yesterday we announced official support for Ruby 2.0.0 Preview1, and announced the dates for our developer conference, Waza 2013, including the Waza call for Speakers.

Browse the blog archives or subscribe to the full-text feed.