The New Heroku (Part 1 of 4): The Process Model & Procfile

In the beginning was the command line. The command line is a direct and immediate channel for communicating with and controlling a computer. GUIs and menus are like pointing and gesturing to communicate; whereas the command line is akin to having a written conversation, with all the nuance and expressiveness of language.

This is not lost on developers, for whom the command prompt and blinking cursor represents the potential to run anything, to do anything. Developers use the command line for everything from starting a new project (rails new) to managing revisions (git commit) to launching secondary, more specialized command lines (psql, mysql, irb, node).

With Celadon Cedar, Heroku's new runtime stack, the power and expressiveness of the command line can be scaled out across a vast execution environment. Heroku provides access to this environment through an abstraction called the process model. The process model maps command-line commands with to app code, creating a collection of processes which together form the running app.

But what does this mean for you, as an app developer? Let's dive into the details of the process model, and see how it offers a new way of thinking about how to build and run applications.

The Foundation: Running a One-Off Process

The simplest manifestation of the process model is running a one-off process. On your local machine, you can cd into a directory with your app, then type a command to run a process. On Heroku's Cedar stack, you can use heroku run to launch a process against your deployed app's code on Heroku's execution environment (known as the dyno manifold).

A few examples:

$ heroku run date
$ heroku run curl http://www.google.com/
$ heroku run rails console
$ heroku run rake -T
$ heroku run rails server

At first glance, heroku run may seem similar to ssh; but the only resemblance is that the command specified is being run remotely. In contrast to ssh, each of these commands is run on a fresh, stand-alone dyno running in different physical locations. Each dyno is fully isolated, starts up with a pristine copy of the app's compiled filesystem, and entire dyno (including process, memory, filesystem) is discarded when the process launched by the command exits or is terminated.

The command heroku run rails server launches a webserver process for your Rails app. Running a webserver in the foreground as a one-off process is not terribly useful: for general operation, you want a long-lived process that exists as a part of a fleet of such processes which do the app's business. To achieve this, we'll need another layer on top of the single-run process: process types, and a process formation.

Defining an App: Process Types via Procfile

A running app is a collection of processes. This is true whether you are running it on your local workstation, or as a production deploy spread out across many physical machines. Historically, there has been no single, language-agnostic, app-centric method for defining the processes that make up an app. To solve this, we introduce Procfile.

Procfile is a format for declaring the process types that describe how your app will run. A process type declares its name and a command-line command: this is a prototype which can be instantiated into one or more running processes.

Here's a sample Procfile for a Node.js app with two process types: web (for HTTP requests), and worker (for background jobs).

Procfile

web:     node web.js
worker:  node worker.js

In a local development environment, you can run a small-scale version of the app by launching one process for each of the two process types with Foreman:

$ gem install foreman
$ foreman start
10:14:40 web.1     | started with pid 13998
10:14:40 worker.1  | started with pid 13999
10:14:41 web.1     | Listening on port 5000
10:14:41 worker.1  | Worker ready to do work

The Heroku Cedar stack has baked-in support for Procfile-backed apps:

$ heroku create --stack cedar
$ git push heroku master
...
-----> Heroku receiving push
-----> Node.js app detected
...
-----> Discovering process types
       Procfile declares types -> web, worker

This Procfile-backed app is deployed to Heroku. Now you're ready to scale out.

Scaling Out: The Processes Formation

Running locally with Foreman, you only need one process for each process type. In production, you want to scale out to much greater capacity. Thanks to a share-nothing architecture, each process type can be instantiated into any number of running processes. Each process of the same process type shares the same command and purpose, but run as separate, isolated processes in different physical locations.

Cedar provides the heroku scale command to make this happen:

$ heroku scale web=10 worker=50
Scaling web processes... done, now running 10
Scaling worker processes... done, now running 50

Like heroku run, heroku scale launches processes. But instead of asking for a single, one-shot process attached to the terminal, it launches a whole group of them, starting from the prototypes defined in your Procfile. The shape of this group of running processes is known as the process formation.

In the example above, the process formation is ten web processes and fifty worker processes. After scaling out, you can see the status of your new formation with the heroku ps command:

$ heroku ps
Process       State               Command
------------  ------------------  ------------------------------
web.1         up for 2s           node web.js
web.2         up for 1s           node web.js
...
worker.1      starting for 3s     node worker.js
worker.2      up for 1s           node worker.js
...

The dyno manifold will keep these processes up and running in this exact formation, until you request a change with another heroku scale command. Keeping your processes running indefinitely in the formation you've requested is part of Heroku's erosion-resistance.

Run Anything

The process model, heroku run, and heroku scale open up a whole new world of possibilities for developers like you working on the Heroku platform.

A simple example: swap out the webserver and worker system used for your Rails app (Heroku defaults to Thin and Delayed Job), and use Unicorn and Resque instead:

Gemfile

gem 'unicorn'
gem 'resque'
gem 'resque-scheduler'

Procfile

web:     bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker:  bundle exec rake resque:work QUEUE=*

For background work, you can run different types of workers consuming from different queues. Add a clock process as a flexible replacement for cron using resque-scheduler:

Procfile

worker:    bundle exec rake resque:work QUEUE=*
urgworker: bundle exec rake resque:work QUEUE=urgent
clock:     bundle exec resque-scheduler

Goliath is an innovative new EventMachine-based evented webserver. Write a Goliath-based app and you'll be able to run it from your Procfile:

Gemfile

gem 'goliath'

Procfile

web: bundle exec ruby hello_goliath.rb -sv -e prod -p $PORT

Or how about a Node.js push-based pubsub system like Juggernaut or Faye?

package.json

{
  "name": "myapp",
  "version": "0.0.1",
  "dependencies": {
    "juggernaut": "2.0.5"
  }
}

Procfile

web: node_modules/.bin/juggernaut

This is just a taste of what you can do with Procfile. The possibilities are nearly limitless.

Conclusion

The command line is a simple, powerful, and time-honored abstraction. Procfile is a layer on top of the command line for declaring how your app gets run. With Cedar, heroku scale becomes your distributed process manager, and heroku run becomes your distributed command line.

We've only just seen the beginning of what the process model can do: over the next year, Heroku will be adding new language runtimes, new routing capabilities, and new types of add-on resources. The sky's the limit, and we can't wait to see what inventive new kinds of apps developers like you will be building.


Other Posts From This Series

Browse the blog archives or subscribe to the full-text feed.