In the beginning was the command line. The command line is a direct and immediate channel for communicating with and controlling a computer. GUIs and menus are like pointing and gesturing to communicate; whereas the command line is akin to having a written conversation, with all the nuance and expressiveness of language.
This is not lost on developers, for whom the command prompt and blinking cursor represents the potential to run anything, to do anything. Developers use the command line for everything from starting a new project (rails new) to managing revisions (git commit) to launching secondary, more specialized command lines (psql, mysql, irb, node).
With Celadon Cedar, Heroku's new runtime stack, the power and expressiveness of the command line can be scaled out across a vast execution environment. Heroku provides access to this environment through an abstraction called the process model. The process model maps command-line commands with to app code, creating a collection of processes which together form the running app.
But what does this mean for you, as an app developer? Let's dive into the details of the process model, and see how it offers a new way of thinking about how to build and run applications.
The Foundation: Running a One-Off Process
The simplest manifestation of the process model is running a one-off process. On your local machine, you can cd into a directory with your app, then type a command to run a process. On Heroku's Cedar stack, you can use heroku run to launch a process against your deployed app's code on Heroku's execution environment (known as the dyno manifold).
A few examples:
$ heroku run date
$ heroku run curl http://www.google.com/
$ heroku run rails console
$ heroku run rake -T
$ heroku run rails server
At first glance, heroku run may seem similar to ssh; but the only resemblance is that the command specified is being run remotely. In contrast to ssh, each of these commands is run on a fresh, stand-alone dyno running in different physical locations. Each dyno is fully isolated, starts up with a pristine copy of the app's compiled filesystem, and entire dyno (including process, memory, filesystem) is discarded when the process launched by the command exits or is terminated.
The command heroku run rails server launches a webserver process for your Rails app. Running a webserver in the foreground as a one-off process is not terribly useful: for general operation, you want a long-lived process that exists as a part of a fleet of such processes which do the app's business. To achieve this, we'll need another layer on top of the single-run process: process types, and a process formation.
Defining an App: Process Types via Procfile
A running app is a collection of processes. This is true whether you are running it on your local workstation, or as a production deploy spread out across many physical machines. Historically, there has been no single, language-agnostic, app-centric method for defining the processes that make up an app. To solve this, we introduce Procfile.
Procfile is a format for declaring the process types that describe how your app will run. A process type declares its name and a command-line command: this is a prototype which can be instantiated into one or more running processes.
Here's a sample Procfile for a Node.js app with two process types: web (for HTTP requests), and worker (for background jobs).
Procfile
web: node web.js
worker: node worker.js
In a local development environment, you can run a small-scale version of the app by launching one process for each of the two process types with Foreman:
$ gem install foreman
$ foreman start
10:14:40 web.1 | started with pid 13998
10:14:40 worker.1 | started with pid 13999
10:14:41 web.1 | Listening on port 5000
10:14:41 worker.1 | Worker ready to do work
The Heroku Cedar stack has baked-in support for Procfile-backed apps:
$ heroku create --stack cedar
$ git push heroku master
...
-----> Heroku receiving push
-----> Node.js app detected
...
-----> Discovering process types
Procfile declares types -> web, worker
This Procfile-backed app is deployed to Heroku. Now you're ready to scale out.
Scaling Out: The Processes Formation
Running locally with Foreman, you only need one process for each process type. In production, you want to scale out to much greater capacity. Thanks to a share-nothing architecture, each process type can be instantiated into any number of running processes. Each process of the same process type shares the same command and purpose, but run as separate, isolated processes in different physical locations.
Cedar provides the heroku scale command to make this happen:
$ heroku scale web=10 worker=50
Scaling web processes... done, now running 10
Scaling worker processes... done, now running 50
Like heroku run, heroku scale launches processes. But instead of asking for a single, one-shot process attached to the terminal, it launches a whole group of them, starting from the prototypes defined in your Procfile. The shape of this group of running processes is known as the process formation.
In the example above, the process formation is ten web processes and fifty worker processes. After scaling out, you can see the status of your new formation with the heroku ps command:
$ heroku ps
Process State Command
------------ ------------------ ------------------------------
web.1 up for 2s node web.js
web.2 up for 1s node web.js
...
worker.1 starting for 3s node worker.js
worker.2 up for 1s node worker.js
...
The dyno manifold will keep these processes up and running in this exact formation, until you request a change with another heroku scale command. Keeping your processes running indefinitely in the formation you've requested is part of Heroku's erosion-resistance.
Run Anything
The process model, heroku run, and heroku scale open up a whole new world of possibilities for developers like you working on the Heroku platform.
A simple example: swap out the webserver and worker system used for your Rails app (Heroku defaults to Thin and Delayed Job), and use Unicorn and Resque instead:
Gemfile
gem 'unicorn'
gem 'resque'
gem 'resque-scheduler'
Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake resque:work QUEUE=*
For background work, you can run different types of workers consuming from different queues. Add a clock process as a flexible replacement for cron using resque-scheduler:
Procfile
worker: bundle exec rake resque:work QUEUE=*
urgworker: bundle exec rake resque:work QUEUE=urgent
clock: bundle exec resque-scheduler
Goliath is an innovative new EventMachine-based evented webserver. Write a Goliath-based app and you'll be able to run it from your Procfile:
Gemfile
gem 'goliath'
Procfile
web: bundle exec ruby hello_goliath.rb -sv -e prod -p $PORT
Or how about a Node.js push-based pubsub system like Juggernaut or Faye?
package.json
{
"name": "myapp",
"version": "0.0.1",
"dependencies": {
"juggernaut": "2.0.5"
}
}
Procfile
web: node_modules/.bin/juggernaut
This is just a taste of what you can do with Procfile. The possibilities are nearly limitless.
Conclusion
The command line is a simple, powerful, and time-honored abstraction. Procfile is a layer on top of the command line for declaring how your app gets run. With Cedar, heroku scale becomes your distributed process manager, and heroku run becomes your distributed command line.
We've only just seen the beginning of what the process model can do: over the next year, Heroku will be adding new language runtimes, new routing capabilities, and new types of add-on resources. The sky's the limit, and we can't wait to see what inventive new kinds of apps developers like you will be building.