One of the many benefits of Rails is database independence. Migrations are particularly nice in this regard; and the easy-to-read / Rubyified display of your schema (via rake db:schema:dump) in schema.rb is icing on the cake.

But what about data? For import and export of the actual data, we’re stuck with mysqldump (or pg_dump, if you’re so inclined). Further, these dump formats are not terribly readable, contain lots of information you may or may not want to copy (like permissions, schema settings, views, triggers…you know, database features that Rails users are supposed to avoid).

Worst of all, ddata dumps are vendor specific, so you can’t move data between databases (e.g., SQLite, MySQL, PostgreSQL). Rails is database-independent, but you’re pretty much stuck with whatever you picked at the start of the project. That’s hardly agile.

Realizing this problem was going to affect us in a pretty direct way with the import/export features of Heroku, Orion and I threw together a plugin yesterday which we’ve given the (rather uninspired) name YamlDb. Once you’ve installed it:
script/plugin install git://github.com/adamwiggins/yaml_db.git

(use http://github.com/adamwiggins/yaml_db.git for Rails < 2.1)

…you’ll have the additional rake tasks db:data:dump and db:data:load, which access a file data.yml. This is a simple dump of your database’s tables, which along with schema.rb, describe everything you need to make a complete copy of your database. (db:dump and db:load are shortcuts to do both schema and data together.)

What’s awesome about this is that you can switch databases in the blink of an eye. Got a SQLite3 database that’s gotten too big for its britches? rake db:dump, create a MySQL database, update your config/database.yml, and rake db:load. Presto, your app is now running in MySQL. Want to give PostgreSQL a try? Same process. Run it for a week, and decided you miss mysqladmin too much? Reverse the process, bringing the data which was stored in Postgres during that week back into MySQL.

Caveats: it’s much slower than dumping and restoring with your database’s native tool, and probably will stop working altogether with a database that is larger than RAM. But for most apps it should work fine. In the future we will most likely write database-specific extensions to make dump/restore process happen in chunks, and thus be more scalable.

As for Heroku, it now uses yaml_db as its native import/export format. So when you export an app from Heroku, you’ll always see a db/data.yml with your current data. It also makes it easy to import data into your Heroku app, at the same time you’re importing the code, or later on. Just install yaml_db in your app locally and run db:dump. Include data.yml with the import (or run and upload it separately), and then run db:load at the rake console.

We’ve been working our tails off over the past few weeks to process all the feedback you guys have been sending (or that we’ve gleaned from the system logs). I think that this photo of the trashcan under Orion’s desk tells the story pretty well:

He bought that case of Rockstar at Costco last week, and consumed it all as part of our mad dash to squash bugs exposed by our sudden surge of users. Bad for Orion’s health, but good for Heroku’s backend stability. :)

One major area we’ve been dealing with in this past week is the issue of failed mongrel starts. That is, exceptions that occur while the Rails framework is booting, rather than on a page request. These sorts of exceptions often happen with imported apps, because certain types of plugins or gem dependencies may not work out of the box with Heroku yet, or you might just have some odd stuff in your environment.rb that isn’t compatible with the version of Rails we’re running.

Up until recently, this was producing an unhelpful HTTP 502 bad gateway page. Now, you’ll see the contents of your mongrel.log so that you can diagnose the issue.

If you were paying close attention, you might have noticed a “Restart” button that existed for a few days. We’ve removed that in favor of automatically restarting whenever you save a file that needs a restart (like routes.rb or environment.rb). So far this seems to work pretty well – but as always, let us know if you find a case where it doesn’t work as expected.

Sometimes, it’s the little things. A few niceties deployed recently:

• The code editor UI now has a liquid layout. If you’re a life hacking / GTD type like me, you’ll especially enjoy this in combination with Firefox’s fullscreen mode. (FF for OS X doesn’t have fullscreen, unfortunately; try this instead.)
• Download files from the context menu. You can use this in conjunction with upload to edit in your local editor, load an image into your photopaint program, etc.
• Speaking of images, if you click on an image, it will display it in the editor pane.
• There’s a link to update your account password on the My Apps page. (Shocked this wasn’t there before? We’re in beta, dammit, we’re allowed to slack off on stuff like this.)
• More bugfixes than you can shake a ticket-tracker at.

Behold: the Heroku gems/plugins manager.

This has been one of our most requested features to date, and we’re glad to finally get this released. Although you could manually upload plugins previously, this will make the process a lot smoother. (You can still manually manipulate the files in your vendor directory if you prefer.)

To get to the manager, open your vendor directory in the lefthand filenav, and click the link that appears at the top:

You can search by name, or browse the list of 2500+ gems and 1000+ plugins. Once you find what you’re looking for, click on Install in the righthand column to install it into your app. Click Remove to remove it if it’s no longer needed.

You can also upload your own plugin as a tarball or a .gem file. Just click the Upload link at the bottom. Once it’s installed, you’ll see it listed alongside your other installed gems and plugins, and you can remove it in the same way. (Though you’ll need to upload it again if you decide you want it back.)

The data for plugins comes from the Agile Web Development plugin directory. Thanks to Ben Curtis, the site’s author and maintainer, for adding some extra fields to the XML API for us. The data for gems comes from the relative newcomer Gemtacular.

Note that gems with binary dependencies are unlikely to work. If you’re trying to install a gem and it doesn’t work, email us and we’ll see what we can do. Additionally, there are quite a lot of gems that don’t make any sense in the context of Heroku – for example, the 3rd party ActiveRecord database adapters, or GUI toolkits like the GTK bindings. In iteration 2 of this component we hope to flag gems and plugins as Heroku-friendly or not.

We made the design decision to mix gems and plugins together in the same view. This will probably cause at least a few folks to protest “But gems and plugins are nothing alike!” Make no mistake, we know the difference quite well. In building the interface, however, it occurred to us that the process of managing them was very similar. And from a high-level point of view, they are two forks of the same tree: add-ons to extend or modify the behavior of the programming environment of your app.

There are now two access levels for collaborators on Heroku apps:

• Full edit access, which allows access to everything: editing code, importing or exporting the database, changing the settings, etc.
• View-only access, which allows the user to view the app only. That is, they can visit the app url (myapp.heroku.com) but not any of the settings pages or the edit url (edit.myapp.heroku.com).

For example, a client who wants to use the app but neither needs nor wants access to the code could be set as a view-only user.

If your app sharing is set to public, the view-only access level has no use.

Do note that these settings have no effect on users changing your app’s data through the normal web front-end. For example, if you have a scaffold page that doesn’t perform any authentication, a view-only user can create, update, and delete records. When we say “full edit access” we’re referring to editing code. What happens when the user views your app is up to you.

RSpec 1.1 is now a part of the default plugin kit for Heroku apps.

We’ve been fans of RSpec for a while now, and feel that it represents the future of TDD/BDD for the Rails world. If you’re not familiar with RSpec, read up and then give it a try.

You don’t need to install anything to use RSpec in your Heroku app, but you do need to initialize the spec/ and stories/ directories by running the rspec generator. Just open the Generate dialog, type in rspec, and click Run.

Once you’ve written some specs, you can run them the usual way: open a rake console and type spec. You can still run your Test::Unit tests with the test command, or you can run tests followed by spec with default.

Don’t worry, support for Test::Unit won’t be going away any time soon – but we do encourage you to consider RSpec the next time you create a new model. Use the generate command rspec_model instead of the usual model to get a blank spec created for you. Or, use rspec_model <model_name> against an existing model and it will generate the spec (in spec/models) without overwriting the existing model or migration.

Backstory: A Fiery Debate

Writing a user model and the standard login authentication code seems like busywork to a lot of coders. In fact, many people expected a next-generation app framework such as Rails to handle this for you. After all, Django does. Initially the login engine for Rails seemed to fill this slot, but following a fair amount of controversy over best practices, the login engine was killed by its creator.

With our BDfL having forever cursed prebuilt login systems, the Rails community mostly stopped trying to make them. Yet, this puts us back at square one: developers are annoyed at the amount of boilerplate busywork that is necessary for almost every web app they write.

acts_as_authencated is the halfway solution that is now popular: it’s a generator, not a drop-in component, so it spits out the boilerplate for you, and then you can modify it. And then of course there’s the idea that logins shouldn’t be maintained by individual sites at all, but stored someplace in the ownership of users. OpenID is the great hope here, but while we wait for this technology to mature (and gain acceptance with less technical audiences), maintaining user logins will continue to be a part of building web apps.

The debate over how to create login authentication will continue to smoulder for some time yet. But in the meantime, Heroku now offers a user login solution that will be handy for apps shared with a small number of people, and requires almost no code.

Heroku Users

Apps created on Heroku are already shared with some number of users, specified by their email addresses (this works the same as other types of collaborative editing apps, such as Google Docs). Since these users are already logging in to access the app, wouldn’t it be handy if you could find out from the Heroku backend who was logged into your app?

We thought so too. Which why we’ve created the heroku_user helper object. It’s a small feature, but a surprisingly convenient one. I’ve already found it quite useful in some of my own personal apps. Our company wiki, for example, uses this method. So how does it work?

Heroku now has an API (accessible from the command line, a Ruby library, or REST calls), revision control on all apps with Git, and remote access to the Git repository.

The combination of these new features means that you can now work on your apps using the local tools you love – like TextMate, vi, or emacs – and still get the benefit of zero-configuration deployment to Heroku.

How does it work? Grab the Heroku gem with “gem install heroku”. A sample work session looks like this:

heroku clone myapp cd myapp ruby script/server …edit locally… git add . git commit -m “local changes” git push

The final step will deploy the app to Heroku, including running the migrations on the database and restarting the server. Watch the screencast to see it in action, or just grab the gem and give it a try yourself. RDocs here.

Combine your local tools and the Heroku in-the-cloud development tools in any combination you like. Perhaps you want to work locally while at home, but use the web editor when traveling. Every commit to the repository is available from both.

The Heroku API gets a major update today; you can now view and manage all of your application’s settings straight from the command line. New in this version:

• Manage sharing (add/remove/list collaborators)
• Manage multiple ssh keys for your user (add/remove/list keys)
• Update settings (public true/false, mode production/development)
• Rename an app
• Run rake tasks remotely

A taste of the new command-line goodness:

adam@kvasir:~$ heroku create gagetron
Created http://gagetron.heroku.com/ | git@heroku.com:gagetron.git

adam@kvasir:~$ heroku info gagetron
=== gagetron
Web URL:        http://gagetron.heroku.com/
Git Repo:       git@heroku.com:gagetron.git
Mode:           development
Public:         false
Collaborators:  adam@example.com (edit)

adam@kvasir:~$ heroku sharing gagetron --add joe@example.com
joe@example.com added as a view-only collaborator on gagetron.

adam@kvasir:~$ heroku rake gagetron routes
(in /mnt/home/userapps/27934)
  /:controller/:action/:id         
  /:controller/:action/:id.:format 

There’s a new screencast which shows managing sharing from the command line. We’ve also updated the screencasts which show how to use the API and Git to edit locally, then deploy to Heroku.

Grab the new gem from Rubyforge with gem install heroku, read the docs, or browse the source.

Say you’re working on a Rails app, and you want to publish your code on Github. Most apps have some deploy-specific private config values – for example, if you’re using the S3 storage backend for Paperclip, and your S3 keys are saved in config/amazon_keys.yml. You certainly don’t want to push those up to Github – what to do?

You could maintain a separate deploy branch, and commit your deploy config only to that. You can then work on the main branch, and rebase the deploy branch whenever you go for a deploy. That’s a bit of extra work you could do without, though – and you know sooner or later, you’re going to accidentally push the wrong branch to Github, putting your S3 keys up for the whole world to see. Oops.

Wouldn’t it be nice avoid all this headache by storing the data specific to your app’s deploy in a separate configuration registry?

Now you can. Check it out:

$ heroku config:add S3_BUCKET=mybucket S3_KEY=8N029N81 S3_SECRET=9s83109d3+583493190
Adding config vars:
  S3_BUCKET => mybucket
  S3_KEY    => 8N029N81
  S3_SECRET => 9s83109d3+583493190
Restarting app...done.

Your config vars are now accessible in the environment variables for your deployed app. Configure Paperclip like this:

has_attached_file :photo,
  :storage => :s3,
  :s3_credentials => {
    :access_key_id => ENV['S3_KEY'],
    :secret_access_key => ENV['S3_SECRET']
  },
  :bucket => ENV['S3_BUCKET'],
  :path => ":attachment/:id"

Set up like this, your public code and your deployed code can be exactly the same. Push to Github all day long with no fear of leaking your private credentials. Clean and neat, no fuss, no muss.

This also means you can set different config vars for different deployed versions of the app. For example, if you run a staging deploy of your code on Heroku, you can set it to use a different S3 bucket – maybe “myapp_assets” for production and “myapp_staging_assets” for staging.

What about running locally, deploying to a traditional host? Since these are just environment variables, you can set them just as you normally would – in your .bashrc (export S3_BUCKET=mybucket), or on the command line when you run the server.

Or, you can check for the presence of the vars, and use defaults when they don’t exist. In the case of Paperclip, you might do:

has_attached_file :photo,
  :storage => ENV['S3_BUCKET'] ? :s3 : :filesystem,
  ...

Now when you run locally, it’ll use the filesystem store; but running on a deployed app with the S3_BUCKET var set, it’ll automatically go to the right place.

There are many other uses for config vars. For example, you can set RACK_ENV to something other than the default of ‘production’. (RAILS_ENV and MERB_ENV will automatically mirror RACK_ENV on Heroku.) So if you prefer the traditional approach of storing different keys for different environments (development/staging/production) in various config/something.yml files, set your RACK_ENV as appropriate for your staging and production deployed apps on Heroku.

Read the full documentation. Note that you’ll need to grab the 0.7 Heroku client gem (it’s in Rubyforge now, so `gem install heroku` will do the trick) to use this feature.

Since we returned from a fun and successful Railsconf in Vegas, we have been in full swing completing the rollout of our paid services. The response has been enormous so far, and paid services are now available to all users.

If you’ve checked out the pricing page, you’ve undoubtedly noticed our line-up of a la carte add-ons. We’re really excited about add-ons becoming a key part of our platform, allowing us to seamlessly deliver popular application services and components with the built-in scalability and ease of use you’ve come to expect from Heroku.

We’ve had a solid first batch of add-ons in beta for a while, and today we’re happy to announce the first graduate: Wildcard Domains, which allows your app to respond on *.yourdomain.com. Since we started supporting custom domains on Heroku this has been high on the list of requested features, and now it’s here! To add it to your app for $5/month, simply go to your app’s Resources page and select turn it on. Enjoy!

Our goal for the Heroku platform has been to create a totally smooth and seamless experience for getting your Ruby web application online. Web apps revolve around one or more dynamic web processes: what Rubyists often call a mongrel, and what we call a dyno. When it comes to dynos, we think we’ve really nailed it, and nothing makes that more tangible than the ease of scaling your app with the dyno slider.

But most serious web apps have a second aspect: one that gets less attention, but one that is often just as important as the web process. Like the dark side of the moon, this second half of your application is not directly visible to users, but without it the app would not be whole. This aspect of the application is often referred to as background jobs. The processes which handle background jobs are called workers.

A worker is used for any heavy lifting your app needs to do. Some examples include: sending email, accessing a remote API (like posting something to Twitter), fetching posts from an RSS feed, converting an image thumbnail, or rendering a PDF. Anything that will take more than 100ms should go in a worker process, not your web process.

Ad-hoc Background Jobs

There are many ways to create worker processes that run asynchronously from your web requests. Examples include forking a web process, or running a cron job every minute to check for work. Those ad-hoc techniques work ok in some situations, but they not highly maintainable or scalable. At Heroku, we don’t want to do something unless we can do it right. We want our workers to be as smooth, powerful, scalable, and easy to use as our dynos.

Background Jobs Done Right

Worker processes on Heroku run completely independently of your dynos. They can be scaled out horizontally across our grid of instances to any size, independent of the number of dynos you’re running. (Some apps need more concurrency for the web front, some for the background workers – scaling the two should be orthogonal.)

One way to describe a mongrel or dyno is as a share-nothing process that consumes web requests. Likewise, a worker can be described as a share-nothing process that consumes jobs from a work queue.

Work Queues

Work queues are currently an area of academic debate among thought leaders in the Ruby community. Work queues consist of two parts: a queueing system and a message bus. The message bus may be an implementation of a messaging standard, or a custom protocol.

Some examples of queueing systems include: Delayed::Job, Workling, BJ, and Starling.

Some examples of message buses include: RabbitMQ (implementing the AMQP protocol), Beanstalkd, Amazon SQS, and Kestrel.

These are some great tools, many of which are in production use by major Ruby sites. For example, Twitter uses Kestrel, while Scribd uses loops and ActiveMQ. But when the largest Rails sites in the world don’t have a consensus on the best tool for the job, what should the rest of us be using?

Luckily, there’s an excellent solution for medium-sized apps that has been quietly gaining momentum in the Rails world. That solution is Delayed::Job.

Delayed Job

DJ is an easy-as-pie plugin for Rails written by Tobias Lütke. (It can be used with Sinatra, too). It uses the database as a message bus instead of an external daemon. Using the database as the message bus isn’t as high-speed or featureful as a dedicated daemon like RabbitMQ, but it’s easier to set up, and more than enough for most sites – including Github.

John Nunemaker has an excellent tutorial on DJ, and I’ve previously illustrated the steps for building a queue-backed feed reader. So there’s plenty of material to get you going with this plugin.

DJ on Heroku

We’re pleased to announce the first iteration of support for background jobs on Heroku, in the form of DJ support. This has been in heavy beta use by a large group of Heroku beta users (thanks guys!) for the last four months, so we feel very confident that this is a solid solution, ready for real production use today.

Our first publicly-priced offering is a single DJ worker, at $15/mo. You can activate it for your app through the usual add-on interface:

$ heroku addons:add dj
Adding dj to myapp...done.

Follow the instructions for installing the DJ plugin and creating the delayed_jobs table, and you’ll be up and running in minutes. (That’s the smooth, seamless experience you’ve come to expect from Heroku.)

More to Come

If you need something beyond the publicly-priced single worker DJ, contact us. Multi-worker setup, and a more powerful message bus (RabbitMQ) are things we already have in late alpha / early beta status. If you’ve got an app that you think can put these technologies to the test, we’d love to hear from you.