Heroku Postgres is Heroku’s database-as-a-service product. With Heroku Postgres, you can easily provision and scale a Postgres database for your Heroku application, or as a stand-alone service. Recently, we’ve blogged about PostgreSQL 9.3 on Heroku and how you can use Heroku Postgres dataclips to build awesome business dashboards with your data. In this post, we talk about how Heroku Postgres delivers continuous protection for your business data using WAL-E, an open source application for archiving PostgreSQL WAL (Write Ahead Log) files quickly, continuously and with a low operational burden.
About Continuous Protection
In order to protect customer data and be resilient to failure modes, we provide all Heroku Postgres databases with continuous protection by replicating data to external storage. We store a base backup of your database as well as a log of all committed transactions in the form of WAL files - the standard method for ensuring data durability in Postgres. This means we can restore your database by fetching the base backup and replaying all of the WAL files on a fresh install in the event of hardware failure, data corruption or other failure modes.
In addition to providing recovery functionality, this methodology lets us expose Heroku Postgres features that make working with your database faster and more flexible. The same mechanism underlies our fork feature, which lets you immediately create a perfect, byte-for-byte clone of your database for easier database migrations, load testing, and creation of development databases. It’s also how we let you spin up followers - read-only replicas of your database you can use for scaling reads, manual failovers and business reporting with no impact to production. These features are used far more often than restoring for disaster recovery, with the added benefit that they’ve let us battle-test the shared underlying mechanism so we can ensure success when disaster does hit. For more information on continuous protection on Heroku, check out the Dev Center article.
Introducing WAL-E
In addition to using tons of open source software here at Heroku, we are also serious and passionate about contributing code back to the community. WAL-E is one such open-source project that performs continuous, automatic archiving of WAL files to S3 across our entire fleet of databases. Initially developed by our resident tuple groomer Daniel Farina, you can now find it on GitHub. WAL-E is quickly becoming the default option for those running PostgreSQL on top of Amazon Web Services, with companies like Instagram using it to perform point-in-time restorations and quickly bootstrap a new read-replica or failover slave and Amazon themselves recommending it.
We wrote the initial version of WAL-E because we needed a programmatic, efficient way to ensure the safety of databases at scale across our entire user base. One of the major requirements was that the software would be easy to set up and administrate - as a result, WAL-E users can generally get up and running in about 15 minutes. In fact, WAL-E only has four operators: backup-fetch and backup-push, which retrieve and send, respectively, base backups; and wal-fetch and wal-push, which send and retrieve the WAL files themselves.
If you use Heroku Postgres, WAL-E is already hard at work for you as the underlying mechanisms for fast and easy disaster recovery, forks, and followers. Those using WAL-E for a private Postgres installation can set it up to archive logs from every few minutes to once a day, depending on their business needs.
Future of WAL-E
WAL-E is at the heart of our continuous protection we provide for our customers, and will be around for some time to come.
We actively maintain WAL-E and accept pull requests, issues and feedback on GitHub. If you’re interested in contributing we’d love to hear from you.
For more on Heroku Postgres and continuous protection, check out the Dev Center article, which also talks about logical backup options with Heroku Postgres.
Special Thanks
Finally we want to offer a special thanks to contributors of WAL-E to date: