The New Heroku Postgres Database Experience
August 12, 2014 by Matthew Soldo
Today Heroku is rolling out one of the most significant upgrades ever to our Postgres Database-as-a-Service. This new release is focused on a set of services that run on top of your Heroku Postgres database, making it easier to understand and operate, especially at scale. In addition, we are rolling out new production database plans with double the memory and 2-3 times the performance of our existing plans at the same cost to you.
These features represent a new experience for our Postgres service, which we collectively call DbX, for database experience. The highlight of these new features is Performance Analytics, a set of analytics and visualization tools that allow you to understand what is happening with your database and optimize its performance. In addition, we are bolstering the security of the service with the introduction of encryption-at-rest.
As with other Heroku products, we hope this focus on experience, and specifically the traditionally hard problems of operating and optimizing databases, will make building apps more productive and enjoyable.
Faster databases are a boon, but even the most powerful plans available will be crushed at scale by un-optimized queries and schemas. Running and supporting tens of thousands of production databases, Heroku has substantial experience in diagnosing and optimizing database performance. Performance Analytics is the embodiment of this experience into a simple and easy-to-use product.
Performance Analytics allows development teams to quickly and easily identify the queries that most impact database performance. We call these expensive queries. Often, simply adding the right index or more aggressively caching data can make enormous improvements to performance. But it can be difficult for teams to identify which queries are most affecting performance and worth the effort to improve. Further complicating optimization is the fact that looking at queries at a single point in time lacks the necessary context to understand the problem.
Performance Analytics makes identifying and optimizing queries easy. It presents a list of each type of query your database runs (normalized to control for differing constants). The queries are sorted by the total time they occupy in the database (their average execution speed multiplied by their volume), so the most expensive queries are shown first. Finally, seven days of trending history of both the latency and throughput are shown in an easy-to-read graph.
In addition to query optimization there are many common problems that effect databases. These include low cache and table hit ratios, high connection counts, unused indexes, bloat, and blocking transactions. These problems are now easily discoverable with the pg:diagnose command in the Heroku toolbelt:
This diagnostics tool encodes much of the database tuning expertise that Heroku’s data team has acquired from running hundreds of thousands of databases over the past five years. Using
pg:diagnose you will be able to find quick and simple ways of optimizing your database’s performance and keep your app running as quickly as possible. Read more about it in this blog post.
Continuous Protection is designed to keep your data safe, secure, and available. With the launch of Heroku Postgres DbX, Continuous Protection is gaining encryption-at-rest on all new Premium and Enterprise tier databases.
We’ve talked in the past about how Continuous Protection keeps your data safe by archiving your new data every 60 seconds. Even in the most catastrophic failures imaginable, important business data stored in Heroku Postgres is recoverable.
Continuous Protection also keeps your database available. By performing 20 diagnostics health checks to your database every 30 seconds, Continuous Protection monitors that your database is up and running to meet your applications’ needs. And if a diagnostic should fail then our automated systems repair the database, automatically.
Finally, Continuous Protection keeps your data secure. These safety measures include paranoid system configurations, automatic security patch application (often before exploits are publicly announced), regular intrusion tests, required enforcement of impossible-to-guess database credentials, and required SSL-encrypted connections from all outside clients.
Today we are bolstering the security of Heroku Postgres by encrypting data-at-rest on all new Premium Tier databases. Doing so allows Heroku Postgres to comply with the most stringent security requirements. In the exceedingly unlikely event of a physical breach of our underlying infrastructure (i.e., if someone broke into the datacenter and removed the disk drives), your data would remain safe and secure.
In addition to Heroku Postgres DbX we are launching new database plans with double the memory and speed improvements of up to 3x at the same price as our current plan lineup. These plans feature an upgraded and re-engineered infrastructure to drastically improve their memory and speed.
We are also simplifying our database plan names. While many users enjoy the Japanese themed naming scheme, the plan names are difficult to remember and aren’t ordinal. To this end we are changing the names of our production plans to a simple scheme that consists of the tier name, plus a numeral that provides an approximation of the order of magnitude of the resources available to the plan. For example Standard Yanari is now Standard0, Standard Tengu is now Standard2, and Premium Ika is now Premium4. Details of the new plans are available here.
The tier names remain unchanged - Standard, Premium, and Enterprise. The plan levels available are 0, 2, 4, 6, 7, and 8. The names of our hobby plans (dev and basic) will remain unchanged.
Heroku Postgres DbX is available today on all existing production Postgres databases (versioned 9.2 and above). The new plans with enhanced speed and memory are available for provisioning immediately. Encryption at rest is available only on newly provisioned Premium or Enterprise Tier databases.
If you need to upgrade your database to take advantage of DbX or our new plans it has never been easier. We have introduced a new tool,
pgbackups:transfer that automatically copies your database to a new one:
$ heroku addons:add heroku-postgresql:standard-4 Adding heroku-postgresql:standard-4 on pg-benchmarks... done, v11 ($50/mo) Attached as HEROKU_POSTGRESQL_BROWN_URL The database should be available in 3-5 minutes. $ heroku pgbackups:transfer DATABASE_URL HEROKU_POSTGRESQL_BROWN_URL
Whether you’re looking to get better visibility, performance, or security around your data we feel that everyone will have something they’ll enjoy in this newest release. Upgrade today to begin taking advantage of all there is in this latest release.