As outlined in a previous blog post, Heroku Data services undergo routine maintenances for security and patching. In this post, we describe the process used to minimize downtime for Heroku Postgres and Heroku Key-Value Store premium ‘High Availability’ plans and how we optimized the process to perform up to 75% faster.

Data Services Architecture

High availability plans for Postgres and Redis are designed to have two database instances running at the same time. One is a writeable primary database server and the other is a read-only hidden standby. Since the standby is hidden, customers cannot access it during normal operations.

Before starting a planned maintenance, we do our best to...

Over the past few weeks, Heroku proactively updated our entire Redis fleet with a version of Redis not vulnerable to CVE-2018-11218. This was an embargoed vulnerability, so we did this work without notifying our customers about the underlying cause. As always, our goal was to update all Heroku Key-Value Store instances well before the embargo expired.

As a Data Infrastructure Engineer at Heroku, I wanted to share how we manage large fleet operations such as this one. The most important aspect of our job is keeping customers safe from security vulnerabilities, while also minimizing disruption and downtime. Those two objectives are often at odds with each other, so we work hard to reduce...