Heroku Postgres Followers Patched
Posted by Harold
On November 18th, a replication bug was found in Postgres that affected the most recent versions of every Postgres release. The corruption that this bug may introduce could go undetected, and it manifests itself as a follower potentially having an inconsistent view of the data. For example, data could be present in the primary and not on the follower, or data deleted or updated on the primary and not from the follower. The likelihood of triggering this bug is higher for write-heavy workloads, such as many OLTP applications seen at Heroku.
We always recommend placing applications in maintenance mode and scaling down workers when performing a follower based changeover, and following this procedure largely decreases the chances of corruption introduced by this bug.
At Heroku Postgres we recognize the importance of data integrity and consistency in your databases: As of now there are no Heroku Postgres databases vulnerable to this corruption bug. Even though new versions of Postgres have not been released yet and are scheduled to ship early December, a patch was made available to the Postgres community on all git branches for affected versions. We have back-ported this patch to all of our supported Postgres versions, and all affected instances have been replaced. As this bug only impacts followers, no primary databases were at risk and no primary databases required the patch.
If you were worried about this bug affecting your Heroku Postgres database, worry no longer: just sit back and enjoy your healthy followers.