For quite some time we've received reports from our larger customers about a mysterious H13 - Connection closed error showing up for Ruby applications. Curiously it only ever happened around the time they were deploying or scaling their dynos. Even more peculiar, it only happened to relatively high scale applications. We couldn't reproduce the behavior on an example app. This is a story about distributed coordination, the TCP API, and how we debugged and fixed a bug in Puma that only shows up at scale.

Connection closed

First of all, what even is an H13 error? From our error page documentation:

This error is thrown when a process in your web dyno accepts a connection, but then...

Setting up a database is a relatively straightforward process (Heroku has an add-on for that), but getting it to run well in production is sometimes another matter. As your application grows and your data grows along with it, you will likely find a number of performance bottlenecks specifically related to your database, and this post aims to help you diagnose and address those issues when they arise.

As with all components of your infrastructure it’s important to have early visibility into the performance characteristics of your database. Watching this data as your application grows will give you a much better chance of spotting performance issues and regressions as they’re introduced.

I...

Heroku has many public API endpoints. Each of these endpoints needs to be tested so that we know how they work, and documented so that our customers (and other API consumers) know how they work. Follow along, and we’ll learn how Heroku uses JSON Schema to test and document our Platform API – and how it helped us uncover an unexpected bug, rooted in the way the Oj gem parses Big Decimals.

JSON Schema files are like blueprints that define the structure and semantics of other JSON documents. When a JSON Schema file is applied to a JSON document, you can determine whether the document is valid (conforms to the schema) or is invalid (does not conform to the schema).

So how do we at Heroku use...

This blog post is adapted from a talk given by Amy Unger at RailsConf 2018 titled "Knobs, buttons & switches: Operating your application at scale."

We've all seen applications that keel over when a single, upstream service goes down. Despite our best intentions, sometimes an unexpected outage has us scrambling to make repairs. In this blog post, we'll take a look at some tools you can integrate into your application before disaster strikes. We'll talk about seven strategies that can help you shed load, fail gracefully, and protect struggling services. We'll also talk about the technical implementations of these techniques—particularly in Ruby, though the...

Debugging is an important skill to develop as you work your way up to more complex projects. Seasoned engineers have a sixth sense for squashing bugs and have built up an impressive collection of tools that help them diagnose and fix bugs.

I'm a member of Heroku’s Ruby team and creator of CodeTriage and today we’ll look at the tools that I used on a journey to fix a gnarly bug in Sprockets. Sprockets is an asset packaging system written in Ruby that lies at the heart of Rails’ asset processing pipeline.

At the end of the post, you will know how Sprockets works and how to debug in Ruby.

Unexpected Behavior in Sprockets

Sprockets gives developers a convenient way to compile, minify,...