There are always challenges when it comes to debugging applications. Node.js' asynchronous workflows add an extra layer of complexity to this arduous process. Although there have been some updates made to the V8 engine in order to easily access asynchronous stack traces, most of the time, we just get errors on the main thread of our applications, which makes debugging a little bit difficult. As well, when our Node.js applications crash, we usually need to rely on some complicated CLI tooling to analyze the core dumps.

Update: On closer inspection, the lock type was not on the table, but on a tuple. For more information on this locking mechanism see the internal Postgresql tuple locking documentation. Postgres does not have lock promotion as suggested in the debugging section of this post.

I maintain an internal-facing service at Heroku that does metadata processing. It's not real-time, so there's plenty of slack for when things go wrong. Recently I discovered that the system was getting bogged down to the point where no jobs were being executed at all. After hours of debugging, I found the problem was an UPDATE on a single row on a single table was causing the entire table to lock, which...

For quite some time we've received reports from our larger customers about a mysterious H13 - Connection closed error showing up for Ruby applications. Curiously it only ever happened around the time they were deploying or scaling their dynos. Even more peculiar, it only happened to relatively high scale applications. We couldn't reproduce the behavior on an example app. This is a story about distributed coordination, the TCP API, and how we debugged and fixed a bug in Puma that only shows up at scale.

Connection closed

First of all, what even is an H13 error? From our error page documentation:

This error is thrown when a process in your web dyno accepts a connection, but then...

Debugging is an important skill to develop as you work your way up to more complex projects. Seasoned engineers have a sixth sense for squashing bugs and have built up an impressive collection of tools that help them diagnose and fix bugs.

I'm a member of Heroku’s Ruby team and creator of CodeTriage and today we’ll look at the tools that I used on a journey to fix a gnarly bug in Sprockets. Sprockets is an asset packaging system written in Ruby that lies at the heart of Rails’ asset processing pipeline.

At the end of the post, you will know how Sprockets works and how to debug in Ruby.

Unexpected Behavior in Sprockets

Sprockets gives developers a convenient way to compile, minify,...