Logs tell the story of your app - a continuous, living stream of events, changes and behaviors. Logs let you rapidly identify and act on critical events, debug issues in your code, and analyze trends to make better decisions over time.
But log management is increasingly complex. As apps scale across distributed infrastructure, many independent processes must be tracked and made sense of. Numerous components and backing services each produce their own log streams. Multiple developers may be collaborating on your app, and multiple services must consume its logs. And logs must be useful not only to machines and applications, but to the humans viewing them.
Heroku brings simplicity and order back to logging. Heroku automatically collates and routes logs from every part of your app into a single channel, providing truly comprehensive, extensible, app-centric logging.
Your log stream comes with rich command line functionality, is easy to plug into other services, and handles the heavy lifting of log management for you. This post presents Heroku's distributed logging platform, shows how to execute basic functions, and introduces logging services available with Heroku Add-ons. Finally, we take a look at two new, experimental logging tools available in Heroku Labs.
Logplex: An Open Source, Distributed Logging Platform
Logplex is Heroku’s distributed log routing and collation platform. Logplex uses a publish/subscribe model, merging and redistributing multiple incoming streams from various application components to individual subscribers. It’s open source, written primarily in Erlang, and relies on standard protocols and formats including syslog, stdout and HTTP.
Logplex collects underlying events from the Heroku platform, API logs with administrative actions performed by you and your collaborators, and output from within your app, app server, installed libraries and any backing services that have been configured to publish to your stream. The result is a full story of your application - logs from every piece of the Heroku platform, each component of your app, all of its processes, and all changes made to it by you or your teammates.
Working with Logs
timestamp source[dyno]: message
Logs on Heroku are designed to be human-readable, with an easy-to-parse format. Logs on Heroku consist of a timestamp, source, the name of the dyno that wrote the log, and the message.
Anything written to stdout or stderr will automatically be routed and collated by Logplex.
$ heroku logs --tail
Run an open session to collect incoming logs right in your terminal, providing insight into live behavior as it happens for rapid debugging.
$ heroku logs --source app
Filter down to an individual source or dyno, i.e. app logs, logs from a particular dyno, or logs from the Heroku platform itself using simple commands like --source app (filters to only app logs) or --source heroku (filters to system logs such as crash processes, error pages, dynos coming up and down, etc). You can even filter to logs from specific dynos using --ps filtering arguments.
Extensibility and Add-ons
$ heroku addons:add papertrail
Logplex maintains the last 1,500 lines of consolidated logs for your app and can be easily plugged into external services for long-term storage, monitoring, alerting and analysis. The Heroku Add-ons platform offers fully-managed tools including Loggly and Papertrail that automatically integrate with your Heroku logs and can be added to your app in a single command. You can also set up your own log drains so you can forward logs to any external syslog server.
Logging for Teams
2013-07-06T12:00:01+00.00 heroku[api]: Release v3 created by email@example.com
Heroku makes it easy to collaborate with other developers and members of the application team simply by adding them to an app. Heroku logging will not only display API logs, but which collaborator initiated commands, releases and other changes. Want to keep an audit log of all changes ever made to your app? Once you’ve set up an Add-on for log persistence, use it to search and filter all historical, developer-initiated events. Got lots of people working on your app at the same time? Open a logs --tail session and see all activity going on in the app in real time. Heroku logging supports team-wide visibility and makes it easier to construct the full picture of your app, narrow in on critical events, and identify problems or changes quickly.
Labs
Heroku Labs lets you enable and test experimental features on the platform. Here are two Labs features around logging we’re currently piloting. Please remember that Heroku Labs features are experimental only and may be changed or removed without notice.
$ heroku labs:enable log-runtime-metrics
Enabling log-runtime-metrics for myapp... done
$ heroku restart
log-runtime-metrics is a Labs feature that will give you in-depth per-dyno stats, including memory use, swap use, and load averages. These stats will appear as part of your app’s normal log stream. For additional information, check out the Dev Center article.
$ heroku labs:enable http-request-id
http-request-id lets you correlate router logs for a given web request against the web dyno logs for that same request. With a simple search of stored logs, look up events or messages correlated with a unique request ID. This helps you find the source of request errors, see how requests and application code may be resulting in behaviors at lower levels of the system, and get greater visibility into how various elements of your app are interacting.
Better Logging for Better Development
Logplex helps manage and deliver millions of messages every minute, managing the complexity of distributed logging so you can focus on making better decisions, solving problems fast, and using awesome services. Email us at labs@heroku.com to let us know what you think of log-runtime-metrics and http-request-id, and what you want to see next from Heroku logging.