Waza 2013: How Ecosystems Build Mastery

When we think of the concept of Waza (技) or "art and technique," it's easy to get caught up in the idea of individual mastery. It's true that works of art are often created by those with great skill, but acquiring that skill is neither solitary nor static. Generations of masters contribute to a canon and it is in that spirit that we built the Heroku platform and the Waza event. This year's Waza was no exception.

On February 28th, more than 900 attendees participated in Waza including Ruby founder Yukihiro "Matz" Matsumoto, Django co-creator Jacob Kaplan-Moss and Codeacademy’s Linda Liukas. True to form, we offered you a platform for experimentation and you surprised us with your contributions.

From your origami creations, to your Arduino hacks, to the technical conversations over craft beer -- you taught us that the definition of software development is ever-evolving. Thank you for allowing us to help you change lives and push boundaries. We will continue our commitment to growing the platform for you and look forward to collaborating with you in the future.

For more event highlights visit the Waza videos and photos. To learn more about Heroku, add yourself to our mailing list.

log2viz: Logs as Data for Performance Visibility


If you’re building a customer-facing web app or mobile back-end, performance is a critical part of user experience. Fast is a feature, and affects everything from conversion rates to your site’s search ranking.

The first step in performance tuning is getting visibility into the app’s web performance in production. For this, we turn to the app’s logs.

Logs as data

There are many ways to collect metrics, the most common being direct instrumentation into the app. New Relic, Librato, and Hosted Graphite are cloud services that use this approach, and there are numerous roll-your-own options like StatsD and Metrics.

Another approach is to send metrics to the logs. Beginning with the idea that logs are event streams, we can use logs for a holistic view of the app: your code, and the infrastructure that surrounds it (such as the Heroku router). Mark McGranaghan’s Logs as Data and Ryan Daigle’s 5 Steps to Better Application Logging offer an overview of the logs-as-data approach.

Put simply, logs as data means writing semi-structured data to your app's logs via STDOUT. Then the logs can be consumed by one or more services to do dashboards, long-term trending, and threshold alerting.

The benefits of logs-as-data over direct instrumentation include:

  • No additional library dependencies for your app
  • No CPU cost to your dyno by in-app instrumentation
  • Introspection capability by reading the logs directly
  • Metrics back-ends can be swapped out without changes to app code
  • Possible to split the log stream and send it to multiple back-ends, for different views and alerting on the same data

Introducing log2viz, a public experiment

log2viz is an open-source demonstration of the logs-as-data concept for Heroku apps. Log in and select one of your apps to see a live-updating dashboard of its web activity.

For example, here’s a screenshot of log2viz running against the Rubygems Bundler API (written and maintained by Terence Lee, André Arko, and Larry Marburger, and running on Heroku):

log2viz gets all of its data from the Heroku log stream — the same data you see when running heroku logs --tail at the command line. It requires no changes to your app code and works for apps written in any language and web framework, demonstrating some of the benefits of logs as data.

Also introducing: log-runtime-metrics

In order to get memory use stats for your dynos, we’ve added a new experimental feature to Heroku Labs to log CPU and memory use by the dyno: log-runtime-metrics.

To enable this for your app (and see memory stats in log2viz), type the following:

$ heroku labs:enable log-runtime-metrics -a myapp
$ heroku restart

This inserts data into your logs like this:

heroku[web.1]: measure=load_avg_5m val=0.0
heroku[web.1]: measure=memory_total val=209.64 units=MB

log2viz reads these stats and displays average and max memory use across your dynos. (Like all Labs features, this is experimental and the format may change in the future.)

Looking under the hood

log2viz is open source. Let’s look at the code:

You can deploy your own copy of log2viz on Heroku, so fork away! For example, Heroku customer Timehop has experimented with trending graphs via Rickshaw.

Logs-as-data add-ons

log2viz isn't the only way to take advantage of your log stream for visibility on Heroku today. Here are a few add-ons which consume your app's logs.

Loggly offers a web console that lets you search your log history, and graph event types over time. For example, let’s search for status=404, logged by the Heroku router whenever your app serves a page not found:

Papertrail offers search archival and history, and can also alert on events when they pass a certain threshold. Here’s how you can set up an email alert every time your app experiences more than 10 H12 errors in a 60 second period. Search for the router log line:

Click “Save Search,” then:

Other add-ons that consume logs include Treasure Data and Logentries.

You can also use non-add-on cloud services, as shown in thoughtbot's writeup on using Splunk Storm with Heroku.


Visibility is a vast and challenging problem space. The logs-as-data approach is still young, and log2viz is just an experiment to get us started. We look forward to your feedback on log2viz, log visibility via add-ons, and your own experiments on performance visibility.

Running Rails on Heroku Update

On February 16th, we published a blog post outlining five specific and immediate actions we would take to improve our Rails customers' experience with Heroku. We want to provide you with an update on where these things stand. As a reminder, here’s what we committed to do:

  1. Improve our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
  2. Remove incorrect and confusing metrics reported by Heroku or partner services like New Relic
  3. Add metrics that let customers determine queuing impact on application response times
  4. Provide additional tools that developers can use to augment our latency and queuing metrics
  5. Work to better support concurrent-request Rails apps on Cedar

We have resolved the first two items:

  1. Improving Documentation: We’ve updated our Dev Center docs and website to more accurately describe how routing occurs on Heroku (e.g., HTTP Routing on Bamboo and How Routing Works on the Bamboo Stack).

  2. Removing Incorrect Metrics: We’ve worked with New Relic to release an updated version of their monitoring tools, which we documented in Better Queuing Metrics With Updated New Relic Add-on.

We are working on the remaining three items in partnership with our early beta users:

  1. Adding New Metrics: We are improving the data available through our logs in two ways. First, we’re adding a transaction ID to every request log. Second, we are working on application middleware to provide additional metrics in customer logs.
  2. Providing Additional Measurement Tools: We have created a data visualization tool that analyzes log data in real-time and helps observe an app’s performance across a few key Heroku-specific metrics. This tool is currently being beta tested.
  3. Improving Cedar Support: Unicorn is now the recommended Rails app server. We are currently beta testing larger dynos to support additional unicorn processes.

Fulfilling these commitments is Heroku’s number one priority. We have dedicated engineering and product teams focused on improving the performance of Rails apps on Heroku.

We’ll detail the availability of our improvements in future blog posts. If you would like to beta test any of these new features, please drop us a note and let us know which features you are interested in testing.

Jacob Kaplan-Moss, Django Co-Creator, Talks Ecosystems at Heroku's Waza

The value of a network is proportional to the square of the number of users connected to the system, according to Metcalfe’s law. Jacob Kaplan-Moss, co-creator of Django, highlights this as a value in creating communities or as he puts it, “ecosystems”. In his talk at Waza last week on building ecosystems, he went on to highlight three key principles of creating ecosystems:

  1. APIs to support extensibility
  2. Conservatism as a value
  3. Empowering the community

Whether building a platform or a framework, these key principles ring true. Check out his talk or read more on creating ecosystems below:

APIs to support extensibility

You don’t choose to become an ecosystem, but you can choose to do the groundwork. By defining APIs you enable others to build on the foundation you lay. Django core member Russ Keith-McGee summed this up quite clearly on the Django mailing list – "I’m not sure if this is appropriate, but it should be possible. If its not possible we should build the APIs to support it." This sentiment is at the core of building ecosystems.

Django succeeded at this with its apps API, and has a full directory in Django Packages to help others discover what has already been built. At Heroku we’re seeing the same phenomon with our buildpacks API resulting in numerous third-party buildpacks. By building APIs and empowering users, you solve more problems.

Conservatism as a value

There’s great value in the “move fast and break stuff” concept. However, creating explicit APIs and contracts that are stable is critical for an ecosystem to thrive longer term. For Django as a framework this means being slow moving and reliable when it comes to modifying any APIs in the core framework. For Heroku as a platform it means a dedication to erosion resistance. Jacob cleanly highlights this by pointing out that “Interoperabilty only works when the common parts don’t change.”

Empowering the community

Ecosystems create ways for the community to engage and contribute. With an ecosystem there’s even additional ways in which users can contribute and those contributions don’t have to be code. When a project first starts, it’s creator tests it and consumes it acting as developer and QA. As it gains popularity a project may attract new developers, then more users, who submit their own bug reports and issues. These newcomers contribute documentation patches, triage tickets, and recruit others to help with the project. Where once a single developer committed quietly, now a thriving ecosystem of contributors and consumers exists.

This change from product to ecosystem doesn’t happen overnight, it must be nurtured and nudged. Kaplan-Moss, a self-proclaimed “documentation nut” has done just that by cultivating a better understanding of Django through documentation and carefully working with patch and issue submitters. Jacob explains that if you document your process, you can increase transparency and understanding of the way things work. Django does this by clearly documenting how users can contribute back to Django; at Heroku we’ve done this by documenting our methodology on how apps should be built.


Building a great product isn’t enough, as Jacob points out: you’ve got to create a vibrant community and build an ecosystem. If it wasn’t for Django’s community there wouldn’t be over 3500 Python packages for Django. If it wasn’t for our community, you wouldn’t be able to deploy Go, common lisp, or GeoDjango to Heroku with just one command.

Learn more about ecosystems by listening to Jacob’s talk, check out other talks from Waza available now, or engage with the ecosystem of your choice today.

Matz on Ruby 2.0 at Heroku's Waza

Matz, the creator of Ruby, spoke at Waza for the 20th anniversary of the language and the release of Ruby 2.0. If you weren't in the sold out crowd, not to worry. Information should flow free and experiences should be shared; in line with those concepts you can watch Matz's talk right here, then read about what's new in this version of Ruby and how to run it on Heroku.

With slides available on speakerdeck

Keep reading for more information on Ruby 2.0 or check out our first batch of videos from Waza 2013. To stay up to date as we post new videos, follow us @heroku.


Iterating quickly means happier developers and happier customers. We optimize our platform to allow you to develop, stage, and deploy faster. Not only does this make running software easier, but it makes trying new technologies like Ruby 2.0 as simple as spinning up a new app. During Matz's talk at Waza, he mentioned that, while 1.9.3 is popular now, it took years after 1.8.7 was released to gain traction. With the release of Ruby 2.0 Matz hopes to reduce upgrading barriers and allow developers to iterate quicker using newer, faster and better tools.

Ruby 2.0 was written to be backwards compatible and it works with Rails 3.2.13 out of the box. If your Ruby apps are running using 1.8.7, you should upgrade. Ruby 1.8.7 is approaching End of Life (EOL) in three months on June 2013. EOL for Ruby 1.8.7 means no security or bug patches will be provided by the maintainers. Not upgrading means you're potentially opening up your application and your users to vulnerabilities. Don't wait till the final hour, upgrade now to be confident and secure.


Ruby 2.0 has a faster garbage collector and is Copy on Write friendly. Copy on Write or COW is an optimization that can reduce the memory footprint of a Ruby process when it is copied. Instead of allocating duplicate memory when a process is forked, COW allows multiple processes to share the same memory until one of the processes needs to modify a piece of information. Depending on the program, this optimization can dramatically reduce the amount of memory used to run multiple processes. Most Ruby programs are memory bound, so reducing your memory footprint with Ruby 2.0 may allow you to run more processes in fewer dynos.

If you’re not already running a concurrent backend consider trying the Unicorn web server.


In addition to running faster than 1.9.3, and having a smaller footprint, Ruby 2.0 has a number of new features added to the language including:

The list of new features is more than we can cover here. If you really wanted to dig in you can check the Ruby changelog

Running 2.0 on Heroku

If you’re interested in taking advantage of these new features give it a try on Heroku today. To run Ruby 2.0 on Heroku you'll need this line in your Gemfile:

ruby "2.0.0"

Then commit to git:

$ git add .
$ git commit -m "Using Ruby 2.0 in production"

We recommend that you test your app using 2.0 locally and deploy to a staging app before pushing to production. Now when you $ git push heroku master our Ruby buildpack will see that you've declared your Ruby version and make sure you get the right one.

20 years of simplicity, elegance, and programmer happiness

Heroku, since its founding, has been aligned with the key values of Ruby – simplicity, elegance, and programmer happiness. Heroku still believes in the power and flexibility of Ruby, and we've invested in the language by hiring Yukihiro "Matz" Matsumoto, Koichi Sasada and Nobuyoshi Nakada. We would like to thank them and the whole Ruby core team for making this release happen. Join us in celebrating Ruby's successes and in looking forward to the next twenty years by trying Ruby 2.0 on Heroku today.

Browse the blog archives, subscribe to the full-text feed, or visit the engineering blog.