The data we store holds value, but refining data into meaning remains a difficult task. Over the last few months, we've taken a step back to figure out what we can do to help our users cross that divide, and rebuilt Heroku Dataclips from scratch with that goal in mind. The result is an experience that makes accessing and working with your data easier than ever, enabling anyone on your team familiar with SQL to take advantage of your most valuable asset without the need for specialized tools or knowledge of the database.
Dataclips is a flexible, lightweight way to query your data in Heroku Postgres and share the results. At Heroku, we use them regularly across all of our departments. Engineers create dataclips to diagnose issues and produce actionable lists. Our Business Operations team crafts evergreen reports on the health of our business, enabling our product teams to prioritize work. Product managers investigate user behavior and monitor feature uptake. Everyone uses dataclips as a rock-solid conduit for getting data into other tools. They are a window into systems of record — for many of our Heroku Postgres customers and us.
How could we democratize access to the most valuable asset of every business? We investigated how our users used (and abused!) dataclips as part of their daily routines, and came away with the following observations:
- Data grows more opaque with time. As data grows and schemas evolve, knowing what to query and how to query can become a barrier of institutional knowledge. It's a hurdle for experienced veterans and new employees alike, preventing them from making data-driven decisions.
- A dialogue with data requires quick iterations. If users need to ask many questions on their path to the right question, then those questions need to be quick to write.
- Dataclips are for sharing. Asking the right question is only half the battle. How we communicate the answers to others depends very much on who that audience is. Are they a colleague? A manager? A customer? How could we improve the presentation of data, presenting the essential with a minimum of noise? Maximizing our data-to-ink ratio is not only healthy for visualizations but applies to our interfaces as well.
- Data pipelines often start with Dataclips. We have customers using Heroku Dataclips as a lightweight ETL framework, performing complex aggregations as the first stage in their existing data workflows. We can't imagine covering all of these use cases. How could we make it easier for our customers to get data out of a dataclip and into their tools?
With these themes in mind, we set out to take what was already great about Dataclips and make it better. Let's take a look at some of the highlights in this new release:
Introducing the New Dataclips
Authoring
We want it to be faster to write queries, and easier to figure out how to query. The new schema explorer is one of our favorite things in this release — everyone on your team who can edit the dataclip can now quickly see all of the tables, the associated columns for each table, and their types. In addition, the editor supports autocomplete for SQL, Postgres functions, and your table names. Taken together these features make writing queries more like writing code in your favorite editor, and provide a significant productivity boost by allowing you to focus on the right questions to ask of your data versus worrying about syntax and the names of your columns. We are also particularly excited about the ability to save private drafts of queries and, when ready, publishing them for anyone with datastore access to see.
Sharing
The previous one-size-fits-most sharing approach for dataclips worked well for some use cases, but users asked us for a sharing model that gave them more fine-grained control over who can see a dataclip’s query separately from who can see the results. Now you can share the read-only results of a dataclip with specific Heroku users, entire Heroku Teams, or publicly without exposing the underlying query or datastore information. Published queries will remain editable by any Heroku user with sufficient access to the clip’s database. For more information on Dataclips access and visibility, see the Dev Center article.
Viewing Results
Say goodbye to waiting for data to refresh. We've made improvements to how we execute clips so you’re looking at the freshest data. While we have always executed clips that you have open in your browser once a minute, we’ve introduced background execution in this release. For any clip that has been accessed in the last week, we automatically refresh the clip once an hour. Now you and the tools you use see current data, so you can make confident, rapid decisions.
Dataclips have always been a great way to share results in tabular form; now you can instantly visualize time series data—results containing a date or timestamp column and a column of numeric data—as a bar or line chart right from within the comfort of the Dataclips UI. We’ve also made lots of little touches to the UI such as sticky column headers, and better responsive behavior, to speed up your dialog with data.
Customer Success Spotlight: Busbud
Heroku customer Busbud provides its users with the fastest experience to book bus tickets online. Data is at the core of their business; the new Dataclips has allowed them to increase accessibility to their data and make better decisions:
“Dataclips allow our employees to explore, mashup and consume our data, and they've become a staple in performance reports, experiment analysis and discussions. But they were almost exclusively authored by a technical subset of our employees.
The new changes have made writing dataclips even more accessible - we now have people across the company excited to learn SQL so they can explore the data they need to make great decisions every day. The new schema explorer and autocomplete features really make it easier to write dataclips and the charting features allow you to understand the data at a glance. The draft and published features allow our employees to collaborate on dataclips to make sure they get them right before including them in the catalog - I'm excited to continue using it with my team to help them improve their SQL skills and increase their autonomy.” —Mike Gradek, Co-founder and CTO, Busbud.com
Get Started
The Dataclips interface now lives in the same web dashboard as all of our other data-related features: http://data.heroku.com/dataclips. To try out the new Dataclips, simply select Dataclips
from the tab menu.
We’ve also teamed up with the good folks at Code for America so you can experience the new Dataclips with real-world data using their Open311 Data Set. See some initial results (and dig into some of the data yourself). Also, check out our podcast channel on April 18, 2019, for a deeper dive into the making of the new Dataclips.