How we built Heroku CI: our product intuition checked against what the market wants (we surveyed ~1000 developers to figure out the latter, and the results were surprising)
Two approaches to building any product are often in tension: designing from inspiration, and designing from information. On the pure inspiration side, you just build the product you dream of, and trust that it will be so awesome and useful, that it will succeed in the market. On the pure information side, you build exactly what the market is asking for, as best you can tell (think: surveys, top customer feature requests, consultants, customer panels).
Our initial design for Heroku CI (currently in public beta) was nearly pure inspiration, in large part from our engineering staff. We had a good idea of our dream CI product, and many of the raw materials to build it (Heroku apps, our build process, Heroku add-ons, etc.). And Heroku, like the rest of Salesforce, strongly encourages such experimentation and innovation when individual product teams are so inspired. We create and improve a lot of amazing products that way.
The Heroku DevEx (Developer Experience) team is distributed across two continents and four countries. We need well-structured, team-oriented developer workflow as much as our users and customers, including, of course, running tests for continuous integration (CI). And we are a pretty experienced team of developers, product, and design. We spend a lot of time with customers, and, of course, live and breathe in today's app dev and DevOps world.
So the Heroku CI design began with our own well-formed and granular ideas on what our customers wanted out of CI, and we quickly arrived at what turned out to be a pretty stable initial design.
Our alpha launch had Heroku CI integrating with, and extending, Heroku's continuous delivery (CD) feature set: Heroku Pipelines, Review Apps, and Heroku GitHub deploys. The initial feature set and developer experience were inspired largely from our own intuitions, experiences, and conceptions of customer need.
Once we had a limited-functionality alpha version (MVP of our pre-conceived design for Heroku CI), we tested it over several weeks internally, then invited Heroku users via Twitter and direct e-mail to try it out.
For users to access this "limited public beta" CI feature, they had to complete a survey on how they use CI and what they want from CI. We wanted to determine how they use (and hope to use) CI, and also wanted to make sure those interested could actually use the new Heroku CI alpha -- e.g. that we supported the language they required.
As it turned out, so many users responded -- about 1000 for most questions -- that our short survey has become the largest of cloud PaaS developers on CI in recent years. The data helped us shape the planned CI feature set for public release. We also think the survey results will help the dev tools community understand how developers want to use CI, and so we are making the full results available in this post.
The survey covers customers' responses on how they want to use CI, and how it fits into their CD and team workflows.
You can read elsewhere about using Heroku CI. And you can just try it out yourself: the feature is in public beta and available by default to all Heroku users (just check the Heroku Pipelines web interface). Right now let's talk about how we decided what to build.
Some customer concerns initially seemed obvious, but we asked anyway: of course we found that CI users want shorter queue times, faster test runs. Some features were not as predictable. And some of what's been most successful in our private beta was not requested by surveyed or otherwise vocal users at all.
Almost a year ago, at the Heroku DevEx team offsite, we met up one morning at a London coffee house, the café at the Hoxton, Shoreditch. An informal conversation on what features/products to build next turned to CI. There had been customer requests to be sure. And it seemed to us like CI would be a great addition to our Heroku Flow continuous delivery features. Most of all, we just wanted to build it. CI seemed to us so compelling for our users as an integrated feature, and so compelling to us as a very cool thing to build, that there was soon consensus on the team. I'll note here that even at a biggish company like Heroku our team excitement around building cool stuff that users will love has a lot of influence over what we build. We didn't do a ton of financial analysis, or call our big-firm Analyst. We, the Developer Experience Team, wanted to build CI; there was roughly adequate customer and strategic reasons to do it, and we had the collective excitement and motivation to give it a shot. Personally, it’s gratifying to me that software can be built that way at Heroku and Salesforce.
The Hoxton café, by the way, is a fabulous place. You can stay there all day, and we did, occupying valuable retail space and working out what Heroku CI might look like: UX, features, integrations, and above all, how the feature will live up to the Heroku promise of being (all at one now) simple, prescriptive, and powerful. Thankfully they serve breakfast, lunch, and a very nice dinner.
CI as a practice in software development has been growing relentlessly over the past decade (even as interest in software development as a holistic discipline has stagnated). "Software development" is increasingly many separate disciplines, each with its own constituency and community -- the DevOps segment alone is projected to be a nearly $3 billion market in 2017. And developer focus on DevOps has given rise to many great products in the CI market. We wanted to understand how we could help developers have even better CI experiences, get better productivity, and build better products of their own.
Our spike on the idea of a "Heroku CI" was essentially a simple test runner (for Ruby and Node) with a nice-looking UI integrated into the Heroku Pipelines interface. This spike, plus a nice UI and some basic supporting features, constituted our the version we released to alpha. Being Heroku we had a lot of the parts we needed to assemble test environments (ephemeral apps, dynos, add-ons, a build system, …) just lying around, and lots of co-workers willing to try out what we built.
Quite quickly, we had an initial CI test-runner bolted onto the existing Pipelines feature. We found it useful to us internally and we felt comfortable inviting a few dozen friends-of-Heroku to use it and give feedback. Within a few more weeks, and a few dozen iterations, we were ready to ask a broader audience to weigh in (and take the survey).
Yes, CI is integrated with Heroku Pipelines, and in fact they share a UX. While we could separate these in the future, the integrated offering is proving popular with users from big enterprise to small teams. We think this is in large part because it’s integrated.
There was a time when "best of breed" was the catchphrase in DevOps. You spun together VMware, Chef, Puppet, Jenkins, Git, etc. So why are integrated offerings popular now? Our thinking is that CI/CD go together like chocolate and peanut butter: it’s a bit messy to put together yourself, but great when someone packages it for you. We noticed that our users, in general, enjoyed using CI products that set-up and integrated easily with Heroku Pipelines (great products like Travis and Circle). The popularity of fully integrated CI offerings wasn't lost on us either (look at the popularity of GitLab's integrated CI, or Atlassian's increasingly integrated devtools products). Among other advantages, you get single auth, add-ons work automatically, and there's no context switching from product to product in use or administration.
As importantly, we've noticed developers are demanding better UX for dev tools in general, and CI in particular (think of the delightful interface of Snap CI and others). We also noticed the pain many users described in setting up Jenkins, which leads the CI industry, but comes with a dizzying array of complex options, plug-ins (1500+), and a typically long set-up time, and maintenance labor. An extreme but real example here: one large Heroku customer needs to open a ticket with its IT department to provision each new Jenkins instance, often adding two months to the start-up of a new project if they want to use CI. This company is now using integrated Heroku CI on some projects, reducing CI setup time from months to minutes.
Developers, statistically, are centering around a few preferred workflow stacks. GitHub and Slack are the dominant VCS and team chat[ops] tools by most current market measures -- strong integration with these seemed necessary for us, and for any DevOps product. Atlassian's Bitbucket, HipChat are each a solid second in these roles. Together with Trello this would seem to give Atlassian a significant enough center of gravity (especially at larger companies) to also require good integration. And GitLab is surging in VCS and CI (and in developer buzz), at a curve that seems poised to challenge much longer-standing products. Being part of these popular toolchains is important to us, and where there is overlap in features, developers can choose where they want to live on the way to deployment and delivery.
Note that, as Heroku is a cloud platform, our user base tends to prefer cloud products. We know also that a significant proportion of our customers who use "on premise" version control options like GitHub Enterprise and Bitbucket Server are hosting them in the cloud, often on AWS. So even a portion of these on-prem components are running as a self-managed cloud-like service.
Jenkins is the leader in CI market share (around 70% by most analyst estimates). And its usage is growing at a pretty good clip. The vast majority of Jenkins installations are on-premise (about 83%) Our customers are always in the cloud, deploying to either Heroku's Common Runtime -- our traditional secure public cloud Heroku -- or to Heroku Private Spaces, our virtual private PaaS cloud offering.
So for the cloud customers, as in our survey, the CI product usage sizes out a bit differently:
I noted earlier the popularity of GitLab's integrated CI being significant to us in our decision to integrate CI. GitLab's been really good at moving fast in bringing popular, highly integrated features to market, building on its popular VCS. Note here that GitLab CI it is clearly the biggest mover in activity on Stack Overflow among popular cloud CI solutions.
But all these cloud CI solutions are still dwarfed by Jenkins (as noted above, only 17% of Jenkins installs are in the cloud).
Interestingly, Jenkins’ lovely new "Blue Ocean" UX, courtesy of CloudBees, seems to underscore the growing importance of simple, prescriptive developer experience in modern CI/CD solutions -- something that has always been, and is still, job 1 at Heroku.
You'll notice in the first chart this post that fast queue time and fast execution time are the top most important features for surveyed users other than price.
We have eliminated queue time altogether with Heroku CI. A new ephemeral CI environment (oh: it’s a Heroku app) is created at the beginning of each test run, and then destroyed when tests are complete. While there is some nominal setup time here, it’s nothing like traditional queue time waiting for a free CI worker.
Note that at the chart at the top of the post, only about 40% of respondents found production environment parity to be "important". When we initially conceived of Heroku CI, we thought one of the biggest selling points would be environment parity between test runs environments (apps) and staging and production apps (environments). To our surprise, users did not care as much as we thought they should.
This fact led us to an innovation around add-ons for ephemeral environments like CI and review apps. We now inform add-on partners that their add-on is being provisioned for an ephemeral Heroku app (a CI app in this case). During provisioning, the add-on vendors can choose to eliminate slow/costly features of the add-on that might take a long time to spin up, but are usually unnecessary for apps that will exist for such a short time – like long-term logging, billing features, etc.
In this way, we are working across the ecosystem of cloud-based services that developers use to make sure each component is automatically configured for CI.
We did make sure that you can customize CI apps in the
environments section of the newly revised Heroku app.json manifest. Customizability, when we can do it in a way that doesn't complicate, is as important as being prescriptive.
Re-running tests was not part of initial CI alpha, and we were somewhat surprised by the high number of people who requested it both in the initial survey and among alpha users. So there’s now a big "Run Again" button on the UI, and we do find this feature frequently used in the current public beta.
Browser testing – often called user acceptance testing or UAT – was popular with surveyed users, so it was moved up in our list of features planned for launch. UAT in a browser is also required by many Salesforce developers, whose applications are often web-deployed and/or exist within the Salesforce interface. (note that Heroku CI will also be used by Salesforce developers using Apex, hybrid-language apps, or frameworks like Lightning.)
Product design by inspiration is exhilarating, and it's great to see a product or feature that arises from the sheer excitement of the team to build it succeeding with users. Verifying that our intuition was right with users takes a lot of time, but it's well worth it. In short what worked for us was to trust our instincts, but verify with users (and adjust).
Most importantly, we view the developer community as a diverse ecosystem of innovators, thought leaders, vendors, and open source efforts, among others. It's a pleasure to share what we know, learn, and create with our vibrant, diverse community.
Let us know what you think of Heroku CI, and what you want in your CI solutions, at email@example.com