All posts tagged with kafka

How Heroku Operates its Multi-Tenant Apache Kafka Services

engineering Last Updated:

This blog post is adapted from a talk given by Ali Hamidi at Data Council SF '19 titled "Operating Multi-Tenant Kafka Services for Developers on Heroku."

Hi. Welcome to Operating Multi-Tenant Kafka Services for Developers. This is the agenda for the talk. I'm going to give you a little intro about myself and Heroku and Heroku Data. We're going to look at the motivation behind building the multi-tenant Kafka service in general, take a look at our existing single-tenant Kafka, compare that with multi-tenancy and what the multi-tenancy implications are, and then we'll go into some of the configuration changes that we made and some of the tuning that we did. Talk...

Read more

Building a Service-oriented Architecture with Rails and Kafka

engineering Last Updated: , Software Engineer

This blog post is adapted from a talk given by Stella Cotton at RailsConf 2018 titled "So You’ve Got Yourself a Kafka."

Hey, everybody. We're gonna get started. I hope that you're here to listen to me talk about Kafka, 'cause that's the room that you are in. So, yeah. First things first, my name is Stella Cotton. I am an engineer at Heroku. And like I said, I'm gonna talk to you today about Kafka. You might have heard that Heroku offers Kafka as a service. We have got a bunch of hosted plans, from tiny plans to giant plans. We have an engineering team that's strictly dedicated to doing cool stuff to get Kafka running on Heroku in super high capacity....

Read more

Managing Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku, Amazon Redshift, and Metabase

engineering Last Updated: , Senior Director, Product Marketing

Building a SaaS product, a system to handle sensor data from an internet-connected thermostat or car, or an e-commerce store often requires handling a large stream of product usage data, or events. Managing event streams lets you view, in near real-time, how users are interacting with your SaaS app or the products on your e-commerce store; this is interesting because it lets you spot anomalies and get immediate data-driven feedback on new features. While this type of stream visualization is useful to a point, pushing events into a data warehouse lets you ask deeper questions using SQL.

In this post, we’ll show you how to build a system using Apache Kafka on Heroku to manage and visualize...

Read more

SHIFT Commerce's Journey: Deconstructing Monolithic Applications into Services

engineering Last Updated: , CTO

Editor’s Note: One of the joys of building Heroku is hearing about the exciting applications our customers are crafting. SHIFT Commerce - a platform helping retailers optimize their e-commerce strategy - is a proud and active user of Heroku in building its technology stack. Today, we’re clearing the stage for Ryan Townsend, CTO of SHIFT, as he provides an overview of SHIFT’s journey into building microservices architecture with the support of Apache Kafka on Heroku.

Software architecture has been a continual debate since software first came into existence. The latest iteration of this long-running discussion is between monoliths and microservices – large self-contained applications vs...

Read more

Kafka Streams on Heroku

engineering Last Updated: , Engineer

Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system.

Unlike other stream processing systems, Kafka Streams frees you from having to worry about building and maintaining separate infrastructural dependencies alongside your Kafka clusters. However, you still need to worry about provisioning, orchestrating, and monitoring infrastructure for your Kafka Streams applications.

Heroku makes it easy for you to deploy, run, and scale your Kafka Streams applications by using supported...

Read more

Browse the blog archives or subscribe to the full-text feed.