The recent introduction of Platform Events and Change Data Capture (CDC) in Salesforce has launched us into a new age of integration capabilities. Today, it's possible to develop custom apps that respond to activity in Salesforce. Whether you're creating a memorable customer interaction or implementing an internal workflow for employees, consider an event-sourced design to improve responsiveness and durability of the app.
In this article, we'll look at an event-sourced app architecture that consumes the Salesforce Streaming API using the elegant jsforce JavaScript library in a Node app on Heroku.
Streaming with jsforce
In summer 2018, the open-source jsforce library implemented a new version of its streaming module, making it a first-class consumer of the Salesforce Streaming API. This new streaming functionality is available in jsforce version 1.9.0 and later.
jsforce's stream subscriptions are based on the Faye pub/sub JavaScript module. The Faye client simplifies using Salesforce streams by handling authentication, maintaining network connectivity, and providing an extensible messaging interface. To consume a Salesforce stream, we construct a Faye streaming client from a jsforce connection, and then subscribe a callback function to receive messages from the desired topic. Let's see this in action.
Basic Example
After running npm install jsforce@^1.9.1 and configuring Salesforce authentication for the jsforce Connection, a Node app can consume data streams from Salesforce with just a few lines of JavaScript code:
const jsforce = require('jsforce');
// Establish an authenticated Salesforce connection. (Details elided)
const conn = new jsforce.Connection({ … });
// The Salesforce streaming topic and position to start from.
const channel = "/data/ChangeEvents";
const replayId = -1; // receive only new messages without replay
// Construct a streaming client.
const fayeClient = conn.streaming.createClient([
new jsforce.StreamingExtension.Replay(channel, replayId),
new jsforce.StreamingExtension.AuthFailure(() => process.exit(1))
]);
// Subscribe to the channel with a function to receive each message.
const subscription = fayeClient.subscribe(channel, data => {
console.log('topic received data', JSON.stringify(data));
});
// Eventually, close the connection.
subscription.cancel();
Using this basic example in the real-world will result in a fragile app. It will loose track of its place in the stream on every server restart. It cannot scale-up for parallel processing, because there's no coordination between each Faye client subscription to the Salesforce data stream. So, let's take a look at how we meet these challenges with a more complete example.
Real-World Example
A Node.js + jsforce Heroku app that displays a feed of Salesforce Account changes using CDC (change-data capture), an easy-to-demo use-case:
https://github.com/heroku-examples/salesforce-streams-nodejs
In order to make our stream consumer durable, the app must keep track of the last message it consumed, the “Replay ID”, and restart from that position. To make our stream consumer scalable, the app must be able to distribute messages to many parallel processes. The app uses a Redis datastore to solve these two problems. With Redis, the Replay ID for each topic can be tracked and messages distributed to many clients through pub/sub or reliable queues.
Design Considerations
As you design your app to integrate with Salesforce Streaming API, keep the following in mind.
Singleton Consumer
The Salesforce Streaming API has no provision for distributing delivery across multiple consumers (e.g. consumer groups in Kafka). So, stick to a single stream consumer process. To support scaling-up work on the incoming messages, the singleton consumer should send messages into a pub/sub channel or a reliable queue for distribution to parallel, web or worker processes.
In the example Heroku app diagrammed above, we put this architecture to work. The Procfile declares these two process types. Redis is used to coordinate these two processes. The stream_consumer process is singleton, run in a single process only, that pushes each message into Redis. The web process can scale-up to multiple clients/processes/dynos in order to handle the messages from Redis.
Authenticated User
When developing an app with Salesforce data, it's super easy to use the default Salesforce Admin user account to do everything, including API integrations. Beyond developer experimentation, always create a separate Salesforce user for API integration. That user should be easy to distinguish by name (e.g. “Follow-up Bot” or “BizOps Warehouse”) and must be given permission to the event objects utilized by the reactive app. The specific permissions required may be setup according to the documentation for Change Data Capture and Platform Events.
In the example Heroku app, the .env file contains local configuration values which might be okay as an Admin user. When you set the deployment config vars on Heroku, always use a separate, non-admin user. Note that other members/collaborators on a Heroku app can see the values of the deployment's config vars. Don't accidentally disclose Admin access to a production Salesforce org.
Flow Monitoring
Anytime external integrations are introduced into an app, consider how to detect and alert on failures. With streaming systems, a fault condition might be represented by simply nothing happening! Instrument your app to provide direct insight into streams' health, like: Salesforce Connection status, active clients, last seen message time, and message processing duration.
Stream On!
While the concept of event-sourcing is old, the implementation of this architecture is still new to many of us. As we transform more and more business workflows into IT, these kinds of lightweight, reactive apps can drive amazing experiences.
Using Node.js with jsforce on Heroku we can quickly implement Salesforce-reactive apps in perhaps the most accessible, popular programming language on the planet. Of course this architecture can be built in other languages too; Java with CometD client is featured in the Salesforce Developer docs.
Let us know what you think in the example app's GitHub issues.