Heroku Connect makes it easy to sync data at scale between Salesforce and Heroku Postgres. You can build Heroku apps that bidirectionally share data in your Postgres database with your contacts, accounts, and other custom objects in Salesforce. Easily configured with a point-and-click UI, you can get the integration up and running in minutes without writing code or worrying about API limits. In this post, we introduce our recent improvements to Heroku Connect on how we handle drift and drift detection for our customers.
PensionBee, the U.K.-based company, is on a mission to make pensions simple and engaging by building a digital-first pension service on Heroku. PensionBee’s consumer-friendly web and mobile apps deliver sophisticated digital experiences that give people better visibility and control over their retirement savings.
PensionBee’s service relies on a smooth flow of data between the customer-facing app on Heroku and Salesforce on the backend. Both customers and employees need to view and access the most current account data in real time. Heroku Connect ensures all of PensionBee’s systems stay in sync to provide the best end-user experience.
Understanding Data Drift
Heroku Connect reads data from Salesforce and updates Postgres by polling for changes in your Salesforce org within a time window. The initial poll done to bring in changes from Salesforce to Postgres is called a “primary poll”. As the data syncs to Postgres, the polling window moves to capture the next set of changes from Salesforce. The primary poll syncs almost all changes, but it's possible to miss some changes that lead to “drift”.
Heroku Connect does the hard work of monitoring for “drift” for you and ensures the data eventually becomes consistent. We have now increased the efficiency of this feature to recognize and address drift detection even faster on your behalf. As before, this process is transparent to you; however, we thought our customers might enjoy understanding a bit more about what is going on behind the scenes.
There are several complications in ensuring that the data sync between the two systems is performant while being reliable. One complication is when Heroku Connect polls a Salesforce object for changes, and a long-running automation associated with record updates doesn’t commit data at that time. When those transactions are committed, the polling window could have already moved on to capture the next set of changes in Salesforce. Those missed long-running transactions result in drift. Heroku Connect handles those missed changes seamlessly for its customers.
Drift Detection: Ensuring Data Accuracy and Consistency
Heroku Connect tracks poll windows for each mapping while retrying any failed polls. Drift detection uses a “secondary poll” to catch and fix any changes the primary poll missed. Heroku Connect tracks the poll bounds of the primary poll and schedules a secondary poll for the same poll bounds after some time. Depending on the size of the dataset the primary poll is synchronizing, Heroku Connect uses either the Bulk API or SOAP API for polling. Heroku Connect leverages Salesforce APIs without impacting your API usage limits and license.
With the Bulk API, Heroku Connect creates a bulk job and adds bulk batches to the bulk job during the primary poll. Heroku Connect tracks the poll bounds for each bulk batch, and then performs a secondary poll corresponding to the poll bounds for each bulk batch in the primary poll. During the secondary poll, Heroku Connect creates a bulk job for each bulk batch processed by the primary poll. Sync using Heroku Connect is asynchronous with retries, so it isn’t real-time, though it appears to be.
Scale and Performance Improvements
As Heroku Connect serves more customers with increasingly large mappings, we continue to ensure we provide a scalable, reliable, and performant solution for our customers. One of the areas where we made significant improvements is the way we manage and schedule secondary polls for drift detection, especially for polls that use the Bulk API.
Reduced load on the Salesforce org
In the old process, the secondary poll created a large number of bulk jobs in Salesforce. Now the secondary poll only creates a single bulk job for each bulk job created by the primary poll. Then, for each bulk batch processed by the primary poll, a bulk batch is added to the secondary poll’s bulk job.
Optimized management of the secondary poll
Previously, there was no limit on the number of bulk tasks processed by the secondary poll at a time. As primary bulk batches completed, any number of secondary bulk tasks were scheduled and executed simultaneously. Now Heroku Connect schedules and executes secondary polls so that there’s limited bulk activity at a time. This helps with:
- Improved availability of database connections: Heroku Connect opens database connections as it updates data in Postgres from Salesforce. With an unlimited number of simultaneous secondary poll tasks, Heroku Connect opens a large number of database connections, leaving fewer connections for your applications accessing the same database. By limiting secondary poll tasks and scheduling them in a controlled way, Heroku Connect uses a much smaller number of database connections at any given time, allowing your applications enough connections to work with.
- Improved operational reliability: Our optimizations in scheduling secondary polls enhance the overall performance, ensuring that even during heavy sync activities, the quality of service remains high for all users sharing the underlying infrastructure.
Conclusion
At Heroku, we take the trust, reliability, and availability of our platform seriously. By investing in projects such as improving drift detection, we’re constantly working to improve the resilience of our systems and provide the best possible experience so our customers like PensionBee can continue to rely on Heroku Connect to keep their data in sync. Thank you for choosing Heroku!
If you have any thoughts or suggestions on future reliability improvements we can make, check out our public roadmap on GitHub and submit an issue!
About the Authors
Siraj Ghaffar is a Lead Engineer for Heroku Connect at Salesforce. He has broad experience in distributed, scaleable, and reliable systems. You can follow him on LinkedIn.
Vivek Viswanathan is a Director of Product Management for Heroku Connect at Salesforce. He has more than a decade of experience with the Salesforce ecosystem, and his primary focus for the past few years has been scalable architecture and Heroku. You can follow him on LinkedIn.