We have concluded our investigation and want to provide our customers with an overview of the threat actor’s actions, direct mitigations we have taken because of this incident, and additional changes we will make in the face of a continually evolving threat landscape. Our incident summary outlines what we have learned during the course of our investigation starting on April 13, 2022, and ending May 30, 2022. This incident summary and numerous actions we’ve taken to add to our overall security posture is part of our ongoing commitment to maintain your trust.
On April 13, 2022, GitHub notified Salesforce of a potential security issue, kicking off our investigation into this incident. Less than three hours after initial notification, we took containment action against the reported compromised account.
As the investigation continued, we discovered evidence of further compromise, at which point we engaged our third-party response partner. Our analysis, based on the information available to us, and supported by third-party assessment, led us to conclude that the unauthorized access we observed was part of a supply-chain type attack. We are continuing to review our third-party integrations and removing any that are not aligned with our security standards and commitment to improving the shared security model.
At Salesforce, Trust is our #1 value, and that includes the security of our customers' data. We know that some of our response and containment actions to secure our customer’s data, in particular cutting off integration with GitHub and rotating credentials, impacted our customers. We know that these actions may have caused some inconvenience for you, but we felt it was a critical step to protect your data.
We continue to engage with the GitHub security and engineering teams to raise the bar for security standards. As we believe RFC-8705 based mutual TLS and private key protection for OAuth, as well as full fidelity between the Heroku GitHub OAuth integration and the GitHub App model provides more modular access privileges to connected repositories, we intend to explore these paths with GitHub.
We also continue to invest in Heroku, strengthen our security posture, and strive to ensure our defenses address the evolving threat landscape. We look forward to your feedback on both the report and our future roadmap. If you would like to offer me feedback directly, please contact me here: www.linkedin.com/in/bobwise.
Incident 2413 - Summary of Our Investigation
The following is a summary, including known threat actor activity and our responses, of our investigation into unauthorized access to Heroku systems taking place between April 13, 2022, and May 30, 2022.
Incident Summary
On April 13, 2022, GitHub notified our security team of a potential security issue they identified on April 12, 2022, and we immediately launched an investigation. Within three hours, we took action and disabled the identified compromised user’s OAuth token and GitHub account. We began investigating how the user’s OAuth token was compromised and determined that, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens.
According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the threat actor downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code. Additionally, according to GitHub, the threat actor accessed and cloned private repositories stored in GitHub owned by a small number of our customers. When this was detected, we notified customers on April 15, 2022, revoked all existing tokens from the Heroku Dashboard GitHub integration, and prevented new OAuth tokens from being created.
We began investigating how the threat actor gained initial access to the environment and determined it was obtained by leveraging a compromised token for a Heroku machine account. We determined that the unidentified threat actor gained access to the machine account from an archived private GitHub repository containing Heroku source code. We assessed that the threat actor accessed the repository via a third-party integration with that repository. We continue to work closely with our partners, but have been unable to definitively confirm the third-party integration that was the source of the attack.
Further investigation determined that the actor accessed and exfiltrated data from the database storing usernames and uniquely hashed and salted passwords for customer accounts. While the passwords were hashed and salted, we made the decision to rotate customer accounts on May 5, 2022, out of an abundance of caution due to not all of the customers having multi-factor authentication (MFA) enabled at the time and potential for password reuse.
As the investigation continued, we confirmed that on the same day the threat actor exfiltrated the GitHub OAuth tokens, they also downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. Once detected on May 16, 2022, we notified impacted customers privately on May 18, 2022, and provided remediation instructions. During this time, we placed further restrictions on token permissions, database access, and architecture changes.
Over the course of our investigation we implemented a production moratorium and disabled or rotated credentials of other critical accounts. We engaged our third party incident response partner for additional assistance on April 14, 2022. We worked with our threat intelligence partners across the industry to gain additional insight into this actor’s activity, which allowed us to expand our investigation, improve detection, and implement additional security controls that were targeted at preventing the threat actor from gaining any further unauthorized access. We engaged GitHub on an ongoing basis for information and checked for other potentially compromised assets, credentials, and tokens. We took further proactive measures, including additional credential and key rotation, re-encryption, disabling internal automation, installing more threat detection tools, and shutting down non-essential systems.
The diligent response efforts, including enhanced detection, comprehensive mitigation, and detailed investigation effectively disrupted the threat actor’s established infrastructure and eliminated their ability to continue their unauthorized access. We have continuous monitoring in place and have no evidence of any unauthorized access to Heroku systems by this actor since April 14, 2022.
Per our standard incident response process, we leveraged this incident to intensely scrutinize our security practices, both offensively and defensively, identified improvements, and have prioritized these actions over everything else.
Security Best Practices for Our Customers
In addition to the actions that have already been communicated to our customers and the additional security enhancements we are making, please keep the following best practices in mind:
- Never re-use your passwords across Heroku and other websites. Password re-use increases the probability of your Heroku account being compromised due to a security issue in another service. We suggest using password managers such as the one available in your operating system, your browser, or open source and commercial password managers.
- Enable MFA on your Heroku account to significantly reduce the probability of password based compromise. Here are some resources to help with your MFA journey:
- For guidance on setting up MFA, visit the MFA article in the Heroku Dev Center .
- Set up recovery codes so you have a backup to your primary MFA verification method.
- For additional details, bookmark the Heroku Multi-Factor Authentication FAQ. This resource is updated regularly with the latest information.
- Audit your GitHub repositories and organizations against GitHub best practices and consider enabling GitHub repository security policies when possible. Review any integration that you connect to your GitHub repositories and ensure that the integration is trusted.