API Key Incident Report (May 2022)

On May 11, we found that ReadMe API tokens were embedded in project login pages. After diagnosing the underlying issue, we released a fix later that day. On May 24, a security researcher noticed the login pages' content had been cached on the Internet Archive and was able to find the exposed tokens and verify that they were still valid. We opened a follow-on incident immediately and began triage and mitigation work. We chose to invalidate and rotate all ReadMe tokens (API keys) as soon as possible, which was performed on May 31.

❗

Please know that we are taking this incident very seriously. Your trust in ReadMe means everything to us, and our support team is available to help answer any questions you have regarding this incident or the resulting token rotation.

Impact

The initial leak was caused by a misapplication of a piece of our middleware stack intended to cull sensitive information (including these tokens) from our response data. This bug was introduced in November when we refactored the login page for ReadMe hubs, and affected all public projects while it was live.

Further exacerbating the issue, some of these pages were cached by web crawlers such as the Internet Archive, limiting our ability to control the leak, a factor in our decision to rotate all tokens. While we have not found any evidence of malicious activity to date, a bad actor could surface a token from a web crawler's cache and use it to access and manipulate a project to their own ends.

Response

We have released multiple fixes and improvements to reinforce the security of these tokens on our platform. After assessing the potential impact of these incidents, we concluded that the best path forward was to revoke and replace all ReadMe tokens that existed during the period that the application was vulnerable. For help updating API keys where they're being used on your end, we've put together a few FAQs here.

While we're aware of the inconvenience — and are deeply sorry for any frustration we've caused — we determined it was a necessary step to avoid exposure of our users’ data. Again, our support team is available to help answer any questions, and we greatly appreciate your understanding as we work to address this incident.

What we’ve done to mitigate this issue:

• Patched our safelisting middleware logic to stem the underlying leak.
• Revoked all tokens generated prior to the underlying fix.
• Disallowed web crawlers on all administrative pages.
• Refactored our token management system to add prefixing and “last-used” tracking.

How we plan to prevent similar incidents in the future:

• Enhancements to our security scanning infrastructure.
• Improved source tracking for ReadMe audit logs.
• Increased investment in security resources.
• Additional token scopes + permissioning options.
• Joining GitHub’s security scanning project.