2022-12-07 7:30PM EST: We became aware of slow API responses and began to investigate.
2022-12-07 7:45PM EST: We identified multiple slow queries in our database and began working to optimize these queries.
2022-12-07 8:43PM EST: We have completed optimization of one of the two identified problematic queries. This optimization alone was not enough to reduce load on the database.
2022-12-07 8:50PM EST: We have blocked subscriber authentication-related API requests at our CDN to reduce database load as we continue to address the remaining slow query. All services except those related to subscriber authentication and subscription should be restored. Subscribers are currently unable to log in and new users are unable to subscribe. Subscribers who are already logged in can watch premium content, and free content is unaffected.
2022-12-07 9:41PM EST: Optimization of the remaining slow query is completed and we are working on deploying the fix.
2022-12-07 10:00PM EST: The fix has been deployed and we have re-enabled access to the authentication API. We are monitoring the impact on database load to ensure the fix resolves the problem.
2022-12-07 10:15PM EST: We have verified authentication functionality and current database load indicates that the fix was successful.
