Follow Up on Site Outage

By now many of you are aware that Twitch had a site outage over the weekend, and many of you were personally affected by it. We’re happy to report that we’ve worked through all of the issues that caused the outage, and it’s now clear sailing for most Twitch users.

While the site is back to normal, there may be a few lingering bugs while we come back to full health — including some issues with chat and our password reset system. Please remember to reset your password according to the process outlined in our Help Center (including information for those of you experiencing some lingering issues like Facebook connect, for example). If for any reason you are still having trouble logging into your account, please go to our contact page and send us a note with details on your situation. We will help as soon as we possibly can.

For more detail on the outage, please review this blog post, the statement below from our CDN partner (Level 3 Communications) and the section “Technically Speaking” below that.

Statement from Level 3 Communications:

On June 21, as part of a maintenance event to improve service, we made a change to Twitch’s caching platform and as a result of the service event, we understand that some basic account information of Twitch users was inadvertently exposed. None of the data exposed included payment information. We worked swiftly with Twitch to correct this error and apologize to Twitch and its community of users for the inconvenience. It’s important to note that this error was not a result of any cybersecurity issues relating to Twitch’s properties.

The Twitch engineering team burned the midnight oil on this to get the site back up and running, and those of us non-engineers on the team want to send a personal thank you to each of them that worked through it.

We held an all-hands engineering postmortem meeting on this incident and have come away with a fresh perspective on security, and a path forward for better managing mishaps like this. We can’t promise mistakes will never occur. We can promise that we take our commitment to you very seriously and that we will always be as transparent with you as possible.

Technically Speaking

Level 3 caches our web pages to optimize delivery to our users. This means that our pages are stored around the world on their network rather than just being stored in our datacenters, so you get faster delivery of our web pages. There is critically important logic that governs which pages are and are not cached, and unfortunately with the recent change Level 3 made, they did not honor our caching logic.

This means they cached pages that included the logged in account information of the user that first hit that particular point in their network, so future users that were served from that location were delivered a logged in page. As a result, some users saw pages that were in the logged in state of another user. Remember, no damaging personal information such as payment or credit card information was exposed. The actions we have taken and will take in the future are in the spirit of an overabundance of caution.

Some of you are asking natural follow-up questions, such as why password hashes were on some pages to begin with. We use password hashes to authenticate users to our chat servers. It’s a way for the chat client to prove that it knows the password (i.e. that the person connecting to the chat server is who they say they are) without sending the password itself.

That said, there are modern techniques for accomplishing this that don’t expose any information about the password. One of the takeaways from our postmortem engineering meeting is to increase the priority of paying down certain kinds of technical debt. Over the coming weeks, we will authenticate chat clients using an oauth token instead of the password hash. We’ll be removing as much sensitive information as possible from assets we serve through the CDN, with the goal to become resilient to unforeseen circumstances in the future.

Writing software is an iterative process in which features are added, bugs are crushed, and overall quality is improved over time. We’re determined to come away from this outage with a clear path forward.

We know how much the community relies on Twitch. Remember — we’re here and we’re listening. Please subscribe to @TwitchTVSupport on Twitter, bookmark the Help Center, and be sure to check in here at the Official Twitch Blog regularly.

Thank you for your patience and thank you for sticking with us through this.