Science: Surfing the 4th Largest Stream of Data

Twitch is the 4th largest stream of data on the internet.((Study by Deepfield original published in Wall Street Journal (paywall :/). Gamespot syndicated article.)) Conclusion: People really like watching each other play video games.

I’ve always been curious about how people find content on Twitch, so I dug into the discovery process in aggregate. The most striking result I found is the terrible performance of sessions (See: Appendix) that start on offline channel pages. Broadcasters get almost nothing from a third of their hard earned direct traffic, because their channel is offline. If you follow a link to a Twitch channel, the broadcaster could easily be sleeping. In contrast Tweets provide a consistent destination; tweets are only offline if Twitter is down.

It’s no surprise, then, that “broadcast more” is one of our most common pieces of advice for growing your audience on Twitch. I recommend broadcasters take advantage of Host Mode ((Host Mode release blog post. Drew Harry’s post on who hosts who.)) to keep their channel live. Part of my motivation for this post is that broadcasters cannot see the size of this problem from the dashboard.

Browsing on Twitch consists primarily of scanning the various directory pages that sort channels by their popularity or game, not actually diving in and watching video. People might take a minute or two to find something to watch, but if their first choice isn’t engaging they’re unlikely to go through that process again.

It’s time I introduce myself; I’m Danny Hernandez, one of the researchers on the Science team at Twitch. I make sure numbers have a seat at the table.

Defining Success

In order to explain why offline channels are terrible, we first need a quantitative definition of success. Success for Twitch is defined as watching a piece of content for longer than 5 minutes. A whopping 91% of Twitch’s minutes watched are on video views longer than 5 minutes. Also, there is strong evidence that viewers decide what they like quickly; 40% of follows happen in the first minute of a view. By the definition above, 46% of sessions succeed.

Offline Channels

It’s helpful to split sessions into two categories: those that start on channel pages and those that do not. 47% of sessions begin on a channel page. These users came to Twitch through a link to a channel they are primed to watch.

Note: A successful session leads to watching a given video for more than 5 minutes. All sessions above start on channel pages. Sessions are split based on whether or not that channel page is live.

Less than 1 in 5 users that start on offline channels find something to watch. People that start on a live channel have a huge head start on discovery, and 3 in 5 of them succeed, i.e. they watch for more than 5 minutes. For users that start on a channel page, the majority of discovery happens outside of Twitch.

This issue is pretty specific to Twitch, because we’re primarily about live content. When you follow a link to a Twitch channel the broadcaster could be out at a dinner. Maybe you clicked because you were promised Zelda and you ended up on a channel playing Super Meatboy. In contrast Tweets provide a consistent destination; tweets are only offline if Twitter is down. Given the difficulty of broadcasting more than 8 hours a day it’s kind of impressive that 2 in 3 people land on a live channel. Identifying and explaining issues like this are one of the most useful things I do at Twitch.

Note: The image above summarizes what users see when they land on an offline channel.

Broadcasters that want to grow should understand this issue. The recently launched Host Mode give broadcasters ways to keep their channel active when they’re offline. A perfect solution to this “opportunity” could push overall discovery success from 46% to 51%. I’m bullish on Twitch’s ability to take advantage of this opportunity.

Browsing

Let’s turn our attention to people who actually try to look for video on Twitch. Channel surfing is not a common behavior on Twitch; only 17% of sessions include “video browsing” (2 or more video views within 5 minutes of each other).

Note: A successful session leads to watching a given video for more than 5 minutes. Browsing video is defined as having 2 or more video views within 5 minutes. Browsing directories is defined as having two or more pageviews within 5 minutes.

When users do browse video, they have a high success rate. Browsing on Twitch primarily consists of browsing the various directory pages that sort channels by their popularity or the game they’re playing, not actually diving in and watching video. 60% of sessions include browsing directories.((Defined as 2 or more pageviews within 5 minutes. Almost all pages have listings of channels on them and are directory-like.))

Browsing indicates engagement and leads to an above average success rate of 55%. Browsers have almost as much success as people who visit live channel pages directly, which is pretty impressive. Visitors who do spend time looking for something to watch spend a median of 45 seconds browsing.

Distribution of Time Spent Browsing Directories

Percentile (%) Time(s) 25th % 17 50th % 47 75th % 147 90th % 285

Note: Sessions with browsing have at least 2 or more pageviews within 5 minutes. The time measured is the difference between the time of the first pageview in a session and the last. This distribution includes all successful and unsuccessful sessions.

People are not investing the amount of time into discovery that video browsing would require.

They might take a minute or two to find something to watch, but if their first choice isn’t engaging they’re unlikely to go through that process again.

Engagement

Logged in users are more successful than logged out users (50% vs 41% success rate). Logged in users are more engaged, have more experience finding what they want, and have built up their following directory. The more striking way in which logged in users are different is they attempt to find something to watch 4.5 times a day whereas logged out users only look for something on Twitch twice a day. Engagement on Twitch is driven by looking more often than rather than by looking more successfully. Users come to Twitch often to find out “What’s happening now?”.

In Closing

Improving session success in and of itself is not a great goal, though. It has the problem many success ratios have: it’s good for Twitch when either the numerator and the denominator go up, but that could actually push the ratio down. When we rolled our localized directories a|b test,((Localized directories release blog post.)) minutes watched went up by 1.5% but the success ratio went down by 12%! People were excited and they looked for something to watch 16% more times a day. Minutes watched is a much more robust acceptance criteria for a|b tests at Twitch. Calculating session success helps us understand what’s happening, but optimizing it in a vacuum could lead us astray.

Understanding the path to video is foundational knowledge for anyone who wants to understand or improve Twitch. If data is your jam, we’re still making huge insights. Consider joining Science.

Good Times,

Danny

Thanks to Alex Matzner, Drew Harry, Elliot Star, and Spencer Nelson for their help with this post.

Summary of set of pageviews for a user without a gap longer than 5 minutes in time and associated video views. Navigation Sessions and all related metrics are only for the desktop web app (no mobile, tablets, consoles, etc). I built Navigation Sessions in our Redshift Cluster. Mode wrote a good blog post explaining how sessions can be built in sql.

Examples:

User shows up on a channel, and then watches it for 50 minutes. Then, the user leaves.
User shows up on the front page, watches video for 30 seconds, clicks browse games, clicks following, clicks on a channel, watches that channel for 20 minutes and leaves.
User appears on the frontpage. They’re scared by loud noises and immediately close the tab.