Just add Time Travel: Building your Games for Both Streamers and Viewers

Twitch Extensions empower new interactions and collaborations between streamers and viewers. Our friends at Amazon Games explored this with their experimental top-down shooter The Collaborative.

The Collaborative is both a game and a Twitch Extension. While the streamers are playing the game, the viewers are using the Twitch Extension. While the streamers are strafing and shooting deadly robots, the viewers are stamping the world with pixel art to ward off the deadly robots.

Let’s take a look at the Twitch video player and how it appears to the viewers.

What is happening here? Each of those lasers is a pixely stamp being sent from a viewer’s browser into the streamer’s game.

As a viewer, you can use the canvas on the right to design your stamp, and once you’re satisfied, you click anywhere on the stream to place it in the game world. Click on the wall, drop a stamp on the wall. Click on the floor, drop a stamp on the floor.

From the comfort of your browser, you can graffiti the game with your stamps by clicking anywhere in the game. And once it is in the game, the streamer can find it, play with it, and share it with the rest of the stream!

But how?

With the magic of Twitch Extensions! From within a video overlay Extension, we listen for the viewer to click anywhere on the stream. When clicked, we capture the position, time, stamp, and username. We send all this data to the streamer’s game.

From within the game, we fire a raycast from the camera’s position.

The biggest problem with this approach is that the player and the camera move over time, and a Twitch stream has some latency. If we naively raycast from where the camera is currently positioned, it’ll appear a few seconds ahead of where we wanted to stamp it. Each and every viewer has a variable amount of latency!

In this example (seen below), the viewers are always two seconds behind the streamer: When the streamer is interacting with the new Purple 6, the viewers are still interacting with the old Green 4!

Fortunately for us, we can check their viewer latency in the Twitch Extension Helper’s onContext method. The hlsLatencyBroadcaster is the amount of seconds between the viewer and the streamer.

So to account for the latency between streamer and viewers, we instruct the game to maintain a history of all the previous camera positions for the last 30 seconds. We then have the Extension include their latency with each stamp they send. When the game receives the stamp, it rewinds the camera position to account for the latency and performs the raycast.

Also, because our timeline between the viewer and streamer is lagging, it means viewers don’t see their stamp dropped into the world right away. Even though the game has updated almost immediately with the stamp, the viewers won’t see it until the stream has caught up. So to ensure the viewer that their stamp is working, we render a loading bar. We run this loading bar long enough to account for the hlsLatencyBroadcaster.

And just like that, we have an end-to-end interactive Twitch experience. Viewers can interact directly with the streamer through Twitch!

If you’re interested in The Collaborative, play it on Twitch today!

For more information on building Twitch Extensions, visit dev.twitch.tv/build or join the TwitchDev Community Discord. Or Ready to build interactive experiences on Twitch? Start now!