// CONTENTS
In 2016, my partner Kohaku and I built a TV studio inside a video game.
That’s the short version. The longer one involves a green screen, two game engines, a handful of phones running puppets, a custom video plugin I’m still a little proud of, and a theme song by a band of friends called the Space Bards. The show was called Kohaku’s World, and it went out live on Twitch.
The Show
Kohaku’s World was a live variety show. On any given night you might get game streaming, a comedy skit, or a movie watched Mystery Science Theater 3000 style, with the two of us riffing on whatever was on screen.
Kohaku was the host. She stood in front of a green screen in our studio, and on stream she appeared inside a 3D world: a video-game basement, an arcade, a retro racing game. I was the developer, which on a two-person team means I was also the set builder, the lighting department, the broadcast engineer, and the person making the title sequences.
We made it at CyberForge, my studio at the time, and nobody handed us a playbook. A few years later the film industry would turn “virtual production” into a buzzword. In 2016 it was mostly a stubborn idea and a lot of late nights.
Building the Sets
Every set was a real-time Unreal Engine 4 environment that I designed, built and dressed myself.
The basement was home base: wood paneling, a glowing neon sign, old hardware, a CRT in the corner. The arcade was full of classic games you could actually play, in virtual and mixed reality. And there was a racing game with retro 1980s graphics and surprisingly serious vehicle physics, because if you’re going to drive a DeLorean down a neon grid toward a setting sun, the car should at least handle properly.
If that DeLorean looks familiar, it should. It’s the image at the top of this site’s home page, and it’s been my LinkedIn banner for years. It started life in Kohaku’s World.
The sets were full of small things that worked. Screens in the world played video. A retro PC booted up on camera, one line at a time. There was a 1980s clock radio. None of it was strictly necessary, and all of it made the place feel lived in. A live show depends on whether the audience believes the room.
I built the physical side too. I set up the professional video-capture hardware and designed the green-screen studio, so that Kohaku could be keyed cleanly and dropped into whichever world the show needed.
The Hard Part: Real Video in a Real-Time World
Here’s the problem that ate most of my time.
A game engine is very good at drawing a 3D world. Out of the box, it isn’t built to take a high-resolution live camera feed, cut the person out of the background, and place her in that world as if she were standing in it, in real time, while the same machine renders the scene and sends the whole thing out as a stream.
And we weren’t doing it on a render farm. We were doing it on commodity hardware, the kind of PC you could buy and put under a desk.
So I wrote a custom video streaming plugin for UE4 that brought real-time video, with transparency, into the engine. The transparency was the key. A rectangle of video floating in a 3D room looks like a TV. A person with the background removed, composited into the room, looks like a host. I wrote custom shaders so the footage sat in the scene instead of on top of it.
Every frame had to arrive, get processed and get drawn before the next one showed up. On a live show there’s no fixing it in post. If a frame is late, everyone watching sees it.
Puppets, Phones and Two Game Engines
Kohaku’s World also had puppets: digital characters performed live, alongside the host.
The puppeteering setup was a mix of off-the-shelf tools and things I built. At its core was a puppeteering app I wrote in Unity, controlled from iOS devices over socket.io, with the result streamed into the Unreal scene.
Yes, that’s two game engines and a handful of phones talking to each other, live, on air. It sounds fragile when I write it down. Making it not fragile was most of the job.
The Theme Song and the Intros
Every show needs a theme song. Ours was written by our friends the Space Bards: pure synthwave, and a perfect fit for a neon basement. You can hear the Kohaku’s World theme on SoundCloud.
My other job was making parody intros. There was a Star Trek: The Next Generation intro retitled “The Previous Generation,” a Stranger Things intro, and a space-opera text crawl for the first episode, “Return of the Floppy.” Recreating an iconic title sequence shot for shot turns out to be a very good way to learn exactly why it works.
What It Taught Me
Kohaku’s World is one of the most fun things I’ve ever built, and one of the most technically demanding.
It taught me that real-time is its own discipline. In most software, a slow operation is an annoyance. In a live show, it’s a visible glitch in front of an audience. You learn to budget every millisecond, and to test the whole chain, from camera to stream, instead of trusting any one piece of it.
It also taught me that the best tech is the tech nobody notices. Nobody watching cared about my video plugin. They cared that Kohaku was standing in a neon basement, cracking jokes about a terrible movie. That was the whole point.
Today a lot of what we built can be bought off the shelf. In 2016 we had to make it ourselves, and I’m glad we did.