How Team Rolfes Delivers Live Animated Content Thanks to Real-Time Performance Capture


Team Rolfes is a boutique creative studio specializing in real-time performance and image creation. The brainchild of brothers Sam and Andy Rolfes, the studio has designed mind-bending visuals for clients such as Super Deluxe, Adult Swim, Nike, and various musicians around the world. Their visuals are incredibly unique, inspired by a fine art aesthetic with some surrealism thrown in, and are achieved specifically through cutting-edge, real-time performance capture techniques.

“The thing that’s most important to me creatively is that we can be improvisational and free in the moment,” explained Sam. “Whether our animation is being pre-recorded for use in a retail venue, or live streamed via Facebook, we always start by performing the content in real-time; that’s the basis for all of our work.”

The Team Rolfes workflow utilizes the power of game engines to direct, perform and deliver 3D animated content live. This process typically involves writing rough script beats and sculpting set elements in Z-Brush, then Sam or another actor getting outfitted with trackers or a motion capture suit. The team uses the HTC Vive VR headset as a camera, capturing performances through eight Vive body trackers or a motion capture suit, alongside Faceware’s Live Server and Glassbox’s Live Client for Unreal plugin, which combine to track and animate facial movements directly onto digital characters inside the Unreal Engine in real-time. If the content is intended to be pre-recorded, the actor can move through the scene multiple times experimenting with story beats and blocking, and then run through it again as a second or even third character. When delivering live-streamed content, the combination of Faceware Live Server and Live Client are crucial for successfully driving these animated digital characters in real-time – a feat that would otherwise be impossible to achieve.

One recent live-streamed project that relied heavily on facial performance capture was the “Build A Candidate” series for Super Deluxe, where Team Rolfes animated a satirical digital politician named Thiccolas P. Beauregard in real-time. In a series of Facebook Live videos, viewers were able to collectively design his campaign ads on the spot through Facebook’s feedback function, offering up comments, likes, dislikes, laughs, and other input. With Faceware Live Server and Live Client for Unreal, the actor playing Beauregard was able to improvise and respond organically to viewer feedback, ultimately creating a truly engaging experience for viewers.

“’Build A Candidate’ blew people away,” said Sam. “You can always have pre-programmed animations and dub the voice live, but that can feel static and staged. To really get a sense of the character’s words and movements, Faceware Live Server and Live Client for Unreal allow us to deliver a much greater 1-to-1 connection with the audience; it builds an interactivity that’s very compelling. The way the character was responding directly to comments and looked authentic in the process, people really got a kick out of it.”

Team Rolfes is planning to create even more live facial performance-driven content in the future, as live-streaming and e-sports become more and more popular.

“Twitch is one area with huge untapped potential,” Sam observed. “Even though you’re playing a game, your face is there and it’s such a conversational platform. It’s a really exciting area to explore live face capture, so that’s something I’m very focused on.”

Beyond the freeform creative process and the level of interactivity they can ultimately achieve for their digital characters, a real-time workflow also gives Team Rolfes a much greater level of flexibility in production. As few as two people can pack up the HTC Vive and a laptop and travel to any location with everything they need. Even with complex productions, the team can stay lean, with Sam often wearing multiple hats as actor, director, and developer.

Coming up in the new few months, Team Rolfes has a music video, retail, commercial, and fashion projects in the works, all over the world from Shanghai to Madrid to New York – and they show no signs of slowing down in creating truly unique content.

“Real-time tools ultimately let our characters feel more human, and that’s the essential thing really. Most of our imagery can end up fairly distorted or surreal, but having something be in the moment and be tracked correctly from a face or a body, grounds it in a certain amount of reality that it makes it more compelling and relatable to people. Today’s real-time technology is so powerful; we like to say that our process is a creative misuse of these tools for novel ends.”