Microsoft IllumiRoom fleshed out for Gaming and Living Room

Microsoft’s IllumiRoom immersive projected gaming system, first shown off at CES, has broken cover again for a more comprehensive demo, complete with more details of how the “TV expanding” augmented reality works. Still described as a proof-of-concept, though thoroughly whetting appetites for what the next-gen Xbox might one day evolve into, IllumiRoom will be presented at CHI 2013 [pdf link] this week, complete with learning the
topography and design of your living room and then digitally manipulating it.

The Microsoft Research team responsible for IllumiRoom is currently using a standard projector with a wide field of view, and a Kinect for Windows sensor bar, though any commercial implementation would probably be designed to sit on a coffee table. Automatic room calibration is included, projecting various patterns and sequences which map the outline of the TV, furniture, and the room’s geography in 3D; future iterations could even identify and track moving objects, such as people, as they move through the projection.

Once the layout of the gaming arena is understood, IllumiRoom’s real magic can begin. Various implementations are proposed, from fully extending what’s on the TV to greater fill the room – making for a more immersive environment – to picking out specific elements to highlight them, such as weapons fire that escapes from the primary display. By changing how physical objects in the room have their own textures projected, furniture could be made to ripple and wobble, change color or desaturate, or have their lighting adjusted.
Alternatively, the IllumiRoom system might just expand on the theme of the current game: having virtual falling snow spread across the living room, perhaps, building up on the actual furniture. The three possibilities, Microsoft Research suggests, are “negating”, “including”, or “augmenting” real-world objects: either digitally masking them, allowing them to remain visible, or adding to them with projected graphics.

“Ideally, IllumiRoom would be directly integrated into a next generation console and new games would be designed for IllumiRoom from the ground up. We envision an API that enables triggering illusions, changing surface appearance, controlling room lighting, inserting objects into the physical environment, etc.” Microsoft Research

While the best result would be if games natively supported, and integrated, IllumiRoom functionality, the system could also fashion a suitable AR scheme by analyzing gameplay in realtime, similar to how Philips’ Ambilight system tracks on-screen colors and matches them with its periphery of multicolor LEDs. The developers also suggest that audio cues could be used, such as triggering a ripple of the surrounding projection whenever the system hears a gunshot sound.

There’s also potential for how IllumiRoom could work with non-game content, such as movies and television; the team fashioned a prototype extended field-of-view camcorder – in effect pairing a standard camcorder with one that has a wide FOV – with the main screen showing the core video and the IllumiRoom projector showing the peripheral footage.

“Can a grenade from the latest Bond film explode in your living room? How would such content be authored? It would be im- portant to investigate how the movie director should deal with the fixed nature of a film and the randomness imbued by the system adapting to the user’s living room” Microsoft Research

Unsurprisingly, there’s still no (public) talk about how IllumiRoom tech and Microsoft’s Xbox ambitions might work together. As it stands, the current projection system relies on a PC for its brain, though it’s worth remembering that Kinect itself started out as a proof-of-concept peripheral, and only later evolved into an Xbox 360 add-on.