While 3-D movies continue to be popular in theaters, they haven’t made the leap to our homes just yet — and the reason rests largely on the ridge of your nose.
Ever wonder why we wear those pesky 3-D glasses? Theaters generally either use special polarized light or project a pair of images that create a simulated sense of depth. To actually get the 3-D effect, though, you have to wear glasses, which have proven too inconvenient to create much of a market for 3-D TVs.
But researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) aim to change that with “Home3D,” a new system that allows users to watch 3-D movies at home without having to wear special glasses.
Home3D converts traditional 3-D movies from stereo into a format that’s compatible with so-called “automultiscopic displays.” According to postdoc Petr Kellnhofer, these displays are rapidly improving in resolution and show great potential for home theater systems.
“Automultiscopic displays aren’t as popular as they could be because they can’t actually play the stereo formats that traditional 3-D movies use in theaters,” says Kellnhofer, who was the lead author on a paper about Home3D that he will present at this month’s SIGGRAPH computer graphics conference in Los Angeles. “By converting existing 3-D movies to this format, our system helps open the door to bringing 3-D TVs into people’s homes.”
Home3D can run in real-time on a graphics-processing unit (GPU), meaning it could run on a system such as an Xbox or a PlayStation. The team says that in the future Home3D could take the form of a chip that could be put into TVs or media players such as Google’s Chromecast.
The team’s algorithms for Home3D also let users customize the viewing experience, dialing up or down the desired level of 3-D for any given movie. In a user study involving clips from movies including “The Avengers” and “Big Buck Bunny,” participants rated Home3D videos as higher quality 60 percent of the time, compared to 3-D videos converted with other approaches.
Kellnhofer wrote the paper with MIT professors Fredo Durand, William Freeman, and Wojciech Matusik, as well as postdoc Pitchaya Sitthi-Amorn, former CSAIL postdoc Piotr Didyk, and former master’s student Szu-Po Wang '14 MNG '16. Didyk is now at Saarland University and the Max-Planck Institute in Germany.
How it works
Home3D converts 3-D movies from “stereoscopic” to “multiview” video, which means that, rather than showing just a pair of images, the screen displays three or more images that simulate what the scene looks like from different locations. As a result, each eye perceives what it would see while really being at a given location inside the scene. This allows the brain to naturally compute the depth in the image.
Existing techniques for converting 3-D movies have major limitations. So-called “phase-based rendering” is fast, high-resolution, and largely accurate, but it doesn't perform well when the left-eye and right-eye images are too different from each other. Meanwhile, “depth image-based rendering” is much better at managing those differences, but it has to run at a low-resolution that can sometimes lose small details. (One assumption it makes is that each pixel has only one depth value, which means that it can’t reproduce effects such as transparency and motion blur.)
The CSAIL team's key innovation is a new algorithm that combines elements of these two techniques. Kellnhofer says the algorithm can handle larger left/right differences than phase-based approaches, while also resolving issues such as depth of focus and reflections that can be challenging for depth-image-based approaches.
“The researchers have used several clever algorithmic tricks to reduce a lot of the artifacts that previous algorithms suffered from, and they made it work in real-time,” says Gordon Wetzstein, an assistant professor of electrical engineering at Stanford University, who was not involved in the research. “This is the first paper that produces extremely high-quality multiview content from existing stereoscopic footage.”
Didyk says that modern TVs are so high-resolution that it can be hard to notice much difference for 2-D content.
“But using them for glasses-free 3-D is a compelling application because it makes great use of the additional pixels these TVs can provide,” Didyk says.
One downside to converting traditional 3-D video to multiview TVs is that the limited resolution can lead to images appearing with duplicates near or around them — a phenomenon referred to as “ghosting.” The team hopes to further hone the algorithm to minimize ghosting, but for now, they say that they are excited that their conversion system has demonstrated the potential for bringing existing 3-D movie content beyond the multiplex.
“Glasses-free 3-D TV is often considered a chicken-and-egg problem,” Wetzstein says. “Without the content, who needs good 3-D TV display technology? Without the technology, who would produce high-quality content? This research partly solves the lack-of-content problem, which is really exciting.”
