New potential in photo and film production: what you should know about light-field

Light-field technology provides major advantage when editing scenes during the postproduction process because it supports the creation of different perspectives. A light-field is defined by the light rays within a specific area. If all light rays are captured within a scene, it is possible to generate a perspective from any position. Researchers in the Moving Picture Technologies department at Fraunhofer IIS are already demonstrating how scenes can be optimized using a set-up with professional cameras in a planar arrangement that capture slightly different scene perspectives. This opens up new creative opportunities. In a recent interview Joachim Keinert, head of the “Computational Imaging and Algorithms“ group, shares his insights.

What are the differences between classic media production and light-field production?

Joachim Keinert: It depends on what is meant by classic media production. Considering how many videos are produced today using compositing, 3D modelling or even volumetric capturing where you can have quite sophisticated virtual camera movements that are close to VR experiences. Light-field captures a scene in such a way that you can virtually move it afterwards. For example if you want to capture the shininess of your surfaces or specular highlights to really make the scene or object feel realistic. 

What are the key facts one should know about light-fields and what makes this technology so appealing for media production?

Joachim Keinert: To avoid confusion, there are two kinds of light-fields. There are light-field cameras based on micro lenses inside the camera. It gives you the option to refocus but they don´t give you a lot of perspective change. So, when we talk about light-field we really mean several perspectives and several cameras. For video this means that you take multiple cameras and mount them on a rig. For static scenes you can take a single camera and capture your scenes sequentially. Then, the next step is to take all the data you processed - this is where our expertise comes into play - to recover geometry or colors. Once you are done with these steps you get the great benefit that you have recorded your scene as it is with high photorealism for good memorization.  

In which part of the media production workflow does light-field open new possibilities?

Joachim Keinert: Light-field can do the most in an application where you really want to reproduce a scene as it is and don´t want to change it too much. Let´s take an example: Imagine you have a nice apartment for sale and you want to reach as many interested parties as possible. You want to give them the possibility to tour it beforehand in a virtual way. For such an application it is important that you stick as close as possible to the original to not disappoint your customers because your 3D model did not reflect the true situation. You don´t want to add or change anything of the real-life place. With the use of light-fields, you really want to reproduce your apartment as it is. 

What are the challenges and opportunities?

Joachim Keinert: A challenge definitively is the large number of views or captures you have to take and the large amount of data that is coming along with the process. But once you are through with this, it gives you the opportunity to memorize and preserve previous scenes or objects for later on. Because you can virtually watch them from different perspectives and you have a much better realistic impression compared to alternative technologies that are available today.

Can you give us a number of cameras for light-field recording?

Joachim Keinert: This obviously depends on the scene complexity. For video it´s much smaller than for static scenes. To give you an order of magnitude for static scenes: It´s similar to photogrammetry. In photogrammetry you need hundreds or thousands of captures and for light-field it´s similar. But compared to photogrammetry you are able to recover a better color impression by looking to the shininess of your surface and on small reflections. This is how light-field tops it off. 

Would you rate light-field as a top technology?

Joachim Keinert: Definitively. When you want to reproduce the scene as it is, you need to observe your objects from different perspectives and this is exactly where light-field is exceptional. 

Joachim, what is your approach to light-field?

Joachim Keinert: My team started with video arrays for light-field. Today, we are concentrating more on static scenes because it is possible to capture all the views you need with this sort of set-up. Our current focus is on reproducing a scene in a faithful way. In the future, we could of course imagine using also video or taking all the data t we have captured, for example all the reflections. This would allow us to estimate material properties and once you know what kind of material your object is composed of, you can relight it much better. This would obviously be a big step forward.  

In comparison to 360 degree virtual walk-arounds, 3D navigation or 3D renderings, what is exceptional about light-field?

Joachim Keinert: Let´s start with the 360 degree photos or videos because they also capture reality. It´s a very good comparison because they have a similar goal but they lack in capturing geometry. You don´t know the distances of your object instead you assume that everything is located on the sphere situated around you and if you move you get distortions because of the lacking distances. Light-field follows the same goal - reproduction of reality - but it gives the correct geometric perspectives. Another technique that we should mention are game engines. You  can move around in amazing worlds but you have to create your photorealistic models first which is a lot of work and requires a lot of expertise, in particular, if you want to reproduce reality. Light-field technology gives you the advantage to cover all the different perspectives so the user can choose which one he wants to have and of course different users can choose different perspectives. 

Thank you for this interview.