6 DOF stabilization using camera tracking and camera mapping

 

For a couple of recent projects we at carbon stumbled upon a way of remapping camera moves in live video footage. The following article describes (technically) how this effect is achieved.
An example of the final effect is seen at the opening of this website: 

http://www.waarzijnwemeebezig.nl/

This article is describing a method for redefining camera paths in real video or film footage, combining camera tracking (also known as match moving) and camera mapping (also known as perspective mapping)
This method is very suitable for perfect stabilization, but can be used to create other effects as well.
This can be used to stabilize the shot even if its handheld, shaky motion) or to change the camera path from the original.
It differs from traditional (post-) stabilizing techniques, which are more simple and quicker but less accurate and give only 3 degrees of freedom instead of 6. It is not only capable of compensating the shaky rotation of the original camera, put also its POSITION. Imagine a bumpy road: normal stabilization might be able to compensate the panning tilting and rolling of the shaky footage, but cannot get rid of the physical up and down motion from bumps and ditches in the road. 6 – axis stabilization using camera tracking and camera mapping can.
It requires two things: an (automated) camera tracker such as Boujou, PFtrack or Syntheyes, and a 3d package with camera mapping functions, such as 3dsMAX or Maya.
We use boujou and 3dsMAX, but the same principle can be used with any other similar package.

The first thing you have to do is track the footage with an automated tracker. If you import this footage in 3dsmax you get an animated (shaky and bumpy) camera and a lot of point helpers that represent tracked points in 3d space. From the perspective of the camera and a viewport background it is possible to understand what those points represent. Now we have to create basic geometry in 3d space that aligns with what we see in our scene. Some automated trackers can help with this as well.
We create a camera mapping material and apply it to our geometry, using the tracked camera as projection camera. We use this term because in fact we use the camera as a projector, not as a camera, because we will not be looking through it. Instead we create a second camera (view camera) that roughly moves as the projection camera, but is in fact smoother because we can steer its motion in 3d completely. It is important to stay close to the projection camera or artifacts will be visible. This is the camera that we will render.
The shaky footage the camera ‘beams’, directly compensates for the shaky movements of the projection camera. From the perspective of a third person it looks stable. With a little work and imagination you can turn handheld footage into the most convincing dolly or crane footage. Usually you have to zoom the view camera in a bit because otherwise the edges of the shaky footage come into view, so it’s usually a good idea to shoot higher resolution than finally needed. Also if shooting footage like this make sure you have a fast shutter speed, so there is little motion blur.
Taking this a step further, you can create completely new camera moves in post, by animating your view camera in other directions, as long as we don’t see outside the ‘projection-pyramid’ of the projector.

Examples:

 

Original Footage
 a quick shot for demonstration purposes. Completely handheld by a shitty cameraman!

 

 

 

 

 

 

 

 

 

3 DOF

Using the described technique, but without any geometry in the scene, just an environment sphere with the footage projected. This gives only rotation stabilization, not position.

Notice the mirrorred edges, the shot should be zoomed in a bit to crop them. In some situations (such as a clear sky) you can get away with some mirroring.

 

 

 

 

6 DOF

This time, some basic geometry has been added to the scene, such as a groundplane, a wall, a cylinder for the revolving door and a plane for the panel on the right.

Now we can see how stable the shot looks. Notice that it had to be zoomed in to get rid of the mirrored edges. The little jitter that still exists in the shot is because of some frames that didn't track well in Boujou. This can be easily resolved, but we didn't bother for the demonstration.

 

 

 

From another perspective, one can see how it works, as the camera lookis like a projector, beaming onto basic geometry, the basic idea behind camera mapping.

Notice that the shaky beamer and the shaky shot cancel eachother out so from a static perspective the shot looks stable.

 

 

 

 

 

 

Multiple variations are now possible as well, such as this dollying-in motion. It is only achieved by animating the view camera in the abovementioned scene. The original footage is still the same.