As mentioned previously, I re-used the frame handling code from Tom Box's implementation of motion tracking. I re-coded the tracking algorithm myself. To track a point, the co-ordinate of the starting point is given to the program. The values of the pixels in a box surrounding the point are then read. The next frame is then loaded, and the box is then moved around the original co-ordinates, or "jittered". The values are compared to the original box, and the co-ordinates for the box which is closest in value are taken. 
The images are output as another series of jpegs, but with the tracking point highlighted in green so a program such as fcheck can easily be used to see the results of the tracking. The system is not perfect, so a method of manual keyframing needs to be introduced. I implemented this in the form of a text file which is read in by the program. The user can type in the frame number, and the x and y co-ordinates, and the program will override its own values and use these in its place. Finally, the co-ordinates for each of the frames are then output into a text file.

Program 1.

I wrote another program for the second part of the process. Once all of the points have been tracked, we need to concatenate all the data into one file. Also, we need to make the co-ordinate positions relative to the nose point, as this will eliminate any movement of the face. The second program does this, and again, outputs a text file.

Program 2.

We can now take this data into Maya, and displace the mouth with it. To simplify this, an instance of the head is created, which we can work on without it moving around, and any changes made to it will affect the original. The first Mel script reads in the motion capture data, and applies it to a polygon with 10 vertices. This is preferable to applying the data directly to the model, as we need to change to scale and position of the co-ordinates relative to the size of the mouth. This is far more easily done interactively in the workspace by moving the polygon around rather than by scaling the co-ordinates in the program.

Mel Script 1.

The second Mel script takes the vertex positions from the polygon and applies them to the face. As the motion tracking points were affixed outside the lips in the original video, and more than one vertex is controlled by each tracking point, we need to be able to give an additional displacement value to each vertex, to make sure they are in the right position.

The other part of the face which is deformed is the chin. As there are many vertices which need to be moved by one tracking point, it is inefficient to work out additional displacement values for each point on the chin. Instead, we work out how much the chin control point has moved between frames, and apply this displacement to every vertex on the chin. For all the displacement on the face, only the x and y values are taken in to account, the z value remains the same as the original model.

Mel Script 2.

The program worked quite well, and despite the fact that the source material was produced using household equipment, relatively few keyframes needed to be set. I found that some points tracked better using the original images, and some tracked better if I applied a threshold filter to the video.

Continue to Blendshapes.
Home.