Face Morphing and Modeling a Photo Collection

Fall 2024

In this project, I use affine transformations to morph some pictures in interesting ways.

1. Defining Correspondences

In order to morph an image into another image, we first must define a set of points, or correspondences, in both images. I used this online tool to choose a set of 30-50 points between two images. Then, I computed a Delaunay triangulation on the first image’s correspondences. The same triangulation was applied on the second image to ensure that the triangles matched between the images (the morph would not work properly otherwise).

The Rock

John Cena

Correspondence points and triangulation

2. Computing the “Mid-Way Face”

To compute the “mid-way face” between two images, I took the average of the correspondence points between the two images to produce an average shape. Then, I warped both images to the average shape. To do this, I computed the affine transformation matrix of each triangle from each image to the average shape. Then I implemented an inverse warp using the inverse of the transformation matrix. As opposed to a forward warp, an inverse warp starts at the destination image and extracts pixel values from the starting image by warping the destination triangle coordinates. Inverse warping can be more advantageous as it does not create holes in the resulting image.

To avoid looping over the raw pixels, I used skimage.draw.polygon to create a mask over each triangle. With this method, I was able to extract multiple pixels from the starting image at once.

For coordinates that landed in between whole numbers during the inverse warp, I determined the pixel value by rounding to the nearest whole number. I also tried bilinear interpolation, but this ended up taking too long,and there was no significant difference between the two methods.

To cross-dissolve, I took the average pixel value between the two warped images.

The final result is an interesting blend:

The Rock

John Cena

Rock Cena

3. The Morph Sequence

To create a morph sequence between two images, I implemented the same methods as in computing the mid-way face, but with a weighted average. In code, this looked something like ((1 - coeff) * im1_pts) * (coeff * im2_pts) for the average shape, and ((1 - coeff) * im1_pixels) * (coeff * im2_pts) for the cross-dissolve. For the purposes of this project, my weighted average (coeff) was i/45, where i was the frame number in the sequence. 0/45 corresponded to the first image, and 45/45 corresponded to the second image.

Final morph sequence between the Rock and John Cena:

Rock Cena morph sequence

I also created a morph sequence between the Rock and a rock (the same rock from the previous project):

The Rockrock returns

4. The “Mean Face” of a Population

In this part, I took 200 smiling faces from the FEI Face Database, a Brazilian face database. Using the 46 correspondences provided, I computed the average shape across all 200 faces and warped each face to the average shape. For cross-dissolve, I took the average pixel value across all warped faces.

Images 0, 1, and 2 from FEI database (left), warped to the average shape (right):

Your average Brazilian smile:

I tried warping my face to this average Brazilian face’s geometry, and vice versa. I found the result to be slightly disturbing:

Me

Average Brazilian face

Me warped to average face geometry, and average face warped to my geometry

5. Caricatures: Extrapolating from the mean

In this part I tried extrapolating my face from the average Brazilian face, and vice versa. To do this, I computed the difference between my image’s correspondence points and the mean image’s correspondence points. I added this difference to my image’s correspondence points, scaled by an alpha coefficient factor. To emphasize my features, I computed ((me - average)*alpha + me), and to emphasize the mean image’s features, I computed ((average - me) * alpha + me). Note again that all of this is done on the correspondence points; I warped my image based on the resulting correspondence points. A higher alpha value resulted in a stronger extrapolation.

There is a warping artifact near the top of my head. I believe this is due to how there are not really any correspondence points for the hair or forehead.

Extrapolating my features:

Extrapolating the mean’s features:

6. Bells & Whistles – Myself over the years

I collected images of myself from 2005 to 2024, aligned them, and added correspondence points following the FEI face database’s point assignment pattern. Then I created a morph sequence across all of these images.

7a. Bells & Whistles – Gender change

In this section, I used average faces I found on the Internet to morph myself (and others) into the opposite gender. I tried 3 different methods (in the same order as the images below): warping input face to average face’s geometry, warping average face to input face’s geometry, and computing a “mid-way face” between both faces. The first two methods only involve inverse warping, while the third method involves both inverse warping and cross-dissolving.

Also, all of the correspondences were marked manually, so any artifacts may be due to human error.

Me morphed to the average Chinese man:

Left to right: me warped to avg's geometry, avg warped to my geometry, mid-way face

Aaron morphed to the average Chinese woman:

Dongsu morphed to the average Korean woman:

7b. Bells & Whistles – Gender change with actor/actress

I also tried morphing with a set of average actor and actress faces. I thought it was interesting to see more accentuated stereotypical features of the other gender.

Me morphed to the average Chinese actor:

Aaron morphed to the average Chinese actress:

Dongsu morphed to the average Korean actress:

7c. Bells & Whistles – Actress morphs

I also tried morphing myself to various average actress faces.

Me morphed to the average Chinese actress:

Me morphed to the average Japanese actress:

Me morphed to the average Korean actress: