Triangulated Sam and Bill
In this project, I tried to create a morph between two images -- my portrait and Bill's mugshot.
I also computed the mean face of a population dataset, and generate caricatures by extrapolating from the population mean.
I achieved this through image warping, cross-dissolving colors, and manipulating facial geometries using keypoints and Delauney triangulation.
First, I resized my portrait's dimensions to Bill's mugshot to ensure correspondence.
Then, I manually selected 55 keypoints on my portrait and Bill's to mark corresponding facial features, 4 of them being the corners. I did this using the labeling tool created by last year's student.
I then computed a Delaunay triangulation based on the average positions of these keypoints to map the two faces.
1. Calculated the average of the corresponding keypoints from both images to get the intermediate shape, using the formula below:
2. Created empty images im1_warped and im2_warped to store the warped versions of each input image.
3. Iterated over each triangle in the Delaunay triangulation computed on the average shape. For each triangle, I:
3a. Extracted the corresponding vertices from both images' keypoints and the average shape.
3b. Computed the affine transformation matrices to map the source triangles from each image to the destination triangle in the average shape.
3c. Used the warp_triangle function to perform the affine warping for each triangle, applying inverse warping and masking to ensure accurate pixel mapping.
4. Averaged the pixel values of the two warped images to blend them together, using the formula below:
I created a morph sequence, defining 45 frames to show the transition between two faces. I calculated the warp and dissolve fractions (ranged from 0 to 1) for each frame, blending them gradually.
Plugging these fractions into the morph function, I generated intermediate images and saved them to the output directory. Finally, I compiled these images into an animated GIF, with each frame being displayed for 30 ms.
In this part, I computed the mean face of a population using the Danes dataset, which includes facial images annotated with 46 keypoints each.
First, I collected all facial images (*a.jpg) and their corresponding keypoint files (*a.pts) from the dataset directories. Then, I used a numerical sort to pair images and keypoint files correctly.
After I loaded each image and keypoints (verifying that there were 46), I calculated the mean shape by averaging the keypoints across all images. This mean shape is the mean facial geometry of the population.
I then used Delaunay triangulation on the mean shape to warp each face to the average geometry with affine transformations, then averaged the pixel values of these warped images to create the population's mean face.
The output I got (shown below) is the average facial structure and appearance of the population.
To visualize how my own face compares to this average, I created a new image of myself called sam2.png
However, since I selected 55 points in the previous step, I reselected 46 points on my portrait to match the .pts file that was attached to each image in the dataset.
To make sure I had the order correct, I implemented a script that ovelays the points on the face, specifying the order of selection based on the line order of the .pts file.
This is what the result of that script looks like:
And with this ordering, I clicked on my face in approximately the same spots in the same order.
Using the same triangulation technique, I warped my face into the average geometry, and the average face into the geometry of my face.
I created caricatures of my face by extrapolating from the population mean computed earlier. Specifically, I used the average expression face derived from the dataset to exaggerate features of my own face.
By setting the exaggeration factor α = 1.5, I amplified the differences between my facial features and the average expression face and created a caricature. This was done by setting the alpha value outside [0, 1].
I calculated the target keypoints using the formula:
I experimented further with appearance extrapolation to adjust attributes like skin tone by leveraging the differences between my face and the average face.
When I chose a high exaggeration factor (α = 1.5), I got my skin to look lighter.
When I chose a negative exaggeration factor (α = -1.5), I got my skin to look darker.
