The system uses a combination of facial recognition technology and breed-specific features to find the best match. Once a match is found, users can see a side-by-side comparison of their photo and the dog's photo, highlighting the similarities.
The model makes matches using ml5.js's API for facial camera recognition capabilities, allowing for real-time analysis and matching. ml5.js tracks facial landmarks using points at specific locations on the face, such as the eyes, nose, and mouth, to create a detailed facial profile. The model uses the data from the landmarks to compare the user's facial features with those of dogs in the database, looking for similarities in shape, structure, and proportions.
The dog's landmarks are from the DogFLW dataset, which annotates various facial points on the dog's face. The dataset contains 4,335 images of dogs' faces in various environments and conditions, each annotated with 46 facial landmarks and a bounding box.
From the 478 landmarks ml5.js detects, 23 proportional features are extracted. Not raw pixel positions, but ratios relative to face size so they hold up regardless of distance from the camera. These include eye width, eye openness, interocular distance, nose width, mouth width, face roundness, brow height, brow-to-eye gap, nose-to-mouth distance, and expression features like brow raise, mouth corner lift, and eye squeeze.
On the dog side, the same 23 features are computed from the 46 DogFLW landmarks and stored as a vector for every dog in the database. Not all 23 features are weighted equally. To find the closest match, three algorithms were tested: weighted Euclidean distance, z-score normalized distance, and cosine similarity. After testing all three against real faces, weighted Euclidean distance consistently produced the most visually convincing matches, so that is what the final model uses.
The weighted distance between your face and each dog is calculated:
d = √( Σ ( wᵢ × (humanᵢ − dogᵢ) )² )
where wᵢ is the weight for feature i, humanᵢ is your extracted ratio, and dogᵢ is the dog's ratio. The dog with the smallest d across all 3,853 dogs in the database is your match.
For example, in one real match the extracted human vector included an eye openness of 0.36, a brow raise of 0.15, and a mouth corner raise of 0.00. The winning dog produced a weighted distance of 0.49, beating out 3,852 other dogs, so that is the match you see.
The matching logic, feature extraction, and weight tuning were built with the help of an AI assistant, used throughout to write, test, and refine the calculation code.