Problem Statement
How Might we Detect a Subject’s Familiarity of Objects Covertly during Investigations?
Discover
Company Introduction
Home Team Science and Technology (HTX)
As the force multiplier of Home Team, HTX uses science and technology to empower and augment front-line operations.
Biometrics and Profiling Center of Expertise
The Biometrics & Profiling Centre of Expertise innovates and enables multi-modal biometric capabilities for fingerprint, facial, and iris recognition. One area of work is the development of profiling capabilities to sieve out people with malicious intent, in an efficient and non-intrusive manner.
Existing Solutions: Contact-based Modalities
Body Sensors
Brain Observations
Polygraph
Heart Rate
Blood Pressure
Sweat Sensors
Existing Solutions: Contactless Modalities
Thermal Imaging
Eye Tracking Glasses
Layered Voice Analysis
Micro-facial Expressions
Body Language
Literature Review
Referenced Paper
Detecting concealed familiarity using eye movements: the role of task demands (Nahari et al., 2019)
The paper demonstrates that theory-based task selection enhances the detection of concealed familiarity. By comparing visual detection and short-term memory tasks, the study found distinct gaze patterns for familiar versus unfamiliar faces, indicating that eye movements reveal recognition.
Define
Definition of Familiarity
Familiarity is defined to be recognition and a state of knowing something very well. This project focused on items that evokes a personal connection.
Stakeholder Analysis
Overview of all the stakeholders involved represented by the Power-Interest Matrix enabled the team to effectively prioritise and strategise stakeholder engagement.
IOs/ Profilers refer to frontline personnel that currently rely on subjective, experience-based methods for investigation and profiling. This stakeholder group seeks more data-driven, intuitive tools that can supplement their expertise and provide additional insights.
Suspects and Witnesses refer to individuals who have a vested interest in avoiding detection or being erroneously identified.
Research Participants refer to individuals who have participated in the research aspect of the project. This stakeholder group expects their data to be handled securely and anonymously.
User Personas
After analysing the stakeholders, the team created four user personas to represent each group from the Power-Interest Matrix. Hover over each person to find out more.
Techy Tara
Traditional Thomas
Criminal Christopher
Nervous Nadia
User Journey Map
This user journey map illustrates a security professional’s transition from polygraph to eye-tracking technology for deception detection.
It progresses through five phases: Pre-Adoption (equipment setup and calibration), Doubt & Deception Phase (reviewing inconsistent results), Discovery of Contactless Platform (introduction to eye-tracking technology), Pilot Trial (parallel testing of both methods with instant data visualization), and Behavioral Edge Realization (recognizing superior accuracy of eye-tracking in detecting deception through pupil responses and gaze patterns). The sentiment line shows initial skepticism about traditional methods, curiosity about new technology, and ultimately enthusiasm for the eye-tracking solution’s speed and reliability.
Develop
Solution Ideation
Following extensive literature reviews, analysis of current solutions, and stakeholder insights, we developed four ideas that address gaps in existing research and solutions.
Selected Solution
Our chosen idea uses a covert eye-tracker during interrogations to assess a subject’s familiarity with evidence-related objects. By analysing eye-tracking metrics such as fixation duration and scan path, investigators can gain insights into potential recognition of items like weapons or drug apparatus.
Wireframing
Experiment Design
The experimental process was designed to include a practice set consisting of three questions to allow the participants to familiarise themselves with the program, which helps to avoid restarting and invalidating their test results. Next, their individual gaze was calibrated using Tobii Pro Lab’s calibrating software before the recording was initialised. They then went through 32 official sets of stimulus before their recording was ended. A familiarity survey was conducted at the end of every experiment to determine if the unfamiliar objects chosen for them were truly unfamiliar to them.
The experimental program included a fixation cross before each stimulus was shown to correct their gaze to the center of the screen, which prevents them from staring into space or being fixated at the top portion of the screen.
There were two different types of stimulus within the 32 sets, where Type A stimulus included one familiar object (highlighted in green) and Type B stimulus contains all unfamiliar objects. For the familiar object in Type A stimuli, it was rotated amongst the four positions of the screen (top left, top right, bottom left, bottom right) to ensure that the participant would not be able to predict its position.
Data Collection
The data collection process was standardised across all 74 participants in the following manner.
Data Analysis
The hypotheses predict that familiar objects will elicit distinct eye movement patterns compared to unfamiliar ones, specifically: higher visit duration and count, increased fixation duration and count, earlier attention in scan paths, higher saccade count, and measurable pupil dilation. These predictions test how object familiarity influences visual attention and physiological responses.
Based on our analysis, the metrics fixation duration, fixation count and scan path support the hypotheses. The metrics pupil dilation and saccade count do not support the hypotheses.
Machine Learning
The project leverages Machine Learning (ML) to analyze eye-tracking data for determining image familiarity, offering enhanced capabilities beyond traditional statistical analyses. After preprocessing eye-tracking data from Tobii software to extract features like fixation duration, fixation count, and pupil diameter, the team compared several supervised ML algorithms and their respective accuracies including logistic regression (38.71%), random forest (65.59%), XGBoost (66.67%), SVM (46.24%), and neural networks (69.89%).
These findings demonstrate ML’s effectiveness in differentiating between familiar and unfamiliar image viewers, with the trained model to be implemented in software for predicting suspect familiarity with objects.
Deliver
Software Architecture
The system architecture diagram depicts a web application with distinct frontend and backend components. The frontend utilizes React, HTML, CSS, and JavaScript to create the user interface. The backend combines JavaScript, Python, and PHP services that interface with Tobii Pro eye-tracking hardware, while featuring an AI component with preprocessing and XGBoost implementation for data analysis. Database management is handled through MySQL and Firebase.
The architecture follows a client-server model with bidirectional data exchange between frontend and backend, creating an integrated environment for eye-tracking experiments with machine learning capabilities.
Implemented Software
This application streamlines investigative processes and securely stores data, making it more efficient and accessible for investigation officers.
The Future of Investigations
Acknowledgements
We sincerely thank Home Team Science and Technology Agency (HTX) for the opportunity to tackle a critical industry challenge, with special appreciation to Mr. Kee Ein Cern and Miss Liang Kai Ying for their collaboration.
Our deepest gratitude goes to Prof. Rakesh Nagi for his unwavering support and invaluable guidance, and to Prof. Kwan Hui for his insightful feedback and resources.
We also appreciate Dr. Bernard Tan and Professor Rashmi for their dedication in refining our clarity in writing and presenting.
Finally, we acknowledge the Capstone Office, Institutional Review Board (SUTD), and Spaces for their support in facilitating our research and administrative processes.
