PROBLEM
With advancement in technology, it is becoming increasingly difficult to identify deepfake images and videos with the human eye.
Deepfakes can be used maliciously in impersonation scams, frauds, and to spread misinformation. These are threats to citizens' safety.
To counter the spread of false information , the legitimacy of the video must be determined quickly and accurately.
USERS
The Digital and Information Forensics (DIF) Department of Singapore's Home Team Science and Technology Agency (HTX) and Singapore Police Force are at the frontline in detecting deepfakes in scams.
They are looking for a deepfake detection solution that improve upon their current verification process and produce results that can serve as evidence to convict offenders in court.
REQUIREMENTS GATHERING
Through an interview with our target users, we gathered the following user requirements:
User Management
Data Security
Fast Processing Speed
Above 90% Detection Accuracy
Evidence From Model Detection
Simple and Intuitive
User Interface
SOLUTION
Introducing VeriFake
An end-to-end deepfake detection webapp
With explainable results via heatmap visualization
Boasting
high detection accuray
DEMO
HOW VERIFAKE WORKS
When the user wants to upload a file, a request is sent via API gateway to obtain a pre-signed URL allowing the user to upload the file securely into the inference S3 bucket.
When the file has been successfully uploaded, an request will be sent via API Gateway that invokes the Sagemaker Prediction Model to process the file.
When the model is done processing the file, it stores the results in the Heatmap S3 Bucket and trigger the Inference Success/Error Topic according to the results. The topics will trigger the appropriate handling code which will send out an email notification to the user.
When the user wants to view the results, a request will be sent to retrieve the results for their viewing.
Our model consists of 3 different Hiera backbone to extract 3 sets of features into the detection models - naive, UCF and IIDD. The results are then concatenated in a fully connected layer to produce the classification scores.
To reduce the number of outputs sent to the user, faces from adjacent frames are grouped together if their bounding boxes are close to each other.
ADDITIONAL INFORMATION
For the first development phase, we focused on establishing the core functionalities - the home page, results page and supporting image detection capabilities. Our objective was to rapidly develop a prototype that allows us to test our idea with minimal risk and cost implications if pivots were necessary.
The focus for this phase is streamlining user interaction through the introduction of user authentication, the establishment of a user history page, and extending support to video inference, thereby significantly enhancing the overall accessibility and utility of our service.
In this phase, we focused on enhancing UI and UX, integrating a vision transformer for heatmap visualisation of model results, and implementing email notifications.
For our final development phase, we focused on creating a model ensemble to enhance the AUC score, backend tuning for efficiency, bug fixes, UI improvements, and code refactoring for AWS environment setup flexibility.
ACKNOWLEDGEMENTS
In collaboration with:
