Wide spread adpontion of high-powered smart-computing devices, like smartphones, has profoundly impacted various aspects of our lives, including law enforcement and legal proceedings. These devices enable us to create, edit, analyze, and share multimedia files, driving significant technological advancements. Nowadays, digital multimedia, such as audio, video, and images, has become the standard form of evidence in litigation and criminal justice cases. However, ensuring the authenticity and integrity of this media poses significant challenges. To be admissible as evidence, digital media must be proven authentic, meaning it provides a complete record of events since its creation. Additionally, its integrity must be verified, involving the detection of any disruptions or intentional manipulations in the recording, such as insertions or compressions. This verification process becomes even more difficult when there is a lack of supporting data, like digital watermarks, and when the media has been altered with anti-forensic intent. Furthermore, the availability of powerful and user-friendly digital editing, open-source availability of generative AI algorithms and manipulation tools has further complicated the authentication and integrity verification of digital audio files.

The goal of this project is to investigate microphone and acoustic environment signatures in the evidentiary recording, to investigate the impact of anti-forensic attacks on existing methods, to design robust attack-aware algorithms against anti-forensic attacks, and to develop a common platform for benchmarking audio forensics algorithms and tools. These objectives have been achieved through the creation and implementation of a following diverse range of methodologies, tools, and collaborative research platforms: 

1. In this project, we have developed several novel methods to improve the capture of microphone signatures from voice replay, voice cloning, and the algorithms used for their synthesis. We have also focused on understanding the dynamic speech variations in genuine signals and distinguishing them from spoofed samples. 
2. Additionally, our research has shed light on the vulnerabilities of automated speaker verification systems to AI-cloned voices and multi-order replay spoofing. Furthermore, we have explored the impact of deepfake, and recorded audio injected through LASER into MEMS microphones, and we have devised countermeasures to detect such attacks.
3. Moreover, we have specifically crafted liveness detection techniques tailored for the purpose of speaker verification. 
4. Unlike other single attack-specific countermeasures, we have developed two novel unified anti-spoofing methods that can handle various attacks on automated speaker verification using a single descriptor.
5. The existing countermeasures against AI-generated voices are unable to reliably detect unseen samples or audio generated by new AI algorithms. Therefore, we have developed a novel spoofing transformer network called SpotNet. 
6. Furthermore, we have developed a secure and lightweight automated speaker verification system that not only reliably authenticates speakers/users but also demonstrates resistance against multiple audio spoofing attacks. Moreover, this system possesses the capability to capture the signature of voice cloning algorithms. 
7. In addition, we have developed deep learning-based methods for detecting both audio and video deepfakes by capturing temporal, spatial, and inter-domain inconsistencies. Furthermore, we have created the first-ever neuro-symbolic deepfake detection framework that leverages the observation that deepfakes often exhibit inter- or intra-modality inconsistencies in the emotional expressions of the manipulated individuals. 
8. We have also developed a framework for securing voice-controlled devices/services, e.g., Amazon Alexa, Google Home, Apple Siri, etc., against emerging threat vectors including LASER injection  attacks.
9. We have created a publicly available voice spoofing detection corpus (VSDC) that includes bonafide samples as well as first-order and second-order replay samples.
10. We have developed a research commons platform that facilitates apple-to-apple comparative analysis of state-of-the-art voice spoofing countermeasures. 
11. We have also investigated the impact of anti-forensic attacks on automatic speaker verification (ASV) and voice countermeasures. This investigation involved using voice samples obtained through facemasks and compressed audio samples, among other techniques.
12. Additionally, we investigated the impact of adversarial attacks, including mole or other perturbation-based attacks, on audio-visual deepfake detection systems, and 
13. We have developed an approach based on game theory to enhance the security of audio-visual deepfake detection systems

The intellectual merit of this project is its interdisciplinary approach that spans multiple domains, including cybersecurity, audio/speech processing, digital forensics, applied probability theory, and large-scale experimentation with audio data. At its core, this project involves the development of novel algorithms capable of accurately modeling and extracting the unique characteristics of acquisition devices, effectively addressing anti-forensic attacks, assessing system performance in the face of such attacks, designing forensic detectors that are resilient to these attacks, and curating comprehensive datasets for benchmarking and platform development purposes.

This project has made substantial contributions with broader impacts spanning multiple domains including multimedia forensics, content integrity verificaion, civil and criminal proceedings, national security, the fintech industry, law enforcement, cyberspace, voice-activated services, and the entertainment industry.

Last Modified: 06/21/2023
Modified by: Hafiz Malik