MULTIMEDIA DEEPFAKE DETECTION (VIDEO AND AUDIO) WITH CROSS-MODAL INCONSISTENCY.

Abstract

Rapid advances in generative artificial intelligence have made manipulated talkingface videos increasingly realistic, inexpensive, and accessible. Such content may alter facial motion, speech, or both while preserving a convincing identity and overall appearance, creating risks for digital forensics, online trust, personal privacy, and the credibility of audio-visual evidence. Lip-sync manipulation is particularly difficult to detect because modifications are often concentrated around the mouth, while the remaining facial appearance, head movement, and background remain largely unchanged. Compression and post-processing may further weaken visible artifacts. In addition, the visual and acoustic streams may appear individually plausible even when subtle inconsistencies exist between speech and facial articulation. This thesis proposes a region-aware multimodal framework for detecting manipulated talking-face videos. Each input video is divided into synchronized threesecond windows containing head, face, and lip sequences together with the corresponding speech waveform. A shared EfficientNet-B3 encoder extracts regional visual representations, while Wav2Vec2-Base produces contextual acoustic representations. The framework combines local region-aware audio-visual interaction, global cross-modal attention, and explicit relation features to preserve visual, acoustic, and cross-modal manipulation evidence. Training is guided by the main binary classification objective, modality-specific auxiliary supervision, region-awareness regularization, and a realalignment contrastive objective. The decision threshold is selected exclusively from development data and remains fixed during final evaluation. The proposed system is designed to detect visual-only, audio-only, and joint audiovisual manipulations while producing a single Real or Fake decision. On the untouched test sets, the framework achieved ROC-AUC values of 91.20% on AVLips and 99.33% on FakeAVCeleb, with Balanced Accuracy values of 82.87% and 94.42%, respectively. The corresponding macro-average ROC-AUC, Balanced Accuracy, and EER were 95.27%, 88.65%, and 10.51%. The system also supports complete-video inference through temporal window aggregation and evidence visualization. Nevertheless, its outputs should be interpreted as model-based forensic evidence rather than definitive proof of manipulation, and further evaluation is required under broader real-world conditions. Sự phát triển của Generative AI làm gia tăng rủi ro từ các video nói chuyện bị giả mạo (talking-face deepfake), đặc biệt là giả mạo nhép môi (lip-sync) vốn khó phát hiện do biến đổi chỉ tập trung ở vùng miệng. Luận văn này đề xuất một khung phát hiện đa phương thức chú trọng vùng không gian (region-aware multimodal framework). Hệ thống chia video thành các cửa sổ 3 giây, sử dụng EfficientNet-B3 để trích xuất đặc trưng hình ảnh theo vùng và Wav2Vec2-Base cho âm thanh, kết hợp với các cơ chế chú ý liên phương thức (cross-modal attention) để phát hiện sự mất bất cân thích giữa hình ảnh và âm thanh. Trên dữ liệu thử nghiệm, mô hình đạt kết quả ấn tượng với ROC-AUC trung bình 95.27%, Balanced Accuracy 88.65% và EER 10.51% trên hai tập dữ liệu AVLips và FakeAVCeleb, hỗ trợ phân loại Real/Fake cũng như trực quan hóa bằng chứng giả mạo cho toàn bộ video.