DEVELOPING A WEB APPLICATION FOR MUSIC GENERATION FROM VIDEO USING NATURAL LANGUAGE AS INTERMEDIARY USING OPEN SOURCE AI MODELS

Abstract

This report presents a language-mediated framework for video-to-music generation, aiming to automatically generate background music that is semantically aligned with video content. Unlike conventional approaches that directly fuse visual and audio features, the proposed system employs natural language as an intermediate representation to bridge video understanding and music generation. The framework integrates scene segmentation, video captioning, music feature inference, and prompt-based music generation using open-source models, enabling improved interpretability and controllability. Experimental evaluation is conducted on a subset of the SymMV dataset, with vocals removed to focus on background music. The system is assessed using both audio quality metrics and cross-modal video-music relationship metrics based on ImageBind embeddings. Results show that the proposed approach achieves strong global semantic alignment and consistent temporal correspondence between video and generated music, outperforming a state-ofthe-art baseline in several semantic alignment metrics. Although beat-level synchronization remains limited, the generated music exhibits stable spectral characteristics suitable for background accompaniment. Overall, the results demonstrate that natural language serves as an effective intermediary modality for semantically coherent video-to-music generation in web-based and interactive applications.