IMAGE CAPTIONING USING INCEPTION-V4, LSTM MODELS AND BERT EMBEDDINGS

Abstract

Image captioning uses image recognition techniques and natural language processing models to generate captions of photos. In this thesis, we perform experiments with several models to automatically create descriptions for images using the Inception-v4, LSTM Models and BERT Embeddings. In particular, the Inception-v4 model extracts image features later fed into the LSTM and BERT Embeddings model to generate image captions. We perform experiments on the Flickr8k dataset in English and Vietnamese and evaluate the models using the BLEU metric. The experimental results show that combining the Inception-v4, LSTM Models and BERT Embeddings helps achieve better BLEU scores than others. The experimental results show that the combination of Inception-v4, LSTM Models and BERT Embeddings help achieve better BLEU scores than other models. The BLEU1, 2, 3, and 4 scores of the Inception-v4, LSTM Models and BERT Embeddings on the English and Vietnamese Flickr8k datasets are 0.689, 0.479, 0.3649, 0.267; and 0.647, 0.501, 0.332, 0.271 respectively.