Exploring Auto-Annotation Using Image-Caption Pairs and Deep Reinforcement Learning

Abstract

Automatic image annotation is important in many computer vision applications, such as autonomous vehicles, medical imaging, intelligent monitoring and robotics. Traditional annotation relies on manually labeled datasets, which makes the process long, costly and difficult. Although vision-language systems have improved referring expression grounding, most methods use the same fixed strategy for localization. To address this limitation, this paper introduces a Deep Reinforcement Learning-based Automatic Image Annotation Framework that combines Detectron2, VisualBERT and an actor–critic reinforcement learning mechanism. Detectron2 first detects regions of interest; VisualBERT then merges the visual and textual information, enabling the actor–critic agent to refine the localization using an IOU-based reward. On the RefCOCO dataset, the framework achieved average accuracies of 49.43% on the validation split and 46.70% on the Test A and Test B splits, with maximum accuracies of 95.16%, 82.96% and 99.01%, respectively, showing that it can associate the textual and visual components of an image.