Retrieval-augmented prompts for text-only image captioning

Jinbo Huang; Gang Xie; Li Cao

doi:10.1117/12.3045215

9 October 2024 Retrieval-augmented prompts for text-only image captioning

Jinbo Huang, Gang Xie, Li Cao

Proceedings Volume 13288, Fourth International Conference on Computer Graphics, Image, and Virtualization (ICCGIV 2024); 1328811 (2024) https://doi.org/10.1117/12.3045215
Event: Fourth International Conference on Computer Graphics, Image, and Virtualization (ICCGIV 2024), 2024, Chengdu, China

Abstract

Supervised image captioning methods have made significant progress, but high-quality human-annotated image-text paired datasets are costly to collect. Recently, pre-trained vision-language models, such as CLIP, have shown exceptional performance in cross-modal associations, offering novel solutions for image captioning, such as zero-shot captioning through purely textual training. However, the modality gap exists between image and text presents an obstacle to cross-modal alignment within text-only captioning. Moreover, insufficient visual understanding and over-reliance on textual data in training lead to hallucinations (e.g., object misidentification and inaccurate object counts), resulting in irrational captioning. To tackle these issues, this paper presents RAPCap, a text-only method with retrieval-augmented prompts for text-only image captioning. Specifically, RAPCap enhances the language model's understanding of images by incorporating similar captions obtained through retrieval augmentation, thereby alleviating hallucinations. During inference, RAPCap translates the image to textual space to bridge the modality gap. Experimental results demonstrate that RAPCap achieves a new state-of-the-art performance on the Flickr30k and performs competitively on the MSCOCO compared to previous zero-shot captioning.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Jinbo Huang, Gang Xie, and Li Cao "Retrieval-augmented prompts for text-only image captioning", Proc. SPIE 13288, Fourth International Conference on Computer Graphics, Image, and Virtualization (ICCGIV 2024), 1328811 (9 October 2024); https://doi.org/10.1117/12.3045215

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
9 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Education and training

Visualization

Visual process modeling

Data modeling

Information visualization

Image retrieval

Performance modeling

Show All Keywords

Keywords/Phrases

Search In:

Publication Years