Objects in the real world are often distinguished not only by their visual appearance, but also by the text printed on them. Locating players by their jersey numbers in sports, navigating highways with traffic signs, or identifying medicine vials based on the dosage, all require linking textual cues with the correct object instance for robust perception. Recent progress in vision and language considers scene-text recognition and object recognition as two independent tasks and overlooks a critical capability of object–text binding. To address this gap, we introduce Read the Object, a carefully curated benchmark for evaluating instance-level associations between text and objects in natural, real-world images. The proposed benchmark explicitly and rigorously annotates each object with its corresponding text, including visual attributes and spatial cues to disambiguate multi-instance scenes. Within this unified framework, we define six complementary tasks for evaluating text-grounded perception and reasoning. Experiments on state-of-the-art models reveal a clear binding gap: despite strong performance on isolated text recognition and global reasoning, they struggle to associate text with the correct object instance in complex scenes. Read the Object establishes a new standard for bridging reading, perception, and reasoning in vision–language research.
Average of 5.44 text regions and 1.35 objects per sample — over 90% of samples contain more than one scene-text instance, with a long tail to 11–50 regions per image. Per-class imbalance factor stays ≤4.
One dataset, six views on the same underlying capability — associating text with the correct object instance.
Given a text string, find all objects in the image that contain or are associated with that text.
Given a natural-language expression combining text and visual attributes, locate the described object and its text region.
Detect and recognize all visible text regions in the image.
Given a query text string, localize the corresponding text region(s) within the image.
Answer questions that require reading text and understanding its relation to the visual context.
Generate a caption that jointly describes objects and the visible text on them.
Prior REC datasets ground objects only by appearance and spatial cues. Read the Object additionally annotates scene-text regions and links them explicitly to object instances.
| Dataset | Images | Samples | Avg. Expr. Len. | Scene-text bbox | Object–text assoc. | Visual attr. |
|---|---|---|---|---|---|---|
| RefCOCO | 3,000 | 7,596 | 3.6 | — | — | — |
| RefCOCO+ | 3,000 | 7,578 | 3.6 | — | — | — |
| RefCOCOg | 3,900 | 7,596 | 8.4 | — | — | — |
| Ref-Adv | 2,833 | 5,000 | 11.5 | — | — | ✓ |
| Read the Object | 6,000 | 25,208 | 17.6 | ✓ | ✓ | ✓ |
Open-vocabulary detection, segmentation, and OCR extract candidates; text is linked to objects via mask-guided association, then verified through multi-round human review.
We present the importance of joint understanding of scene-text and objects for real-world visual reasoning. Read the Object introduces dense scene-text and object annotations that establish relationships between objects and their associated scene text. Our comprehensive analysis — evaluating diverse vision, OCR, and multimodal models across six complementary tasks — reveals that current object grounding and vision-language models struggle significantly with text–object association, especially in cluttered environments. We believe this benchmark will steer research toward multimodal models that can navigate and understand the text-rich complexity of the real world.
@inproceedings{chowdhury2026readtheobject,
title = {Read the Object: A Dataset and Task for Text-Grounded Object Understanding in Real-World Images},
author = {Chowdhury, Arpita and Park, Hyojin and Le, Hoang and Chao, Wei-Lun and Hayat, Munawar and Porikli, Fatih and Mahajan, Shweta},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}