STRAP
STRuctured Annotations single Pass dataset

Klara Janouskova* ✉ Zuzana Moravcikova* Nikita Kisel Jiri Matas
Visual Recognition Group, Czech Technical University in Prague  ·  * Equal contribution
Paper Annotation Gallery Dataset Code

Region annotations for 2M web images, from one structured MLLM pass

2M images from DataComp-1B and CC3M, datasets commonly used for contrastive VLM training/finetuning. Annotated by an MLLM in one schema-constrained pass. We release the precomputed image annotations, as well as the data annotation pipeline, allowing others to scale up the dataset, bring it to another domain, or adapt the prompt to their needs.

What's different: region annotations are normally produced by chaining multiple models. Whichever direction the chain runs, the stages have to be glued together with hyperparameters — thresholds, matching rules — and descriptions can end up bound to the wrong region.

✕ Top-down: caption, then ground
  • Detail is capped by what fits in natural prose — a caption also carrying attributes, hierarchy and relations would be unreadable
  • Noun phrases have to be parsed out and matched to regions, so errors compound down the chain
✕ Bottom-up: detect, then describe
  • Bounded by whatever the detector predicts — class-agnostic proposals widen it, but anything not proposed is still never annotated
  • Regions are described independently, so nothing ties objects to each other or to the scene
✓ STRAP: one pass
  • One frozen MLLM writes every field at once, constrained to a JSON schema — nothing to stitch, no thresholds to tune
  • Open vocabulary; shared object ids keep every field bound to its own region, with relations and scene context alongside
Objects & boxes 3-level hierarchy labels Region captions Object interactions Scene context 1 JSON record per image
STRAP annotation pipeline: a frozen MLLM receives an image, a fixed prompt and a JSON schema, and emits one structured record per image — image-level fields plus one detailed card per object, bound by shared object ids.
Abstract

Overview

STRAP is a dataset of region–text annotations produced by a single structured pass of a frozen multimodal LLM. The model gets three inputs per image: the image, a fixed text prompt, and a fixed JSON schema for the output. It returns one machine-readable record that localizes and describes each object, their interactions, and the image as a whole. Decoding is constrained to the schema, so every record parses without post-editing.

STRAP covers 2M web images: 1M from CC3M and 1M from DataComp-1B, two datasets commonly used to train contrastive VLMs. Only their images are used, never their captions. The MLLM model is the openly available Qwen3.6-35B-A3B, so the run is exactly reproducible. The cost is one MLLM pass per image — the pipeline scales to far larger corpora. The prompt and schema are released with the dataset.

2M annotated web images 1M CC3M + 1M DataComp‑1B
1 structured pass of a frozen MLLM per image
40 GB one GPU runs the MLLM model open Qwen3.6-35B-A3B — exactly reproducible
10.4M annotated objects 6.2M CC3M + 4.1M DataComp‑1B
Annotation Pipeline

One Structured Pass

Any MLLM that can follow an output schema can generate STRAP records. The model is frozen: no fine-tuning, no iterative prompting, no post-processing. Commercial APIs and local inference frameworks alike support schema-constrained decoding, so the pipeline is a single pass end-to-end.

Common practice packs every instruction into the free-form prompt, gets free-form text back, and parses it heuristically. The output schema is the overlooked half of the interface: constrained decoding guarantees the structure, so the prompt only says what to annotate — never how to format it.

Prior Approaches vs. STRAP

Standard large-scale region-annotation pipelines chain multiple models into a cascade, either top-down or bottom-up. Both directions share two costs before their individual limits even come up: the stages have to be glued together with hyperparameters — score thresholds, NMS settings, matching rules — and every hand-off is a chance to bind a description to the wrong region.

① :(

Top-down: captions → grounding

Caption the image, then ground the noun phrases with a separate detector.

  • multiple models, compounding failure modes
  • caption parsing required, then each phrase has to be matched to a region
  • Detail is capped by natural language: a caption carrying per-object attributes, hierarchy and relations would be deeply unnatural text
② :(

Bottom-up: detections → region descriptions

Detect or segment regions first, then describe each one independently.

  • multiple models, compounding failure modes
  • Nothing links objects to each other or to the scene
  • Bounded by whatever the detector predicts. A class-agnostic proposal model lifts the vocabulary limit, but the annotation is still confined to what it proposes — anything it does not is never annotated
✓

STRAP — one structured pass

A single frozen MLLM emits every field together, constrained to the schema.

  • Boxes, labels, attributes, relations and scene context in one record
  • No stages to glue together: nothing to threshold, match or tune between models
  • Shared object ids keep the record consistent by construction, so no field is bound to the wrong region
  • Open vocabulary: the model decides what is in the image and describes it in full view of the scene
Input
Web image + fixed text prompt + fixed JSON schema
→
Frozen MLLM
Decoding constrained to the schema
→
One record per image ✓
Parses without post-editing; objects localized and described

Choosing the MLLM

We compared two MLLMs against Visual Genome ground truth: GPT-5.4 (closed API) and Qwen3.6-35B-A3B (open weights, ~35B parameters, runs locally on a single 40 GB A100 GPU). The open model localizes better; the closed one is slightly ahead on semantics. While a promptable segmenter like SAM3 can mitigate the localization limitations of GPT-5.4 (or extend Qwen annots with masks), the cost makes the choice clear: a paid API is prohibitively expensive for 2M images. STRAP is therefore generated with Qwen3.6-35B-A3B, which also makes the run exactly reproducible.

Qwen3.6-35B-A3B (chosen) — open weights, better localization (immediately usable without detector/segmenter refinements), runs on a single 40 GB A100, reproducible, scales to 2M images GPT-5.4 — closed API, slightly better semantics, prohibitively expensive at 2M-image scale
Annotation Schema

What a Record Contains

Each record carries two kinds of fields. All fields of one object share an object id, so each stays bound to its region.

①
Image-level fields

A short summary of the image, the scene context, and the interactions between objects.

②
Object-level fields

For each object: a normalized bounding box, a basic-level label, coarser and finer hierarchy levels, and a caption-style description of the region.

The basic level is the one people name by default — dog rather than animal or retriever. The coarser and finer hierarchy levels sit on either side of it, giving each object a three-level semantic hierarchy.

Web photo of a horse race: the leading jockey in lime green and blue silks rides a dark horse, with more jockeys and horses behind. Boxes 1 and 2 mark the jockey and horse described in the record. 1 2 
{
  "image_summary": "A dynamic action shot of a horse race, featuring several jockeys on their horses galloping towards the viewer. …",
  "scene_context": "An outdoor horse racing track during a race, with a blurred background of spectators and track infrastructure.",
  "objects": [
    { "id": 1, "basic_label": "jockey", "salience": 0.95, "visibility": "fully_visible",
      "brief_description": "Jockey in lime green and blue silks riding a dark horse in the foreground.",
      "location": { "bounding_box": [0.60, 0.07, 0.85, 0.88], … },
      "hierarchy": { "coarser_levels": [{ "label": "person", … }, { "label": "human", … }], … },
      "intrinsic_parts": ["head", "torso", "left_arm", "right_arm", "left_hand", …],
      "associated_items": [{ "item": "light green helmet", "relation": "worn_on", "covers": ["head"], … }, …],
      "text_on_object": [{ "text": "AIR MURYS", … }], … },
    { "id": 2, "basic_label": "horse", "salience": 0.9, … },
    … 7 more objects
  ],
  "interactions": [
    { "object_ids": [1, 2], "kind": "riding", "note": "Jockey 1 is riding horse 2.", … }, …
  ],
  …
}

Excerpt of a real record. Abridged from the record for the horse-race example in the gallery (CC3M image 003111502); ellipses mark elided fields. The exact prompt and JSON schema are released with the dataset and shown in full in the examples gallery.

The schema is designed to cover a wide range of object-centric data — from hierarchy, pose and state to multi-object interactions and OCR — spanning common use cases. It is not fixed: the community can easily extend the annotation pipeline to other use cases by adapting the prompt and the output schema.

Annotation Examples

See the Records

Real web images with their complete STRAP records — boxes, labels, descriptions and scene context — exactly as the MLLM model produced them.

Open the Annotation Gallery
Data

The STRAP Release

2M structured records Annotation prompt JSON output schema Open MLLM model (Qwen3.6-35B-A3B) Annotation pipeline

Images come from two standard contrastive-VLM training corpora: CC3M (1M) and DataComp-1B (1M). STRAP uses only their images — every word of text in the dataset is generated by the MLLM model.

Dataset Summary

cc3mdatacomp
Images (rows)1,000,0001,000,000
Annotated objects6,229,1954,121,158
Objects per image (mean)6.24.1
Image sourceCC3MDataComp-1B
Primary join keytsv_index (row in Google's CC3M TSV)uid (DataComp-1B sample ID)
MLLM modelQwen3.6-35B-A3B — the same open model for both subsets

Join keys. STRAP ships annotations only. Each record carries the key of its source sample, so the images are fetched from CC3M or DataComp-1B and joined locally.

The release also includes the annotation pipeline itself. Point it at data of your choice, adapt the prompt, and use any MLLM that supports structured JSON output.

Licensing. The annotations — the records, the annotation prompt, and the JSON schema — are released under CC BY 4.0, and the annotation-pipeline code under the MIT License. The source images are not redistributed and remain under the terms of their original collections (CC3M, DataComp-1B).
Using STRAP? We would love to hear from you. Your experience with the data, the cool stuff you built with it, ideas for extending the schema — drop us a line or start a discussion on the Hugging Face community tab.
Paper & Citation

Coming Soon

Paper coming soon. STRAP is introduced as the supervision behind SEAR-VLM, a method for turning a pretrained VLM into a region encoder. The preprint will be posted on arXiv soon and linked here.
Preprint · on arXiv soon

SEAR-VLM: Semantic Embedding for Any Region
Learned with Structured MLLM Annotations

Klara Janouskova*, Zuzana Moravcikova*, Nikita Kisel, Jiri Matas

We introduce SEAR-VLM, which turns a pretrained vision–language model (VLM) into a region encoder by conditioning its pooling head on the query region. Each image is encoded once, after which the supplied boxes or masks restrict pooling attention over its cached patch tokens to produce embeddings in the pretrained image–text space.

To supply its supervision, we introduce the STRAP (STRuctured Annotations single Pass) dataset. A single pass of an open-weight multimodal LLM (MLLM) returns a machine-readable record that jointly localizes and describes the objects in each image, avoiding multi-stage annotation pipelines. We annotate 2M web images with no human labels.

When finetuning only the pooling head, SEAR-SigLIP2 improves over the frozen SigLIP2 readout on five fine-grained benchmarks by 15.6 percentage points on average, and by 11.8 percentage points on seven open-vocabulary region-recognition benchmarks. We also apply SEAR-VLM to DINOv3 and CLIP, outperforming all prior CLIP-based baselines.

BibTeX

If you use this work, please cite:

@article{janouskova2026sear,
  author  = {Janouskova, Klara and Moravcikova, Zuzana and Kisel, Nikita and Matas, Jiri},
  journal = {arXiv preprint},
  title   = {SEAR-VLM: Semantic Embedding for Any Region Learned with Structured MLLM Annotations},
  year    = {2026}
}