Doomed to Re-Annotate, Forever:
The ImageNet Story

Overview

Top-1 accuracy on ImageNet‑1k remains the most commonly reported metric in visual recognition. Quality issues with the dataset have been repeatedly reported, yet the original 2012 noisy labels are still predominantly used. The paper presents a comprehensive effort, which goes well beyond prior correction attempts, towards obtaining accurate and complete ImageNet‑1k validation set annotations. The result, ReImageNet, includes multilabel correction, object localization, revised class definitions, and semantic attributes (text-recognition, rendition, reflection, crowd, dominant). The reannotation reveals that ≈12% of the original ImageNet‑1k labels are incorrect, 33.3% of images are multilabel and 3.8% contain no object from an ImageNet‑1k class. With the new labels, top-1 accuracy increases by up to 1.2% for supervised models and by 5–6% for MLLMs. We argue that annotation at ImageNet scale cannot realistically be completed in one pass, as errors and definitional issues are discovered only through annotating, and we build our pipeline around repeated refinement and error checking. We observed that human and LLM collaboration with appropriate tooling represents the current quality ceiling for annotation at this scale. ImageNet‑1k issues propagate into its derivative test sets, indicating that the problem is structural rather than specific to any single benchmark. All annotations, class definitions, and guidelines have been publicly released; analysis code follows soon.

What Reannotation Looks Like

Real ImageNet‑1k validation images. Each card shows the original ImageNet label and the new ReImageNet annotation.

Original computer mouse

ReImageNet computer mouse · space bar · computer keyboard · sweatshirt or hoodie · mobile phone · T-shirt · window covering · desk

Correct but incomplete — 8 objects present

Original pier

ReImageNet suspension bridge · cliff

Labeling error — not a pier

Original barbershop

ReImageNet barbershop · television · typewriter · vase · wall clock · digital clock

Correct but incomplete — television is a reflection in mirror

Original Saharan horned viper

ReImageNet sidewinder rattlesnake

Fine-grained correction — confirmed via reverse image search

Bounding box colors encode semantic attributes (see Dataset section). Browse all 50,000 annotations →

What We Contribute

The ReImageNet Dataset

Every object that belongs to one of the 1,000 classes and stands out immediately when the image is viewed gets a bounding box, a label, and optional semantic attributes. Annotators label what they notice on first viewing rather than every object they could find — the original protocol asked for the latter, but objects spanning a few dozen pixels, recognizable only from context and only after zooming in, made the label set depend on how hard each annotator chose to look. Annotators follow a structured per-class preparation protocol — studying images, revising class definitions, resolving ambiguities — before any image-level labeling begins.

Image Partition

Image partition by label agreement between ReGT and original labels (ImGT). ~12% of images have an incorrect original label (S⁻ + M⁻ + N).

Multilabel Rate Comparison

Multilabel rate compared to prior reannotation efforts. ReImageNet's rate is substantially higher, reflecting exhaustive labeling of all immediately visible objects.

Evaluation Protocol

All results use ReaL accuracy (Beyer et al., 2020) as the scoring function: a prediction is correct if it matches any ground-truth label for the image. For semantically equivalent ImageNet‑1k class pairs (e.g. the two distinct classes both named laptop computer), either class in the pair is accepted as correct — this is applied to both corrected and original labels to ensure a fair comparison. The 3.8% of images that carry no valid ImageNet‑1k label are left out of the evaluation entirely — no prediction can be right on an image that contains nothing to recognize — leaving 48,114 images in the denominator, for both the corrected and the original labels.

MLLMs Score 5–6% Higher With Corrected Labels and Proper Evaluation Protocol

If you've been reporting ImageNet‑1k accuracy for an MLLM, its score is understated. Supervised models gain far less — the gain nearly vanishes for the strongest of them (EVA-02: +0.05%). On the hard subsets S⁻ (single-label images where the original label is wrong) and M⁻ (multilabel images where the original label is absent), MLLMs score 40–46% — ahead of VLMs (35–43%) and supervised models (30–36%). Language-vision training gives MLLMs a clear edge on genuinely ambiguous images that pure supervised models struggle with.

Top-1 accuracy on original labels (ImGT) vs. corrected labels (ReGT), with S⁻ and M⁻ hard-image accuracy.

Semantic Attributes

Five attributes mark objects with specific visual properties, enabling targeted evaluation of model capabilities that standard top-1 accuracy blends into a single number. Two of them shape the evaluation protocol — dominant identifies the object a single-label prediction should match, crowd marks groups of instances covered by one box — while rendition, reflection, and text-recognition isolate specific model capabilities. Click any card for an example image.

Rendition

The depicted object appears as a toy, drawing, or other stylized representation.

The depicted object visually mimics a class instance but appears as a toy, drawing, or other artificial or stylized representation. Useful for measuring model generalisation and robustness.

rendition crowd text-recognition reflection dominant no attribute

Crowd

Five or more instances of the same class appear in the image.

Five or more instances of the same class appear in the image. To reduce annotation effort, they are collapsed into a single bounding box. Instance-level counting is not supported for images with this attribute.

rendition crowd text-recognition reflection dominant no attribute

Text-recognition

Classification of the object requires reading text visible in the image.

Classification of the object requires reading text visible in the image. Tests whether models can leverage visible text as a recognition cue.

rendition crowd text-recognition reflection dominant no attribute

Reflection

The object appears in a mirror or water surface.

The object appears in a mirror or water surface, annotated independently of whether the reflected object itself is also visible. Measures model robustness to reflected appearances and affects instance counting.

rendition crowd text-recognition reflection dominant no attribute

Dominant

Assigned to objects that a person notices immediately upon viewing the image.

Assigned to objects that a person notices immediately upon viewing the image, based on annotator judgment rather than size alone. An object may be the only ImageNet class and still not be dominant.

Both Sealyham Terriers are dominant

Crown bottle cap — only class, not dominant

rendition crowd text-recognition reflection dominant no attribute

Accuracy by Semantic Attribute

Rendition degrades every model, but unevenly: MLLMs cope best (−1.3% to −2.3%) while supervised models fall hardest (EffNetV2: −9.8%, DINOv3: −7.3%) — language supervision generalizes to stylized depictions better than supervised training alone. Text-recognition shows the sharpest split: language-aligned models gain +6–16% by leveraging visible text, while pure supervised models suffer (EffNetV2: −6.7%). Crowd helps most models (up to +3%), with SigLIP2-g the lone small exception. Reflection is positive overall and strongest for VLMs (+1.8% to +3.7%), while GPT-5.4, Gemma 4, and EffNet-L2 dip slightly.

Accuracy change by semantic attribute relative to single-label (S) accuracy baseline.

Object-Centric Evaluation Protocols

ReImageNet's bounding boxes unlock object-centric evaluation. We cut a tight crop around each of the 96,051 boxes (avg. 33% of the image area) and evaluate them in four settings. Removing scene context reveals a stark gap: supervised models lose up to 33% accuracy on tight crops, while MLLMs are substantially more robust — reflecting their diverse training data. The gap between Any crop and Largest crop shows that in ~10% of cases a model misses the largest object in an image yet classifies a smaller, less salient one correctly — which questions the standard center-crop evaluation. ExCrops recover ~10% over tight crops while remaining backward-compatible with single-label evaluation.

Accuracy under crop-based evaluation. Δ relative to full-image accuracy.

Errors Don't Stay in ImageNet

Label noise in ImageNet‑1k propagates directly into derivative test sets, amplified at 1.7–5.8× the original rate. If you evaluate on ImageNet-A or ImageNet-Sketch, 13–15% of the labels you score against are wrong. Any benchmark built on the ImageNet‑1k taxonomy inherits its annotation flaws by construction — the problem is structural, not specific to any single benchmark.

Estimated annotation error rates in ImageNet-derived benchmarks.

The Quality Ceiling for Large-Scale Annotation

ReImageNet was produced by 7 trained in-house annotators following a structured per-class preparation protocol, supported by detector and MLLM predictions as optional reference. Continuous communication uncovered edge cases throughout, requiring repeated mid-annotation revisions of guidelines, class definitions, and attributes. A case study found that when GPT-4o disagreed with trained annotators, the annotators were wrong approximately 50% of the time — confirming that MLLMs and humans are complementary, and that reliable annotation at this scale is inherently iterative.

Because a single pass cannot catch everything, the current release is accompanied by an active second verification phase: annotators revisit already-annotated images against the now-finalised guidelines, with anonymized MLLM+SAM predictions, first-pass annotations, and original ImageNet labels as optional reference. Only minor refinements are expected and the published results are representative of the final dataset.

We do not claim the annotations are error-free. Residual inter-annotator variance remains: which objects count as immediately visible, where exactly to draw a bounding box, and which instances carry the dominant attribute all involve judgment calls.

BibTeX

If you use this work, please cite:

@article{volkov2026imagenet,
  title   = {Doomed to Re-Annotate, Forever:
             The {ImageNet} Story},
  author  = {Volkov, Illia and Kisel, Nikita and
             Mishkina, Tetiana and
             Janouskova, Klara and Matas, Jiri},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.13783}
}