STRAP Annotation Examples

Analyze the input image and return a structured inventory of visible objects with hierarchical classification, confidence scores, spatial location, salience-based detail tiering, contextual typicality, confusable alternatives, structured attributes, interactions, readable text, and part-level annotation for salient objects.

Instructions:

Object identification
- Identify all objects that are visible or partially visible in the image, including background elements.
- List only independently meaningful objects. Parts of a larger visible object (e.g., a car wheel, a table leg) belong in that object's intrinsic_parts field, not as separate entries — unless they are independently salient objects in the scene in their own right.
- Assign each object a unique integer id starting from 1.
- If an object is only partly visible, note which parts are visible, whether it is occluded by another object or out of frame. When occluded by another listed object, record that object's id in occluded_by_id.

Detail tiering by salience
- Assign each object a salience score (0.0 to 1.0): 1.0 = clear main subject, near 0.0 = minor background element.
- High-salience objects (salience >= 0.5): provide full detail — hierarchy, confusable classes, context typicality, rich attributes, intrinsic_parts, and associated_items.
- Low-salience objects (salience < 0.5): provide minimal detail — basic_label, location, salience, visibility, and brief attributes (colors and pose_or_state only). Set hierarchy, confusable_with, context_typicality, intrinsic_parts, and associated_items to null / empty.

Intrinsic parts and associated items (salient objects only)
For objects with salience >= 0.5, provide:
- intrinsic_parts: structurally integral, clearly visible components of the object — listed as plain strings.
- associated_items: items worn, carried, mounted, or otherwise added to the object, but not structural parts of it — listed as structured objects.

Rules for intrinsic_parts:
- List only clearly visible parts — do not hallucinate hidden, occluded, or assumed parts.
- Keep parts reasonably coarse and natural for the object category; avoid overly fine-grained decomposition.
- Include only parts that belong to this specific object instance.
- Treat structural components as intrinsic_parts (e.g., head, arm, wheel, door, petal, trunk).
- Do not include clothing, accessories, or carried items here — those go in associated_items.
- Use consistent snake_case part names (e.g., left_arm, nose_bridge, rear_wheel).
- Typical examples by category:
  - person: head, torso, left_arm, right_arm, left_hand, right_hand, left_leg, right_leg
  - plant: leaves, stem, flower, petals, trunk, branches
  - car: front_wheel, rear_wheel, door, window, windshield, mirror, hood
  - chair: seat, backrest, legs, armrest

Rules for associated_items:
- Each entry is an object with four fields:
  - item: descriptive name of the item, including visible color or type when relevant (e.g., "red t-shirt", "leather backpack").
  - relation: how the item is associated — use one of: worn_on, carried_on, carried_in_hand, mounted_on, attached_to, tied_to, held_by.
  - covers: list of intrinsic part names (from this object's intrinsic_parts) that the item sits on, covers, or is in contact with (e.g., ["torso", "upper_arms"]). Use an empty array when no specific parts apply.
  - object_id: if this item is also annotated as a top-level object with its own id, set this to that id; otherwise null.
- For people: clothing, jewelry, accessories, or carried items (e.g., "red t-shirt" worn_on [torso, upper_arms], "backpack" carried_on [upper_back], "glasses" worn_on [eyes, nose_bridge]).
- For vehicles: mounted accessories (e.g., "roof rack" mounted_on [roof], "bike rack" attached_to [rear]).
- For animals: collar, leash, harness, saddle (e.g., "leather collar" worn_on [neck]).
- Do not include nearby unrelated objects that are not directly associated with this object instance.
- Typical examples by category:
  - person: red t-shirt (worn_on torso), backpack (carried_on upper_back), glasses (worn_on eyes/nose_bridge)
  - car: roof rack (mounted_on roof), bike rack (attached_to rear)
  - dog: collar (worn_on neck), leash (attached_to neck)
  - plant: support stick (attached_to stem)

Cross-referencing with object_id in associated_items:
- When a listed associated item is also annotated as a top-level object (with its own id in the objects list), set object_id to that object's id.
- Example: a person's backpack annotated separately as object id 7 → {"item": "black backpack", "relation": "carried_on", "covers": ["upper_back"], "object_id": 7}.
- Example: a person's glasses not listed as a separate object → {"item": "glasses", "relation": "worn_on", "covers": ["eyes", "nose_bridge"], "object_id": null}.
- Avoid duplicating items as separate top-level objects unless they are independently salient in the scene.

Grouping
- When two or more similar objects form a natural group (e.g., a pair of shoes, three sheep grazing together, a row of chairs), give each its own entry with individual attributes and location, but assign them the same group_id. A group requires at least 2 objects.
- Collapse into a single entry with count > 1 only when 5 or more identical instances cannot be meaningfully distinguished individually (e.g., a large flock of birds). The location then covers the whole group.

Renditions
- If an object is a depiction or representation of something rather than the real thing (e.g., a painting of a dog, a bronze sculpture of a horse, a plush toy cat, a crochet flower), set its rendition field with:
  - medium: the physical form — "painting", "sculpture", "plush toy", "crochet", "figurine", "drawing", "photograph", "mural", "embroidery", "mosaic", "relief", etc.
  - depicts: what concept or subject it represents — "dog", "horse", "landscape", "Abraham Lincoln", etc.
- For renditions, basic_label should be what the object physically is (e.g., "painting", "sculpture", "plush toy"), and the hierarchy classifies the physical form. What it depicts goes in rendition.depicts.
- For real objects (not renditions), set rendition to null.

Basic label
- Provide a basic_label for every object: the name most people would use first ("what is this?"). Examples: "chair", "dog", "car", "mug".
- For renditions, the basic_label is the physical medium: "painting", "statue", "plush toy".
- Use common English, not scientific names. Each label is a single noun phrase.
- If uncertain whether a term is basic vs. finer: "Would most people say this without prompting?" If yes, it is basic.
- Example: image shows a common house cat. WRONG: "tabby cat" (finer). RIGHT: "cat".

Hierarchical naming (salient objects only)
For objects with salience >= 0.5, provide a hierarchy with variable-depth levels above and below the basic label:
- coarser_levels: one or more levels above basic, sorted from nearest parent to broadest. Example for "dog": [{"label": "mammal", "confidence": 0.99}, {"label": "animal", "confidence": 0.99}]. Include as many levels as are informative; stop when a level becomes too generic to be useful.
- finer_levels: zero or more levels below basic, sorted from nearest child to most specific. Example for "dog": [{"label": "Golden Retriever", "confidence": 0.85}]. Add levels only when visually supportable. Empty array when no finer distinction is apparent.
- basic_confidence: confidence in the basic_label (0.0–1.0).
- confidence_note: optional explanation when confidence scores need clarification or when basic vs. finer level assignment is ambiguous.
- Assign confidence scores based on visual certainty, not prior knowledge.
- Use common English terms at every level. Each label is a single noun phrase. Use Latin or scientific names only when they are the standard way to refer to a category and no common English equivalent exists (e.g., "Canidae" for the dog family, genus names for plants).
For low-salience objects, set hierarchy to null.

Recognition basis
For each object:
- recognition_requires_text: true when identity is only determinable by reading text on or near it (e.g., a generic can identified as "Coca-Cola" only by its label).
- recognition_requires_context: true when identity is only determinable from surrounding scene context (e.g., a metal instrument identified as a "stethoscope" because the scene is a hospital).
Both can be false or both true simultaneously.

Confusable classes (salient objects only)
For objects with salience >= 0.5, list plausible visually similar alternatives a reasonable observer might consider, with reasons. For low-salience objects, use an empty array.

Location
For each object:
- bounding_description: textual location (e.g., "center-left foreground").
- anchor_point: [x, y] in normalized image coordinates (0.0–1.0, (0,0) = top-left). Place this point on the most visually central part of the object's surface. For concave or irregularly shaped objects (e.g., an L-shaped sofa, a person with arms outstretched), pick the point on the largest solid region of the object, near its visual mass center. For objects extending beyond the image boundary, the point must lie within the visible portion.
- bounding_box: [x_min, y_min, x_max, y_max] axis-aligned rectangle tightly enclosing the visible portion of the object.
Prioritize approximate correctness over false precision.

Contextual typicality (salient objects only)
For objects with salience >= 0.5: is_typical_context = true if the object is in a normal setting; false if unusual (explain in context_note). For low-salience objects, set to null.

Attributes
Record visible attributes as a structured object per object:
- colors: dominant visible colors.
- materials: visible material cues (wood, metal, glass, fabric, plastic, etc.).
- pose_or_state: pose or state (sitting, standing, open, closed, parked, blooming, etc.).
- size_cue: relative size when informative (e.g., "large", "small relative to nearby chair").
- expression: facial expression if applicable.
- markings: visible patterns, logos, or markings.
- clothing: worn items if applicable.
- condition: wear, damage, or age cues.
- other: any other directly observable traits not covered above.
For low-salience objects, only colors and pose_or_state are required; other fields may be empty or null.

Text
- Text on a specific object goes into that object's text_on_object field. Extract exactly when possible.
- Text visible but not tied to one object goes into global_text.

Interactions
- Identify pairs or groups of objects that visually interact.
- Describe the interaction kind (holding, riding, sitting_on, contained_in, playing, chasing, leaning_against, stacked_on, standing_next_to, etc.).
- Specify directionality: from_object_id → to_object_id. For symmetric interactions (next to, facing each other), set is_directional to false, both ids to null, and explain in note.
- Only report visually evident interactions.

Scene context
- Provide a brief description of the overall scene setting, environment, and background in scene_context (e.g., "sunny suburban street", "dimly lit restaurant interior", "studio portrait with white backdrop"). Set to null if the scene is too ambiguous or featureless to characterize.

Uncertain items
Place ambiguous objects or interpretations in uncertain_or_ambiguous_items.

Output rules:
- Include only what is visible in the image.
- Return valid JSON only.
- Follow the JSON schema exactly.
- Do not add extra keys.
- Use null where the schema allows null and the detail is not visually supportable.
- Set intrinsic_parts and associated_items to empty arrays for low-salience objects (salience < 0.5).

{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "image_summary": {
      "type": [
        "string",
        "null"
      ],
      "description": "Short visual summary of the image."
    },
    "scene_context": {
      "type": [
        "string",
        "null"
      ],
      "description": "Brief description of the overall scene setting, environment, and background (e.g., 'sunny suburban street', 'dimly lit restaurant interior', 'studio portrait with white backdrop'). Null if the scene is too ambiguous or featureless to characterize."
    },
    "objects": {
      "type": "array",
      "description": "Visible objects in the image.",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "id": {
            "type": "integer",
            "description": "Unique identifier starting from 1."
          },
          "count": {
            "type": "integer",
            "minimum": 1,
            "description": "Number of instances. >1 only for indistinguishable identical groups."
          },
          "group_id": {
            "type": [
              "integer",
              "null"
            ],
            "description": "Shared id linking objects that form a natural group. Null if not grouped."
          },
          "salience": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0,
            "description": "Prominence: 1.0 = main subject, near 0.0 = minor background."
          },
          "brief_description": {
            "type": "string",
            "description": "Short phrase to distinguish this object from others, e.g. 'the red chair in the foreground', 'large brown dog on the left'."
          },
          "basic_label": {
            "type": "string",
            "description": "Basic-level name most people would use first: chair, dog, car, mug."
          },
          "rendition": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": false,
                "description": "Present when the object is a depiction or representation (painting, sculpture, plush toy, etc.) rather than the real thing.",
                "properties": {
                  "medium": {
                    "type": "string",
                    "description": "Physical form of the rendition: painting, sculpture, plush toy, crochet, figurine, drawing, photograph, mural, embroidery, mosaic, relief, etc."
                  },
                  "depicts": {
                    "type": "string",
                    "description": "What concept or subject it represents: dog, horse, landscape, Abraham Lincoln, etc."
                  }
                },
                "required": [
                  "medium",
                  "depicts"
                ]
              },
              {
                "type": "null"
              }
            ],
            "description": "Null for real objects. For renditions, basic_label is the physical medium and depicts captures the subject."
          },
          "recognition_requires_text": {
            "type": "boolean",
            "description": "True when identity is only determinable by reading text on or near the object."
          },
          "recognition_requires_context": {
            "type": "boolean",
            "description": "True when identity is only determinable from surrounding scene context."
          },
          "hierarchy": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": false,
                "description": "Variable-depth naming hierarchy above and below the basic label. Provided for salient objects (salience >= 0.5).",
                "properties": {
                  "coarser_levels": {
                    "type": "array",
                    "description": "Levels above basic, sorted from nearest parent to broadest. E.g. for 'dog': [{label: 'mammal', confidence: 0.99}, {label: 'animal', confidence: 0.99}].",
                    "items": {
                      "type": "object",
                      "additionalProperties": false,
                      "properties": {
                        "label": {
                          "type": "string",
                          "description": "Category name at this level."
                        },
                        "confidence": {
                          "type": "number",
                          "minimum": 0.0,
                          "maximum": 1.0,
                          "description": "Visual certainty for this label."
                        }
                      },
                      "required": [
                        "label",
                        "confidence"
                      ]
                    }
                  },
                  "basic_confidence": {
                    "type": "number",
                    "minimum": 0.0,
                    "maximum": 1.0,
                    "description": "Confidence in the basic_label."
                  },
                  "finer_levels": {
                    "type": "array",
                    "description": "Levels below basic, sorted from nearest child to most specific. Can range from broad subcategories to extremely fine-grained identifications such as a specific car model, plant cultivar, bird species, or furniture product line. E.g. for 'dog': [{label: 'Golden Retriever', confidence: 0.85}]; for 'car': [{label: 'sedan', confidence: 0.95}, {label: 'Tesla Model 3', confidence: 0.7}]. Empty array if no finer distinction is apparent.",
                    "items": {
                      "type": "object",
                      "additionalProperties": false,
                      "properties": {
                        "label": {
                          "type": "string",
                          "description": "More specific name at this level."
                        },
                        "confidence": {
                          "type": "number",
                          "minimum": 0.0,
                          "maximum": 1.0,
                          "description": "Visual certainty for this label."
                        }
                      },
                      "required": [
                        "label",
                        "confidence"
                      ]
                    }
                  },
                  "confidence_note": {
                    "type": [
                      "string",
                      "null"
                    ],
                    "description": "Explanation when confidence scores need clarification or basic vs. finer level assignment is ambiguous."
                  }
                },
                "required": [
                  "coarser_levels",
                  "basic_confidence",
                  "finer_levels",
                  "confidence_note"
                ]
              },
              {
                "type": "null"
              }
            ],
            "description": "Full naming hierarchy for salient objects. Null for low-salience objects."
          },
          "confusable_with": {
            "type": "array",
            "description": "Plausible alternative classes based on visual similarity. Empty for low-salience objects.",
            "items": {
              "type": "object",
              "additionalProperties": false,
              "properties": {
                "class": {
                  "type": "string",
                  "description": "The alternative class label."
                },
                "reason": {
                  "type": "string",
                  "description": "Why this alternative is plausible."
                }
              },
              "required": [
                "class",
                "reason"
              ]
            }
          },
          "context_typicality": {
            "anyOf": [
              {
                "type": "object",
                "additionalProperties": false,
                "description": "Whether this object appears in a typical context. Provided for salient objects (salience >= 0.5).",
                "properties": {
                  "is_typical_context": {
                    "type": "boolean",
                    "description": "True if the object is in a normal, expected setting."
                  },
                  "context_note": {
                    "type": [
                      "string",
                      "null"
                    ],
                    "description": "Explanation when context is atypical. Null when typical."
                  }
                },
                "required": [
                  "is_typical_context",
                  "context_note"
                ]
              },
              {
                "type": "null"
              }
            ],
            "description": "Null for low-salience objects."
          },
          "location": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
              "anchor_point": {
                "type": "array",
                "description": "[x, y] in normalized coordinates (0.0–1.0, (0,0) = top-left). Placed on the most visually central part of the object's surface. For concave or irregular shapes, on the largest solid region near its visual mass center. Must lie within the visible portion for objects extending beyond the image boundary.",
                "minItems": 2,
                "maxItems": 2,
                "items": {
                  "type": "number",
                  "minimum": 0.0,
                  "maximum": 1.0
                }
              },
              "bounding_box": {
                "type": "array",
                "description": "[x_min, y_min, x_max, y_max] enclosing rectangle in normalized coordinates.",
                "minItems": 4,
                "maxItems": 4,
                "items": {
                  "type": "number",
                  "minimum": 0.0,
                  "maximum": 1.0
                }
              }
            },
            "required": [
              "anchor_point",
              "bounding_box"
            ]
          },
          "bounding_description": {
            "type": [
              "string",
              "null"
            ],
            "description": "Textual location such as 'center-left foreground'."
          },
          "visibility": {
            "type": "string",
            "enum": [
              "fully_visible",
              "mostly_visible",
              "partially_visible",
              "small_or_distant",
              "occluded"
            ],
            "description": "How visible the object is in the image."
          },
          "occluded_by_id": {
            "type": [
              "integer",
              "null"
            ],
            "description": "Id of the object that occludes this one, if applicable."
          },
          "intrinsic_parts": {
            "type": "array",
            "description": "Structurally integral, clearly visible components of this object, listed as plain part-name strings. Provided for salient objects (salience >= 0.5). Empty array for low-salience objects or when no parts are distinctly visible. Use snake_case names (e.g. 'left_arm', 'rear_wheel', 'nose_bridge').",
            "items": {
              "type": "string",
              "description": "Part name, e.g. 'head', 'torso', 'left_arm', 'rear_wheel', 'door', 'petal'."
            }
          },
          "associated_items": {
            "type": "array",
            "description": "Items worn, carried, mounted, or otherwise added to this object that are not structural parts of it. Provided for salient objects (salience >= 0.5). Empty array for low-salience objects or when no associated items are visible.",
            "items": {
              "type": "object",
              "additionalProperties": false,
              "properties": {
                "item": {
                  "type": "string",
                  "description": "Descriptive name of the item, including visible color or type when relevant, e.g. 'red t-shirt', 'black backpack', 'leather collar', 'roof rack'."
                },
                "relation": {
                  "type": "string",
                  "enum": [
                    "worn_on",
                    "carried_on",
                    "carried_in_hand",
                    "mounted_on",
                    "attached_to",
                    "tied_to",
                    "held_by"
                  ],
                  "description": "How the item is associated with the object."
                },
                "covers": {
                  "type": "array",
                  "description": "Intrinsic part names (from this object's intrinsic_parts) that the item sits on, covers, or is in contact with. Empty array when no specific parts apply.",
                  "items": {
                    "type": "string"
                  }
                },
                "object_id": {
                  "type": [
                    "integer",
                    "null"
                  ],
                  "description": "Id of the corresponding top-level annotated object if this item is also listed independently in the objects array; otherwise null."
                }
              },
              "required": [
                "item",
                "relation",
                "covers",
                "object_id"
              ]
            }
          },
          "attributes": {
            "type": "object",
            "additionalProperties": false,
            "description": "Structured visible attributes. For low-salience objects, only colors and pose_or_state are required; other fields may be empty arrays or null.",
            "properties": {
              "colors": {
                "type": "array",
                "description": "Dominant visible colors.",
                "items": {
                  "type": "string"
                }
              },
              "materials": {
                "type": "array",
                "description": "Visible material cues (wood, metal, glass, fabric, plastic, etc.).",
                "items": {
                  "type": "string"
                }
              },
              "pose_or_state": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "Pose or state: sitting, standing, open, closed, parked, blooming, etc."
              },
              "size_cue": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "Relative size when informative, e.g. 'large', 'small relative to nearby chair'."
              },
              "expression": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "Facial expression if applicable."
              },
              "markings": {
                "type": "array",
                "description": "Visible patterns, logos, or markings.",
                "items": {
                  "type": "string"
                }
              },
              "clothing": {
                "type": "array",
                "description": "Worn items if applicable.",
                "items": {
                  "type": "string"
                }
              },
              "condition": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "Wear, damage, or age cues."
              },
              "other": {
                "type": "array",
                "description": "Other directly observable traits not covered above.",
                "items": {
                  "type": "string"
                }
              }
            },
            "required": [
              "colors",
              "materials",
              "pose_or_state",
              "size_cue",
              "expression",
              "markings",
              "clothing",
              "condition",
              "other"
            ]
          },
          "text_on_object": {
            "type": "array",
            "description": "Readable text on this object.",
            "items": {
              "type": "object",
              "additionalProperties": false,
              "properties": {
                "text": {
                  "type": "string"
                },
                "location_on_object": {
                  "type": [
                    "string",
                    "null"
                  ],
                  "description": "Where text appears: shirt front, sign top line, mug side, etc."
                },
                "legibility": {
                  "type": "string",
                  "enum": [
                    "clear",
                    "partial",
                    "uncertain"
                  ]
                }
              },
              "required": [
                "text",
                "location_on_object",
                "legibility"
              ]
            }
          }
        },
        "required": [
          "id",
          "brief_description",
          "count",
          "group_id",
          "salience",
          "basic_label",
          "rendition",
          "recognition_requires_text",
          "recognition_requires_context",
          "hierarchy",
          "confusable_with",
          "context_typicality",
          "location",
          "bounding_description",
          "visibility",
          "occluded_by_id",
          "intrinsic_parts",
          "associated_items",
          "attributes",
          "text_on_object"
        ]
      }
    },
    "interactions": {
      "type": "array",
      "description": "Pairwise or multi-object interactions visible in the image.",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "object_ids": {
            "type": "array",
            "description": "IDs of the interacting objects. Minimum 2.",
            "minItems": 2,
            "items": {
              "type": "integer"
            }
          },
          "kind": {
            "type": "string",
            "description": "Interaction type: holding, riding, sitting_on, contained_in, etc."
          },
          "direction": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
              "from_object_id": {
                "type": [
                  "integer",
                  "null"
                ],
                "description": "Agent/source of the interaction. Null when non-directional."
              },
              "to_object_id": {
                "type": [
                  "integer",
                  "null"
                ],
                "description": "Patient/target of the interaction. Null when non-directional."
              },
              "is_directional": {
                "type": "boolean",
                "description": "False for symmetric interactions like 'next to'."
              }
            },
            "required": [
              "from_object_id",
              "to_object_id",
              "is_directional"
            ]
          },
          "note": {
            "type": [
              "string",
              "null"
            ],
            "description": "Clarification for ambiguous directionality or multi-object interactions."
          }
        },
        "required": [
          "object_ids",
          "kind",
          "direction",
          "note"
        ]
      }
    },
    "global_text": {
      "type": "array",
      "description": "Readable text not tied to a specific object.",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "text": {
            "type": "string"
          },
          "location": {
            "type": [
              "string",
              "null"
            ],
            "description": "Approximate location such as 'top-center poster'."
          },
          "legibility": {
            "type": "string",
            "enum": [
              "clear",
              "partial",
              "uncertain"
            ]
          }
        },
        "required": [
          "text",
          "location",
          "legibility"
        ]
      }
    },
    "uncertain_or_ambiguous_items": {
      "type": "array",
      "description": "Objects or interpretations too ambiguous for confident inclusion.",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "description": {
            "type": "string"
          },
          "possible_classes": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "reason": {
            "type": [
              "string",
              "null"
            ],
            "description": "Why the item is uncertain: blur, occlusion, low resolution, etc."
          }
        },
        "required": [
          "description",
          "possible_classes",
          "reason"
        ]
      }
    }
  },
  "required": [
    "image_summary",
    "scene_context",
    "objects",
    "interactions",
    "global_text",
    "uncertain_or_ambiguous_items"
  ]
}