MS COCO: Dataset Guide, API Setup, and Metrics

Q: What does the iscrowd flag mean in an annotation?

The iscrowd boolean flag indicates whether an annotation represents an individual object ( iscrowd=0 ) or a dense cluster of multiple objects ( iscrowd=1 ), such as a crowd of spectators. Single objects use polygon coordinate lists, whereas crowd instances use Run-Length Encoding (RLE). In benchmark evaluations, crowd regions are ignored during false positive calculation if a detection overlaps them, preventing models from being penalized for detecting individuals inside a labeled group.

The Microsoft Common Objects in Context (MS COCO) dataset is an established benchmark for object detection, instance segmentation, keypoint detection, and scene captioning in computer vision research. Developed to shift machine learning evaluation from isolated, centered objects to complex everyday scenes, COCO provides standard annotations across 80 object categories. This guide covers how to access the dataset, configure the official pycocotools Python API, inspect annotation schemas, evaluate detector performance using official metrics, and troubleshoot common integration issues.

What it is used for in research

In machine learning and computer vision research, MS COCO serves as a benchmark for evaluating multi-task scene understanding. Unlike earlier datasets that featured single, prominent objects against uncluttered backgrounds, COCO captures objects in their natural environments, with multiple overlapping instances, occlusions, and varied scale distributions. Introduced by Lin et al. (2014) at ECCV, the dataset supports five principal research tracks:

Object Detection: Localizing objects with axis-aligned bounding boxes across 80 thing classes, evaluated on mean Average Precision (mAP).
Instance Segmentation: Delineating individual object boundaries using pixel-level polygon masks or Run-Length Encoding (RLE), separating overlapping objects of the same category. For broader annotation workflows, see the guide to image annotation formats and tools.
Keypoint Detection: Identifying 17 anatomical keypoints on human subjects for 2D pose estimation, tracking joint visibility flags.
Panoptic and Stuff Segmentation: Unifying countable objects (things) with amorphous background regions (stuff such as sky, grass, pavement, and water).
Image Captioning: Generating descriptive natural language sentences, with five independent human annotations provided per image.

The dataset serves as the standard evaluation target for benchmark suites, detailed in the model evaluation suite definition, alongside classification benchmarks such as ImageNet. The standardized 2017 dataset split provides 118,287 training images, 5,000 validation images, 40,670 test images, and 123,403 unlabeled images, providing a rigorous foundation documented under the datasheet for datasets standard.

Install or access (official source only)

COCO images and annotations are hosted on Google Cloud Platform and direct web archives by the COCO Consortium. Researchers can access the dataset through cloud bucket synchronization or direct HTTPS downloads, and interface with the metadata using the official Python package.

Cloud and Web Access

For efficient data transfer without downloading multi-gigabyte monolithic archives, download splits directly from the Google Cloud Storage bucket using gsutil:

# Install Google Cloud SDK if not present, then synchronize specific splits
mkdir -p coco/val2017
gsutil -m rsync gs://images.cocodataset.org/val2017 coco/val2017

# Synchronize instance, caption, and keypoint annotations
mkdir -p coco/annotations
gsutil -m rsync gs://images.cocodataset.org/annotations coco/annotations

Direct HTTP archive downloads are also available directly from images.cocodataset.org:

2017 Train Images: http://images.cocodataset.org/zips/train2017.zip (118K images, 18 GB)
2017 Validation Images: http://images.cocodataset.org/zips/val2017.zip (5K images, 1 GB)
2017 Test Images: http://images.cocodataset.org/zips/test2017.zip (41K images, 6 GB)
2017 Train/Val Annotations: http://images.cocodataset.org/annotations/annotations_trainval2017.zip (241 MB)

Python API Installation

Install the official pycocotools library from PyPI into your virtual environment:

pip install pycocotools numpy

For pipelines integrating automated data loaders, see the walkthrough on Hugging Face Datasets integration.

Step-by-step task (numbered, with menu names or commands exactly as in the current docs)

The following seven-step workflow demonstrates how to download validation annotations, parse instance data, query specific categories, extract bounding boxes, and evaluate model predictions using pycocotools.

Download and extract annotation metadata: Create a project directory and retrieve the 2017 train/validation annotations package.
```
mkdir -p coco/annotations
curl -O http://images.cocodataset.org/annotations/annotations_trainval2017.zip
unzip annotations_trainval2017.zip -d coco/
```
This unpacks instances_train2017.json, instances_val2017.json, and corresponding files for captions and person keypoints.
Initialize the COCO API: In your Python script or notebook, instantiate the COCO object pointing to the instance annotation file.
```
from pycocotools.coco import COCO

annotation_path = "coco/annotations/instances_val2017.json"
coco = COCO(annotation_path)
```
On initialization, the API parses the JSON and builds in-memory index dictionaries (imgs, anns, cats, imgToAnns, and catToImgs).

Inspect and filter categories: Extract all 80 object categories or retrieve numeric category identifiers for specific targets.

# Retrieve all category records
cat_ids = coco.getCatIds()
categories = coco.loadCats(cat_ids)
print(f"Loaded {len(categories)} categories")

# Query specific categories by name
target_cat_ids = coco.getCatIds(catNms=["person", "bicycle"])
print(f"Target Category IDs: {target_cat_ids}")

Query images containing specific categories: Retrieve matching image IDs and inspect image metadata.

# Find images containing both target categories
img_ids = coco.getImgIds(catIds=target_cat_ids)
print(f"Matching images found: {len(img_ids)}")

# Load metadata for the first matching image
image_info = coco.loadImgs(img_ids[0])[0]
print(f"Image ID: {image_info['id']}")
print(f"File Name: {image_info['file_name']}")
print(f"Resolution: {image_info['width']} x {image_info['height']}")

Load ground truth annotations: Fetch all instance annotations belonging to the selected image and target classes.

ann_ids = coco.getAnnIds(imgIds=image_info["id"], catIds=target_cat_ids, iscrowd=None)
annotations = coco.loadAnns(ann_ids)

for ann in annotations:
    cat_name = coco.loadCats(ann["category_id"])[0]["name"]
    bbox = ann["bbox"]  # [x, y, width, height]
    print(f"Category: {cat_name} | BBox: {bbox} | Area: {ann['area']} | iscrowd: {ann['iscrowd']}")

Format model predictions: Export detection predictions as a JSON list matching the COCO results specification. Each detection requires image_id, category_id, bbox (in [x, y, width, height] format), and score.
```
# Example predictions structure (predictions.json)
[
  {
    "image_id": 397133,
    "category_id": 1,
    "bbox": [102.5, 45.2, 85.0, 192.4],
    "score": 0.942
  }
]
```

Execute standardized evaluation: Initialize the COCOeval evaluation class, execute IoU matching, accumulate statistics, and print the summary table.

from pycocotools.cocoeval import COCOeval

# Load predictions into a COCO detection object
coco_dt = coco.loadRes("predictions.json")

# Initialize evaluator with ground truth and detections
coco_eval = COCOeval(coco, coco_dt, iouType="bbox")

# Run evaluation protocol
coco_eval.evaluate()
coco_eval.accumulate()
coco_eval.summarize()

Output and how to read it

Working with MS COCO produces two primary outputs: the structured JSON annotation format and the 12-metric evaluation summary generated by COCOeval.

JSON Annotation Architecture

The annotation file is a single JSON dictionary with five primary keys: info, images, annotations, categories, and licenses. An instance annotation contains the following critical fields:

id (integer): Unique annotation identifier.
image_id (integer): Links to the image ID defined in the images array.
category_id (integer): Category identifier. Note that IDs range from 1 to 90, with 80 active categories and 11 unused slots.
bbox (array of 4 floats): Bounding box coordinates formatted as [x, y, width, height]. Coordinates are measured in pixels starting from the top-left corner (0-indexed).
segmentation: When iscrowd=0, this field contains an array of polygon coordinate pairs [[x1, y1, x2, y2, ...]]. When iscrowd=1, it contains a Run-Length Encoding (RLE) dictionary with counts and size keys.
area (float): Total pixel area enclosed by the segmentation mask.
iscrowd (0 or 1): Flag indicating whether the annotation represents a single object (0) or a dense cluster of multiple objects (1).

The 12-Metric Evaluation Matrix

The coco_eval.summarize() command computes and displays 12 metrics that characterize detector performance across localization stringencies and object scales:

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.425
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.612
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.463
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.231
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.470
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.584
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.345
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.551
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.589
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.338
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.771

To interpret these metrics accurately in research reports:

Primary Metric (AP @ IoU=0.50:0.95): Computed by calculating Average Precision across 10 IoU thresholds from 0.50 to 0.95 with a step size of 0.05, then averaging across all 80 categories. This rewards models that predict tight bounding boxes rather than loose approximations.
PASCAL VOC Metric (AP @ IoU=0.50): Measures precision at a permissive 50% overlap threshold.
Strict Metric (AP @ IoU=0.75): Evaluates precision when 75% overlap is required.
Scale Breakdown: COCO categorizes objects by pixel mask area into small (area < 32² pixels, ~41% of instances), medium (32² < area < 96² pixels, ~34% of instances), and large (area > 96² pixels, ~24% of instances).
Average Recall (AR): Evaluates recall limits when allowing 1, 10, or 100 detections per image.

Troubleshooting

Researchers encountering errors during setup, parsing, or evaluation should review the following solutions to common failure points.

1. Visual C++ Build Failures on Windows

Symptom: Running pip install pycocotools fails with error: Microsoft Visual C++ 14.0 or greater is required or C-extension compile errors.
Resolution: The C extensions in pycocotools require a C compiler. On Windows systems without Visual Studio build tools, install the community-maintained pre-compiled wheel via pip install pycocotools-windows, or install via conda using conda install -c conda-forge pycocotools.

2. Non-Contiguous Category ID Indexing

Symptom: Neural network models throw IndexError: Target 90 is out of bounds or produce misclassified labels during training.
Resolution: While COCO contains 80 classes, the category IDs range from 1 to 90 because 11 categories from the original 91-class set were omitted from the detection track. Deep learning loss functions require contiguous 0-indexed targets (0 to 79). Build an explicit bidirectional lookup map:

cat_ids = sorted(coco.getCatIds())
coco_to_model = {cat_id: i for i, cat_id in enumerate(cat_ids)}
model_to_coco = {i: cat_id for i, cat_id in enumerate(cat_ids)}

3. Zero AP Due to Incompatible Bounding Box Formats

Symptom: coco_eval.summarize() prints 0.000 across all AP and AR metrics despite functional model predictions.
Resolution: The COCO evaluator expects bounding boxes strictly in [x_min, y_min, width, height] pixel coordinates. If your model outputs two-corner coordinates [x_min, y_min, x_max, y_max] (PASCAL VOC format) or normalized center coordinates [x_center, y_center, width, height] (YOLO format), IoU overlap calculation fails. Convert your model predictions before serializing to JSON:

# Converting from [xmin, ymin, xmax, ymax] to COCO [xmin, ymin, width, height]
coco_bbox = [xmin, ymin, xmax - xmin, ymax - ymin]

4. Evaluation Out-of-Memory Errors

Symptom: The process crashes or is terminated by the OS during coco_eval.evaluate() when evaluating dense predictions across 40,000 images.
Resolution: By default, COCOeval constructs large multi-dimensional evaluation arrays in memory. To reduce RAM usage, filter low-confidence predictions (e.g., threshold score > 0.05) before exporting the result JSON, and cap detections at 100 per image.

Free alternatives

Several alternative open-access datasets support object detection and scene understanding benchmarks, each differing in domain focus, scale, and annotation depth.

Dataset	Primary Focus	Categories	Scale	Segmentation Masks	Official Access
MS COCO	Common objects in context; multi-task vision	80 thing, 91 stuff	118K train, 5K val	Polygons and RLE masks	Free open access (cocodataset.org)
PASCAL VOC	Baseline object detection and classification	20 classes	~11.5K images (VOC 2012)	Per-pixel semantic masks	Free academic access (host.robots.ox.ac.uk)
Open Images V7	Large-scale object detection and visual relationships	600 box classes	~9M images, 16M boxes	Instance segmentation masks	Free open access (storage.googleapis.com)
ImageNet (ILSVRC)	Image classification and visual recognition hierarchy	1,000 synsets	1.28M train images	Bounding boxes (subset)	Free academic registration (image-net.org)
ADE20K	Dense scene parsing and semantic segmentation	150 semantic concepts	20K train, 2K val	Full per-pixel semantic parsing	Free open access (groups.csail.mit.edu)

FAQ

What is the difference between MS COCO 2014 and 2017 splits?

The image contents are largely identical, but the partition sizes were restructured. The 2014 release divided images into 83,000 training images and 41,000 validation images. Because 41,000 validation images created unnecessary computational overhead for iterative benchmarking, the 2017 split shifted most validation images into the training set, resulting in 118,287 training images and a compact, standardized 5,000-image validation set (val2017). Current research should utilize the 2017 splits.

Does MS COCO use normalized or absolute coordinates for bounding boxes?

MS COCO stores absolute floating-point coordinates measured in pixels from the upper-left corner (0-indexed). The format is strictly [x_min, y_min, width, height]. It does not normalize values between 0.0 and 1.0, unlike formats used by YOLO architectures.

Why are there 80 categories when category IDs reach 90?

The original COCO dataset collection examined 91 object and stuff categories. During benchmark finalization, 11 categories that lacked sufficient instance segmentations were excluded from the core detection task. To maintain backward compatibility with early releases, the remaining 80 categories kept their original IDs (1 through 90), creating intentional gaps in the ID sequence.

What does the iscrowd flag mean in an annotation?

The iscrowd boolean flag indicates whether an annotation represents an individual object (iscrowd=0) or a dense cluster of multiple objects (iscrowd=1), such as a crowd of spectators. Single objects use polygon coordinate lists, whereas crowd instances use Run-Length Encoding (RLE). In benchmark evaluations, crowd regions are ignored during false positive calculation if a detection overlaps them, preventing models from being penalized for detecting individuals inside a labeled group.

Sources (official docs)

COCO Consortium. “COCO: Common Objects in Context — Data Format, Download, and Evaluation Guidelines.” URL: https://cocodataset.org/ (verified 2026-11-08).
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). “Microsoft COCO: Common Objects in Context.” European Conference on Computer Vision (ECCV 2014). arXiv:1405.0312. URL: https://arxiv.org/abs/1405.0312 (verified 2026-11-08).
Python Package Index (PyPI). “pycocotools: Official COCO Python API.” URL: https://pypi.org/project/pycocotools/ (verified 2026-11-08).
COCO Consortium GitHub Repository. “cocodataset/cocoapi: COCO API – Dataset, Format, and Evaluation.” URL: https://github.com/cocodataset/cocoapi (verified 2026-11-08).