COCODetection

class mmeval.metrics.COCODetection(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', iou_thrs: Optional[Union[float, Sequence[float]]] = None, classwise: bool = False, proposal_nums: Sequence[int] = (1, 10, 100), metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, gt_mask_area: bool = True, backend_args: Optional[dict] = None, print_results: bool = True, **kwargs)

COCO object detection task evaluation metric.

Evaluate AR, AP, and mAP for detection tasks including proposal/box detection and instance segmentation. Please refer to https://cocodataset.org/#detection-eval for more details.

Parameters

• ann_file (str, optional) – Path to the coco format annotation file. If not specified, ground truth annotations from the dataset will be converted to coco format. Defaults to None.

• metric (str | List[str]) – Metrics to be evaluated. Valid metrics include ‘bbox’, ‘segm’, and ‘proposal’. Defaults to ‘bbox’.

• iou_thrs (float | List[float], optional) – IoU threshold to compute AP and AR. If not specified, IoUs from 0.5 to 0.95 will be used. Defaults to None.

• classwise (bool) – Whether to return the computed results of each class. Defaults to False.

• proposal_nums (Sequence[int]) – Numbers of proposals to be evaluated. Defaults to (1, 10, 100).

• metric_items (List[str], optional) – Metric result names to be recorded in the evaluation result. Defaults to None.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• outfile_prefix (str, optional) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. Defaults to None.

• gt_mask_area (bool) – Whether to calculate GT mask area when not loading ann_file. If True, the GT instance area will be the mask area, else the bounding box area. It will not be used when loading ann_file. Defaults to True.

• backend_args (dict, optional) – Arguments to instantiate the preifx of uri corresponding backend. Defaults to None.

• print_results (bool) – Whether to print the results. Defaults to True.

• logger (Logger, optional) – logger used to record messages. When set to None, the default logger will be used. Defaults to None.

• **kwargs – Keyword parameters passed to BaseMetric.

Examples

>>> import numpy as np
>>> from mmeval import COCODetection
>>> try:
>>>     from mmeval.metrics.utils.coco_wrapper import mask_util
>>> except ImportError as e:
>>>     mask_util = None
>>>
>>> num_classes = 4
>>> fake_dataset_metas = {
...     'classes': tuple([str(i) for i in range(num_classes)])
... }
>>>
>>> coco_det_metric = COCODetection(
...     dataset_meta=fake_dataset_metas,
...     metric=['bbox', 'segm']
... )
>>> def _gen_bboxes(num_bboxes, img_w=256, img_h=256):
...     # random generate bounding boxes in 'xyxy' formart.
...     x = np.random.rand(num_bboxes, ) * img_w
...     y = np.random.rand(num_bboxes, ) * img_h
...     w = np.random.rand(num_bboxes, ) * (img_w - x)
...     h = np.random.rand(num_bboxes, ) * (img_h - y)
...     return np.stack([x, y, x + w, y + h], axis=1)
>>>
>>> def _gen_masks(bboxes, img_w=256, img_h=256):
...     if mask_util is None:
...         raise ImportError(
...             'Please try to install official pycocotools by '
...             '"pip install pycocotools"')
...     masks = []
...     for i, bbox in enumerate(bboxes):
...         mask = np.zeros((img_h, img_w))
...         bbox = bbox.astype(np.int32)
...         box_mask = (np.random.rand(
...             bbox[3] - bbox[1],
...             bbox[2] - bbox[0]) > 0.3).astype(np.int32)
...         mask[bbox[1]:bbox[3], bbox[0]:bbox[2]] = box_mask
...         masks.append(
...             mask_util.encode(
...                 np.array(mask[:, :, np.newaxis], order='F',
...                          dtype='uint8'))[0])  # encoded with RLE
...     return masks
>>>
>>> img_id = 1
>>> img_w, img_h = 256, 256
>>> num_bboxes = 10
>>> pred_boxes = _gen_bboxes(
...     num_bboxes=num_bboxes,
...     img_w=img_w,
...     img_h=img_h)
>>> pred_masks = _gen_masks(
...     bboxes=pred_boxes,
...     img_w=img_w,
...     img_h=img_h)
>>> prediction = {
...     'img_id': img_id,
...     'bboxes': pred_boxes,
...     'scores': np.random.rand(num_bboxes, ),
...     'labels': np.random.randint(0, num_classes, size=(num_bboxes, )),
...     'masks': pred_masks
... }
>>> gt_boxes = _gen_bboxes(
...     num_bboxes=num_bboxes,
...     img_w=img_w,
...     img_h=img_h)
>>> gt_masks = _gen_masks(
...     bboxes=pred_boxes,
...     img_w=img_w,
...     img_h=img_h)
>>> groundtruth = {
...     'img_id': img_id,
...     'width': img_w,
...     'height': img_h,
...     'bboxes': gt_boxes,
...     'labels': np.random.randint(0, num_classes, size=(num_bboxes, )),
...     'masks': gt_masks,
...     'ignore_flags': np.zeros(num_bboxes)
... }
>>> coco_det_metric(predictions=[prediction, ], groundtruths=[groundtruth, ])  
{'bbox_mAP': ..., 'bbox_mAP_50': ..., ...,
 'segm_mAP': ..., 'segm_mAP_50': ..., ...,
 'bbox_result': ..., 'segm_result': ..., ...}

add(predictions: Sequence[Dict], groundtruths: Sequence[Dict]) → None

Add the intermediate results to self._results.

Parameters

• predictions (Sequence[dict]) –

  A sequence of dict. Each dict representing a detection result for an image, with the following keys:

  • img_id (int): Image id.

  • bboxes (numpy.ndarray): Shape (N, 4), the predicted bounding bboxes of this image, in ‘xyxy’ foramrt.

  • scores (numpy.ndarray): Shape (N, ), the predicted scores of bounding boxes.

  • labels (numpy.ndarray): Shape (N, ), the predicted labels of bounding boxes.

  • masks (list[RLE], optional): The predicted masks.

  • mask_scores (np.array, optional): Shape (N, ), the predicted scores of masks.

• groundtruths (Sequence[dict]) –

  A sequence of dict. If load from ann_file, the dict inside can be empty. Else, each dict represents a groundtruths for an image, with the following keys:

  • img_id (int): Image id.

  • width (int): The width of the image.

  • height (int): The height of the image.

  • bboxes (numpy.ndarray): Shape (K, 4), the ground truth bounding bboxes of this image, in ‘xyxy’ foramrt.

  • labels (numpy.ndarray): Shape (K, ), the ground truth labels of bounding boxes.

  • masks (list[RLE], optional): The predicted masks.

  • ignore_flags (numpy.ndarray, optional): Shape (K, ), the ignore flags.

add_predictions(predictions: Sequence[Dict]) → None

Add predictions only.

If the ann_file has been passed, we can add predictions only.

Parameters

predictions (Sequence[dict]) – Refer to COCODetection.add.

property classes: list

Get classes from self.dataset_meta.

compute_metric(results: list) → dict

Compute the COCO metrics.

Parameters

results (List[tuple]) – A list of tuple. Each tuple is the prediction and ground truth of an image. This list has already been synced across all ranks.

Returns

The computed metric. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

gt_to_coco_json(gt_dicts: Sequence[dict], outfile_prefix: str) → str

Convert ground truth to coco format json file.

Parameters

• gt_dicts (Sequence[dict]) – Ground truth of the dataset.

• outfile_prefix (str) – The filename prefix of the json files. If the prefix is “somepath/xxx”, the json file will be named “somepath/xxx.gt.json”.

Returns

The filename of the json file.

Return type

str

results2json(results: Sequence[dict], outfile_prefix: str) → dict

Dump the detection results to a COCO style json file.

There are 3 types of results: proposals, bbox predictions, mask predictions, and they have different data types. This method will automatically recognize the type, and dump them to json files.

Parameters

• results (Sequence[dict]) – Testing results of the dataset.

• outfile_prefix (str) – The filename prefix of the json files. If the prefix is “somepath/xxx”, the json files will be named “somepath/xxx.bbox.json”, “somepath/xxx.segm.json”, “somepath/xxx.proposal.json”.

Returns

Possible keys are “bbox”, “segm”, “proposal”, and values are corresponding filenames.

Return type

dict

xyxy2xywh(bbox: numpy.ndarray) → list

Convert xyxy style bounding boxes to xywh style for COCO evaluation.

Parameters

bbox (np.ndarray) – The bounding boxes, shape (4, ), in xyxy order.

Returns

The converted bounding boxes, in xywh order.

Return type

list[float]