OIDMeanAP¶
- class mmeval.metrics.OIDMeanAP(iof_thrs: Union[float, List[float]] = 0.5, use_group_of: bool = True, get_supercategory: bool = True, filter_labels: bool = True, class_relation_matrix: Optional[numpy.ndarray] = None, **kwargs)[source]¶
Open Images Dataset detection evaluation metric.
The Open Images Dataset detection evaluation uses a variant of the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5. There are some key features of Open Images annotations, which are addressed by the new metric.
For more see: https://storage.googleapis.com/openimages/web/evaluation.html
- Parameters
iof_thrs (float | List[float]) – IoF thresholds. Defaults to 0.5.
use_group_of (bool) – Whether consider group of groud truth bboxes during evaluating. Defaults to True.
get_supercategory (bool, optional) – Whether to get parent class of the current class. Defaults to True.
filter_labels (bool, optional) – Whether filter unannotated classes. Defaults to True.
class_relation_matrix (numpy.ndarray, optional) – The matrix of the corresponding relationship between the parent class and the child class. If None, it will be obtained from the ‘relation_matrix’ field in
self.dataset_meta
. Defaults to None.**kwargs – Keyword parameters passed to
VOCMeanAP
.
Examples
>>> import numpy as np >>> from mmeval import OIDMeanAP >>> num_classes = 4 >>> # use a fake relation_matrix >>> relation_matrix = np.eye(num_classes, num_classes) >>> oid_map = OIDMeanAP( ... num_classes=4, class_relation_matrix=relation_matrix) >>> >>> def _gen_bboxes(num_bboxes, img_w=256, img_h=256): ... # random generate bounding boxes in 'xyxy' formart. ... x = np.random.rand(num_bboxes, ) * img_w ... y = np.random.rand(num_bboxes, ) * img_h ... w = np.random.rand(num_bboxes, ) * (img_w - x) ... h = np.random.rand(num_bboxes, ) * (img_h - y) ... return np.stack([x, y, x + w, y + h], axis=1) >>> >>> prediction = { ... 'bboxes': _gen_bboxes(10), ... 'scores': np.random.rand(10, ), ... 'labels': np.random.randint(0, num_classes, size=(10, )) ... } >>> instances = [] >>> for bbox in _gen_bboxes(20): ... instances.append({ ... 'bbox': bbox.tolist(), ... 'bbox_label': random.randint(0, num_classes - 1), ... 'is_group_of': random.randint(0, 1) == 0, ... }) >>> groundtruth = { ... 'instances': instances, ... 'image_level_labels': np.random.randint(0, num_classes, size=(10, )), # noqa: E501 ... } >>> oid_map(predictions=[prediction, ], groundtruths=[groundtruth, ]) {'AP@50': ..., 'mAP': ...}
- add(predictions: Sequence[dict], groundtruths: Sequence[dict]) → None[source]¶
Add the intermediate results to
self._results
.- Parameters
predictions (Sequence[dict]) –
A sequence of dict. Each dict representing a detection result for an image, with the following keys:
bboxes (numpy.ndarray): Shape (N, 4), the predicted bounding bboxes of this image, in ‘xyxy’ foramrt.
scores (numpy.ndarray): Shape (N, 1), the predicted scores of bounding boxes.
labels (numpy.ndarray): Shape (N, 1), the predicted labels of bounding boxes.
groundtruths (Sequence[dict]) –
A sequence of dict. Each dict representing a groundtruths for an image, with the following keys:
instances (List[dict]): Each dict representing a bounding box with the following keys:
bbox (list): Containing box location in ‘xyxy’ foramrt.
bbox_label (int): Box label index.
is_group_of (bool): Whether the box is group or not.
image_level_labels (numpy.ndarray): The image level labels.
- calculate_class_tpfp(predictions: List[dict], groundtruths: List[dict], class_index: int, pool: Optional[multiprocessing.pool.Pool]) → Tuple[source]¶
Calculate the tp and fp of the given class index.
This is an overridden method of
VOCMeanAP
.- Parameters
predictions (List[dict]) – A list of dict. Each dict is the detection result of an image. Same as
VOCMeanAP.add
.groundtruths (List[dict]) – A list of dict. Each dict is the ground truth of an image. Same as
VOCMeanAP.add
.class_index (int) – The class index.
pool (Pool, optional) – An instance of class:multiprocessing.Pool. If None, do not use multiprocessing.
- Returns
tp (numpy.ndarray): Shape (num_ious, num_scales, num_pred), the true positive flag of each predict bbox.
fp (numpy.ndarray): Shape (num_ious, num_scales, num_pred), the false positive flag of each predict bbox.
num_gts (numpy.ndarray): Shape (num_ious, num_scales), the number of ground truths.
- Return type
tuple (tp, fp, num_gts)
- property class_relation_matrix: numpy.ndarray¶
Returns the class relation matrix.
The class relation matrix should be set during initialization, otherwise it will be obtained from the ‘relation_matrix’ field in
self.dataset_meta
.- Returns
The class relation matrix.
- Return type
numpy.ndarray
- Raises
RuntimeError – If the class relation matrix is not set.
- get_class_gts(groundtruths: List[dict], class_index: int) → Tuple[source]¶
Get prediciton gt information of a certain class index.
This is an overridden method of
VOCMeanAP
.- Parameters
groundtruths (list[dict]) – Same as
self.add
.class_index (int) – Index of a specific class.
- Returns
gt bboxes.
- Return type
List[np.ndarray]