Implementing a Metric¶
To implement a metric in MMEval
, you should implement a subclass of BaseMetric that overrides the add
and compute_metric
methods.
In the evaluation process, each metric will update self._results
to store intermediate results after each call of add
. When computing the final metric result, the self._results
will be synchronized between processes.
An example that implementing simple Accuracy
metric:
import numpy as np
from mmeval.core import BaseMetric
class Accuracy(BaseMetric):
def add(self, predictions, labels):
self._results.append((predictions, labels))
def compute_metric(self, results):
predictions = np.concatenate(
[res[0] for res in results])
labels = np.concatenate(
[res[1] for res in results])
correct = (predictions == labels)
accuracy = sum(correct) / len(predictions)
return {'accuracy': accuracy}
Use Accuracy
:
# stateless call
accuracy = Accuracy()
metric_results = accuracy(predictions=[1, 2, 3, 4], labels=[1, 2, 3, 1])
print(metric_results)
# {'accuracy': 0.75}
# Accumulate batch
for i in range(10):
predicts = np.random.randint(0, 4, size=(10,))
labels = predicts = np.random.randint(0, 4, size=(10,))
accuracy.add(predicts, labels)
metric_results = accuracy.compute()
accuracy.reset() # clear the intermediate results