MAITE Evaluation Tutorial¶
A guide to measuring a model's baseline performance with MaiteEvaluation via CheckMAITE.
Who this is for: Data Scientist choosing among candidate models for a project.
Where this fits: stage 3, Evaluate the model, of the T&E workflow. Run it on the dataset curated in stage 2, before NRTK or XAITK. The baseline score on unperturbed data is the reference that robustness and explainability results are read against.
What you'll learn
- How to evaluate one model on one dataset with several metrics in a single run
- How to read the overall and per-class results
- How to filter detections before scoring, and why that does not invalidate cached predictions
- How to render a report and extract records for the Analytics Store
NOTE:
MaiteEvaluationcan be used in the CheckMAITE framework for both image classification (IC) and object detection (OD) tasks. This tutorial will only cover the OD scenario.
What is MaiteEvaluation?¶
MAITE defines common protocols for datasets, models, and metrics, and an evaluate workflow that runs a model over a dataset in batches and scores its predictions against the dataset's ground truth.
MaiteEvaluation is the CheckMAITE capability built on that workflow. It answers the first question asked of any candidate model: how well does it perform on our data, as it is?
The CheckMAITE MaiteEvaluation Capability¶
In short, the MaiteEvaluation capability works as follows:
- Specify one MAITE-compatible model, one dataset, and one or more metrics. Each metric must have a unique
metadata["id"]. - Optionally configure the run with
MaiteEvaluationConfig:batch_size- the number of dataset items passed to the model per inference call.- For object detection only, CPU postprocessing of the model's detections:
confidence_threshold,nms_iou_threshold,class_agnostic_nms, andmax_detections. All are disabled by default, so predictions reach the metrics unchanged.
- Execute the capability, which will:
- Run inference once and score the predictions with every metric
- Return one result per metric, keyed by metric ID, including per-class values when the metric produces them
- Generate a report and Analytics Store records from those results
Keep in mind these two considerations when using MaiteEvaluation.
Results are cached by default. With
use_cache=True, a repeated run with the same model, dataset, metrics, and configuration loads its earlier result instead of recomputing it. This tutorial passesuse_cache=Falseso every cell recomputes.The scores are only as meaningful as the dataset. The demonstration below uses a three-image test subset, so its values illustrate the workflow and say nothing about the model.
Demonstration¶
In plain language, we will measure how well a VisDrone object detector finds objects in a small set of aerial images, overall and for each object class.
First, we create the MAITE-wrapped objects - a dataset, a compatible model, and two metrics.
A load_visdrone_detection_dataset wrapper is generated using data found in our test directory, a three-image subset of the VisDrone test dataset.
Then a VisdroneODModel wrapper with resnet18 architecture is generated using a ResNet model with weights pre-trained by Kitware, Inc. The weights are downloaded on first use.
Finally, two metric factories each generate a mean average precision (mAP) metric at an IoU threshold of 0.5:
map50_torch_metric_factoryreturns the overall mAP@50 under the keymap_50.multiclass_map50_torch_metric_factoryalso returns mAP@50 for every class, under the keymap_50_classwise.
from pathlib import Path
from checkmaite.core.object_detection.dataset_loaders import load_visdrone_detection_dataset
from checkmaite.core.object_detection.metrics import (
map50_torch_metric_factory,
multiclass_map50_torch_metric_factory,
)
from checkmaite.core.object_detection.models import VisdroneODModel
BASE_DIR = Path.cwd().parents[1]
dataset_root_path = BASE_DIR / "tests/data_for_tests/visdrone_dataset"
print("Loading Visdrone dataset...")
dataset = load_visdrone_detection_dataset(root=dataset_root_path)
print(f"Dataset loaded with {len(dataset)} images")
model = VisdroneODModel(arch="resnet18", model_pickle_dir="visdrone_model_weights")
overall_metric = map50_torch_metric_factory()
classwise_metric = multiclass_map50_torch_metric_factory()
print("Metric IDs:", overall_metric.metadata["id"], classwise_metric.metadata["id"])
/home/runner/work/checkmaite/checkmaite/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Loading Visdrone dataset... Dataset loaded with 3 images
/home/runner/work/checkmaite/checkmaite/.venv/lib/python3.10/site-packages/smqtk_detection/impls/detect_image_objects/centernet.py:897: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
@autocast()
/home/runner/work/checkmaite/checkmaite/.venv/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py:54: UserWarning: CUDA is not available or torch_xla is imported. Disabling autocast.
super().__init__(
Metric IDs: mAP_50_285766a2 mAP_50_classwise_ae4dd8b7
Next, we initialize a MaiteEvaluation object, pass it the dataset, model, and both metrics wrapped above, and run it.
Inference runs once. Every metric scores the same predictions.
from checkmaite.core.object_detection import MaiteEvaluation, MaiteEvaluationConfig
capability = MaiteEvaluation()
baseline_config = MaiteEvaluationConfig(batch_size=2)
baseline_run = capability.run(
use_cache=False,
datasets=[dataset],
models=[model],
metrics=[overall_metric, classwise_metric],
config=baseline_config,
)
0%| | 0/2 [00:00<?, ?it/s]
/home/runner/work/checkmaite/checkmaite/.venv/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: Encountered more than 100 detections in a single image. This means that certain detections with the lowest scores will be ignored, that may have an undesirable impact on performance. Please consider adjusting the `max_detection_threshold` to suit your use case. To disable this warning, set attribute class `warn_on_many_detections=False`, after initializing the metric. warnings.warn(*args, **kwargs) 50%|█████ | 1/2 [00:07<00:07, 7.02s/it]
100%|██████████| 2/2 [00:09<00:00, 4.59s/it]
100%|██████████| 2/2 [00:09<00:00, 4.96s/it]
Results¶
baseline_run.outputs.metrics holds one MaiteMetricResult per metric, keyed by metric ID. Each result has:
overall_metric_nameandoverall_metric_value- the metric's headline key and valuescalar_values- every numeric top-level value the metric returnedclass_metrics- per-class values, orNoneif the metric does not produce them. A class in the model's label set that does not appear in the dataset has the valueNone.
for metric_id, result in baseline_run.outputs.metrics.items():
print(f"{metric_id}: {result.overall_metric_name} = {result.overall_metric_value:.4f}")
classwise_result = baseline_run.outputs.metrics[classwise_metric.metadata["id"]]
print("\nPer-class mAP@50:")
for class_name, value in classwise_result.class_metrics.items():
print(f" {class_name:<16} {'not in dataset' if value is None else f'{value:.4f}'}")
mAP_50_285766a2: map_50 = 0.1628 mAP_50_classwise_ae4dd8b7: map_50_classwise = 0.1628 Per-class mAP@50: ignored regions 0.0000 pedestrian 0.1603 people 0.1758 bicycle 0.2222 car 0.7041 van 0.0227 truck 0.0000 tricycle 0.0058 awning-tricycle 0.2176 bus not in dataset motor 0.1193 others not in dataset
Filtering Detections Before Scoring¶
Detectors often return many low-confidence or overlapping boxes. For object detection, MaiteEvaluationConfig can filter each image's detections on the CPU before they reach the metrics:
- confidence_threshold – Minimum confidence required to keep a detection.
- nms_iou_threshold – IoU threshold for non-maximum suppression; disabled when unset.
- class_agnostic_nms – Whether non-maximum suppression also suppresses overlapping boxes with different labels; requires
nms_iou_threshold. - max_detections – Maximum number of highest-confidence detections to keep per image, applied after the other filters.
The filters run on the raw predictions after inference. With use_cache=True, those raw predictions are loaded from the prediction cache, so changing the filters does not run inference again. This tutorial passes use_cache=False, so the comparison below does rerun inference.
Below, we keep detections with a confidence of at least 0.3, suppress boxes of the same class that overlap by more than 0.5 IoU, and keep at most 100 detections per image, then compare the result with the baseline.
filtered_config = MaiteEvaluationConfig(
batch_size=2,
confidence_threshold=0.3,
nms_iou_threshold=0.5,
max_detections=100,
)
filtered_run = capability.run(
use_cache=False,
datasets=[dataset],
models=[model],
metrics=[overall_metric, classwise_metric],
config=filtered_config,
)
print(f"{'Metric':<28} {'Baseline':>10} {'Filtered':>10}")
for metric_id, baseline_result in baseline_run.outputs.metrics.items():
filtered_result = filtered_run.outputs.metrics[metric_id]
print(
f"{metric_id:<28} {baseline_result.overall_metric_value:>10.4f} {filtered_result.overall_metric_value:>10.4f}"
)
0%| | 0/2 [00:00<?, ?it/s]
50%|█████ | 1/2 [00:05<00:05, 5.83s/it]
100%|██████████| 2/2 [00:08<00:00, 4.09s/it]
100%|██████████| 2/2 [00:08<00:00, 4.35s/it]
Metric Baseline Filtered mAP_50_285766a2 0.1628 0.0857 mAP_50_classwise_ae4dd8b7 0.1628 0.0857
Filtering is not a way to raise the score. mAP is computed across all confidence levels, so discarding low-confidence detections removes correct detections as well as false positives and usually lowers it. Choose filter settings that match how the model's detections will be used in deployment, and compare candidate models under the same settings.
Report¶
Once the run has completed, the code below uses the markdown output functionality to create a report of the baseline results. We also load a threshold, which is drawn as a reference line on the report's plots. In the example below, 0.15 was chosen arbitrarily to appear relevant next to the actual metric outputs.
from checkmaite.core.report._markdown import create_markdown_output
THRESHOLD = 0.15
report_path = Path("report")
report_path.mkdir(parents=True, exist_ok=True)
report_filename = "MAITE_Evaluation_Sample_Report.md"
report = create_markdown_output(
baseline_run.collect_md_report(threshold=THRESHOLD).content, report_path, report_filename
)
print(f"Markdown report saved in {report_path}.")
Markdown report saved in report.
Recording Results¶
extract() turns a run into MaiteEvaluationRecord rows: one per numeric overall value of each metric, plus one per class for per-class metrics. Writing these rows to an AnalyticsStore lets you compare the baseline across models, datasets, and later runs without re-running anything. The Analytics Store tutorial shows how.
records = baseline_run.extract()
print(f"{len(records)} records")
for record in records[:5]:
print(record.metric_id, record.scope, record.class_name, record.output_key, round(record.output_value, 4))
12 records mAP_50_285766a2 overall None map_50 0.1628 mAP_50_classwise_ae4dd8b7 overall None map_50_classwise 0.1628 mAP_50_classwise_ae4dd8b7 class ignored regions map_50_classwise 0.0 mAP_50_classwise_ae4dd8b7 class pedestrian map_50_classwise 0.1603 mAP_50_classwise_ae4dd8b7 class people map_50_classwise 0.1758
Next Steps¶
With a baseline in hand, continue with the rest of stage 3:
- NRTK measures how the baseline degrades under realistic perturbations.
- XAITK shows what drives the model's detections.
- Analytics Store records the results from every stage so candidate models can be compared.