Skip to content

CheckMAITE Documentation

CheckMAITE is a Python API which makes testing and evaluation of models and datasets straightforward and reproducible. It can be used to run a wide variety of Model Evaluation and Dataset Analysis investigations for both Object Detection and Image Classification computer vision problems.

It was built as an integration point for all CDAO JATIC tools and has since expanded focus. Our goal is to make T&E easier for analysts!

Supported T&E tasks

Each task maps to the CheckMAITE feature that performs it. Model wrappers and dataset loaders come from the models and dataset_loaders modules of checkmaite.core.object_detection (OD) and checkmaite.core.image_classification (IC). The dataset and model evaluation capabilities are available from both of those packages. Other features show their full import path.

Workflow stage T&E task Feature Docs
Bring your model and dataset Wrap an OD model TorchvisionODModel, VisdroneODModel, OnnxODModel Object Detection Workflow via API, Wrap an ONNX object detection model, ONNX model wrappers
Bring your model and dataset Wrap an IC model TorchvisionICModel, OnnxICModel Image Classification Workflow via API, ONNX model wrappers
Bring your model and dataset Load an OD dataset (COCO, YOLO, VisDrone) load_coco_detection_dataset, load_yolo_detection_dataset, load_visdrone_detection_dataset Object Detection Workflow via API
Bring your model and dataset Load an IC dataset (YOLO) load_yolo_classification_dataset Image Classification Workflow via API
Curate the dataset Find low-quality and duplicate data DataevalCleaning Cleaning
Curate the dataset Detect bias and correlations DataevalBias Bias
Curate the dataset Compare with operational data (drift and OOD) DataevalShift Shift
Curate the dataset Check that the task is achievable DataevalFeasibility Feasibility
Evaluate the model Measure baseline performance MaiteEvaluation MAITE Evaluation
Evaluate the model Test robustness to realistic perturbations NrtkRobustness NRTK
Evaluate the model Explain what drives predictions XaitkExplainable XAITK
Record and compare results Store and query results across runs checkmaite.core.analytics_store.AnalyticsStore Analytics Store
Run at scale Run capabilities on a Ray cluster checkmaite.jobs.RaySimpleJobBackend, checkmaite.jobs.RayJobBackend Ray Simple Job Submission, Ray Job Submission
Run at scale Serve models remotely checkmaite.core.serving.rayserve: RayObjectDetectionDeployment, RayImageClassificationDeployment, RayObjectDetectionClient, RayImageClassificationClient Run remote inference with Ray Serve

See the T&E workflow for how the stages fit together and who leads each one.

Limitations and prerequisites

CheckMAITE evaluates models and datasets you supply.

  • Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO image classification.
  • Models and datasets expose an index2label map as dict[int, str]. Default torchvision weights build it from the weights. Custom torchvision weights and ONNX models need an index2label in their JSON config (default key index2label, override with index2label_key), given as a list or a dict; both are normalized to dict[int, str]. num_classes defaults to len(index2label). For object detection, set it when the model head has a different class count (e.g. a background slot). For classification it must equal len(index2label).
  • Where a model's and a dataset's index2label overlap, a shared index must name the same class and a shared class name must use the same index. CheckMAITE does not validate this; mismatches produce silently wrong per-class results.
  • ONNX models need the onnx (or onnx-cuda) extra. ONNX wrappers accept only uint8 images or finite float images in [0, 1] (cast to float32); other integer dtypes raise TypeError. Non-finite or out-of-[0, 1] float images raise ValueError.
  • Object-detection boxes are float32 xyxy arrays of shape (N, 4). See Conventions.
  • This package provides the Python API and the batch container. The web UI is the separate CheckMAITE application (checkmaite-frontend). PDF export needs the reporting extra.
  • CPU is the supported baseline. Machine, OS, and Python bounds are in the supported environment.

Who is CheckMAITE for?

  • No prior knowledge of specific JATIC tools - Our users may or may not be familiar with the individual JATIC products. They are familiar with their datasets and models and simply want to perform testing without needing to learn a new tool (or suite of tools).
  • Have their own models and datasets - Our users are expected to come with arbitrary models and datasets. Among other out-of-the-box models, we enable loading default weights from torchvision while also providing a framework for users to bring their own.

Quick start

New to CheckMAITE? Start with the Get started guide for setup, API workflows, remote inference, and saving or querying results.

About the JATIC program and CDAO

The Joint AI Test Infrastructure Capability (JATIC) program develops software products for AI Test & Evaluation (T&E) and AI Assurance. The program is managed by the Assessment & Assurance Division of the DoD Chief Digital and Artificial Intelligence Office (CDAO). It is funded from FY23-FY29.

Program Mission

Develop software to accelerate and enable AI model test and evaluation for testers across the Department of Defense (DoD) enterprise, including DoD programs, research laboratories, industry partners, and academia in order to provide insight on the performance, effectiveness, robustness, and safety of the DoD's AI-enabled systems.

Learn more in the CDAO JATIC program documentation.

CDAO Funding Acknowledgment

This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-25-9-2041. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.