CheckMAITE Documentation
CheckMAITE is a Python API which makes testing and evaluation of models and datasets straightforward and reproducible. It can be used to run a wide variety of Model Evaluation and Dataset Analysis investigations for both Object Detection and Image Classification computer vision problems.
It was built as an integration point for all CDAO JATIC tools and has since expanded focus. Our goal is to make T&E easier for analysts!
Supported T&E tasks
Each task maps to the CheckMAITE feature that performs it. Model wrappers and
dataset loaders come from the models and dataset_loaders modules of
checkmaite.core.object_detection (OD) and
checkmaite.core.image_classification (IC). The dataset and model evaluation
capabilities are available from both of those packages. Other features show
their full import path.
| Workflow stage | T&E task | Feature | Docs |
|---|---|---|---|
| Bring your model and dataset | Wrap an OD model | TorchvisionODModel, VisdroneODModel, OnnxODModel |
Object Detection Workflow via API, Wrap an ONNX object detection model, ONNX model wrappers |
| Bring your model and dataset | Wrap an IC model | TorchvisionICModel, OnnxICModel |
Image Classification Workflow via API, ONNX model wrappers |
| Bring your model and dataset | Load an OD dataset (COCO, YOLO, VisDrone) | load_coco_detection_dataset, load_yolo_detection_dataset, load_visdrone_detection_dataset |
Object Detection Workflow via API |
| Bring your model and dataset | Load an IC dataset (YOLO) | load_yolo_classification_dataset |
Image Classification Workflow via API |
| Curate the dataset | Find low-quality and duplicate data | DataevalCleaning |
Cleaning |
| Curate the dataset | Detect bias and correlations | DataevalBias |
Bias |
| Curate the dataset | Compare with operational data (drift and OOD) | DataevalShift |
Shift |
| Curate the dataset | Check that the task is achievable | DataevalFeasibility |
Feasibility |
| Evaluate the model | Measure baseline performance | MaiteEvaluation |
MAITE Evaluation |
| Evaluate the model | Test robustness to realistic perturbations | NrtkRobustness |
NRTK |
| Evaluate the model | Explain what drives predictions | XaitkExplainable |
XAITK |
| Record and compare results | Store and query results across runs | checkmaite.core.analytics_store.AnalyticsStore |
Analytics Store |
| Run at scale | Run capabilities on a Ray cluster | checkmaite.jobs.RaySimpleJobBackend, checkmaite.jobs.RayJobBackend |
Ray Simple Job Submission, Ray Job Submission |
| Run at scale | Serve models remotely | checkmaite.core.serving.rayserve: RayObjectDetectionDeployment, RayImageClassificationDeployment, RayObjectDetectionClient, RayImageClassificationClient |
Run remote inference with Ray Serve |
See the T&E workflow for how the stages fit together and who leads each one.
Limitations and prerequisites
CheckMAITE evaluates models and datasets you supply.
- Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO image classification.
- Models and datasets expose an
index2labelmap asdict[int, str]. Default torchvision weights build it from the weights. Custom torchvision weights and ONNX models need anindex2labelin their JSON config (default keyindex2label, override withindex2label_key), given as a list or a dict; both are normalized todict[int, str].num_classesdefaults tolen(index2label). For object detection, set it when the model head has a different class count (e.g. a background slot). For classification it must equallen(index2label). - Where a model's and a dataset's
index2labeloverlap, a shared index must name the same class and a shared class name must use the same index. CheckMAITE does not validate this; mismatches produce silently wrong per-class results. - ONNX models need the
onnx(oronnx-cuda) extra. ONNX wrappers accept onlyuint8images or finite float images in[0, 1](cast tofloat32); other integer dtypes raiseTypeError. Non-finite or out-of-[0, 1]float images raiseValueError. - Object-detection boxes are
float32xyxyarrays of shape(N, 4). See Conventions. - This package provides the Python API and the batch container. The web UI is the
separate CheckMAITE application (
checkmaite-frontend). PDF export needs thereportingextra. - CPU is the supported baseline. Machine, OS, and Python bounds are in the supported environment.
Who is CheckMAITE for?
- No prior knowledge of specific JATIC tools - Our users may or may not be familiar with the individual JATIC products. They are familiar with their datasets and models and simply want to perform testing without needing to learn a new tool (or suite of tools).
- Have their own models and datasets - Our users are expected to come with arbitrary models and datasets. Among other out-of-the-box models, we enable loading default weights from torchvision while also providing a framework for users to bring their own.
Quick start
New to CheckMAITE? Start with the Get started guide for setup, API workflows, remote inference, and saving or querying results.
About the JATIC program and CDAO
The Joint AI Test Infrastructure Capability (JATIC) program develops software products for AI Test & Evaluation (T&E) and AI Assurance. The program is managed by the Assessment & Assurance Division of the DoD Chief Digital and Artificial Intelligence Office (CDAO). It is funded from FY23-FY29.
Program Mission
Develop software to accelerate and enable AI model test and evaluation for testers across the Department of Defense (DoD) enterprise, including DoD programs, research laboratories, industry partners, and academia in order to provide insight on the performance, effectiveness, robustness, and safety of the DoD's AI-enabled systems.
Learn more in the CDAO JATIC program documentation.
CDAO Funding Acknowledgment
This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-25-9-2041. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.