Skip to content

Glossary

Key terms used across the CheckMAITE documentation. Each entry links to the page that covers it in depth.

CheckMAITE

  • Harness: CheckMAITE itself: a test and evaluation (T&E) harness that runs evaluation workflows over MAITE-compliant models, datasets, and metrics, and collects their results in a consistent form.
  • Capability: One evaluation workflow the harness can run, such as model inference plus metrics on a dataset, or a dataset bias analysis. Each capability defines the configuration it accepts and the outputs it produces. See Key concepts.
  • Run: Everything associated with one execution of a capability: its configuration, its outputs, and an optional report. See Key concepts.
  • Analytics store: Machine-readable results of capability runs, stored as one table per capability so tools and SQL queries can consume them across runs, datasets, and capabilities. This is separate from the human-readable reports a run produces. See Save & query results.
  • Record: One flat, scalar-only row in an analytics store table, produced by a run's extract() method. See the Analytics store guide.
  • Job backend: The execution target for asynchronous capability runs, selected with configure_job_backend(...). CheckMAITE ships ray (registry-backed, reattachable jobs) and ray-simple (process-local Ray tasks). See Job backend configuration.
  • Job: A non-blocking handle to a submitted capability run, with lifecycle states and a reference-first result(). See Job protocol and lifecycle.
  • Worker environment: The software environment, such as a container image and its installed packages, that remote workers use to run capabilities. See Worker environments.
  • Plugin: An installable package that adds capabilities to CheckMAITE through entry points. See Plugin system.
  • Persona: A representative user role, such as Data Scientist or ML Engineer, used to frame tutorials and guide design. See Personas.

MAITE and data

  • MAITE: The Modular AI Trustworthy Engineering protocols that CheckMAITE models, datasets, and metrics conform to. See the MAITE documentation.
  • Wrapper: A MAITE-compliant object around a model, dataset, or metric that CheckMAITE capabilities can consume. See Conventions.
  • index2label: The dict[int, str] map from class index to class name that models and datasets expose. Where a model's and a dataset's maps overlap, they should agree. Nothing checks this, and a mismatch gives silently wrong per-class results. See Conventions.
  • Annotation format: The on-disk dataset layout CheckMAITE can load, such as COCO, YOLO, or VisDrone. See Supported dataset annotation formats.

JATIC tools

  • DataEval: JATIC's dataset analysis toolkit. In CheckMAITE it powers the bias, feasibility, cleaning (linting), and shift capabilities. See the Dataeval bias tutorial.
  • Bias: Correlations or imbalances in a dataset that can lead a model to learn shortcuts. See the Dataeval bias tutorial.
  • Feasibility: An estimate of the irreducible error of a dataset, which bounds how well any model can perform on it. See the Dataeval feasibility tutorial.
  • Cleaning (linting): Finding invalid, irrelevant, or low-quality data such as duplicates and statistical outliers. See the Dataeval linting tutorial.
  • Shift: Statistical differences between operational data and training data, found by drift and out-of-distribution detection. See Distribution shift and the Dataeval shift tutorial.
  • NRTK (Natural Robustness Toolkit): Generates operationally realistic image perturbations to measure how model performance degrades. See the NRTK tutorial.
  • Perturber: An NRTK algorithm that modifies an image according to a fixed configuration, such as halving its brightness. See the NRTK tutorial.
  • Perturbation factory: An NRTK generator of perturbers that vary one or more parameters (theta keys) across a range. See the NRTK tutorial.
  • XAITK (Explainable AI Toolkit): Generates visual saliency maps that show which image regions drive a model's prediction. See Saliency and explainability and the XAITK tutorial.
  • Saliency map: A visual representation that highlights the input regions most relevant to one model prediction. See Saliency and explainability and the XAITK tutorial.