Using JATIC tools
These tutorials follow the T&E workflow. Each one names its persona and the workflow stage it covers.
Dataset evaluation
-
The Dataeval bias detection tools identify biases or correlations present in a dataset that may lead to shortcut learning in models.
-
The Dataeval feasiblity tools measure the irreducible error of a dataset.
-
The Dataeval cleaning tools analyze datasets to identify and remove invalid, irrelevant, or low-quality data, such as duplicates and statistical outliers.
-
The Dataeval shift tool identifies statistical differences between operational data and training data
Model evaluation
-
The MAITE evaluation tool measures a model's baseline performance on a dataset with one or more metrics, overall and per class.
-
The Natural Robustness Toolkit (NRTK) is an open source toolkit for generating operationally realistic perturbations to evaluate the natural robustness of computer vision algorithms.
-
The Explainable AI Toolkit for saliency (XAITK - Saliency) is an open source framework for visual saliency algorithm interfaces and implementations.
Results & analysis
-
The analytics store persists capability results as queryable records, enabling SQL-based comparison across runs, datasets, and capabilities.
Job submission
-
Run a capability asynchronously with the lightweight process-local Ray task backend.
-
Run a capability asynchronously with the registry-backed Ray backend for tracked and reattachable jobs.
End-to-end workflows
-
Object Detection Workflow via API
Run every capability on one object detection model and dataset, then build a single report.
-
Image Classification Workflow via API
The same end-to-end workflow for image classification.
For task-focused steps, see the how-to guides.