Using JATIC tools
Dataset evaluation
-
The Dataeval bias detection tools identify biases or correlations present in a dataset that may lead to shortcut learning in models.
-
The Dataeval feasiblity tools measure the irreducible error of a dataset.
-
The Dataeval cleaning tools analyze datasets to identify and remove invalid, irrelevant, or low-quality data, such as duplicates and statistical outliers.
-
The Dataeval shift tool identifies statistical differences between operational data and training data
Model evaluation
-
The Natural Robustness Toolkit (NRTK) is an open source toolkit for generating operationally realistic perturbations to evaluate the natural robustness of computer vision algorithms.
-
The Explainable AI Toolkit for saliency (XAITK - Saliency) is an open source framework for visual saliency algorithm interfaces and implementations.
Results & analysis
-
The analytics store persists capability results as queryable records, enabling SQL-based comparison across runs, datasets, and capabilities.
Job submission
-
Run a capability asynchronously with the lightweight process-local Ray task backend.
-
Run a capability asynchronously with the registry-backed Ray backend for tracked and reattachable jobs.