Analytics Store Tutorial¶
Who this is for: Data Scientist who needs reproducible, searchable results across analyses.
Where this fits: stage 4, Record and compare results, of the T&E workflow. It collects the dataset-curation and model-evaluation results from stages 2 and 3 so they can be compared without re-running anything.
What you'll learn
- How to share one
AnalyticsStoreacross multiple capabilities - What each capability's record looks like in the store
- How to query a single capability's results with SQL
- How to JOIN across capabilities on a shared
dataset_id - How to read the results from outside Python
What is the Analytics Store?¶
The Analytics Store is a thin write-and-query layer over your capability outputs. Each capability emits one or more records via .extract(); the store writes them to a backing format (Parquet by default) and lets you query the result with SQL. Records from different capabilities live in different tables but share key columns — most importantly dataset_id — so you can answer questions that span capabilities without re-running anything.
flowchart LR
A[COCO dataset] --> B1[Dataeval Bias]
A --> B2[Dataeval Feasibility]
A --> B3[Dataeval Shift]
A --> B4[NRTK Robustness]
A --> B5[XAITK Explainable]
B1 --> S[(Analytics Store)]
B2 --> S
B3 --> S
B4 --> S
B5 --> S
S --> Q[Cross-capability SQL JOINs]
Scope
Dataeval Bias, Dataeval Feasibility, and Dataeval Shift are executed for real against the bundled COCO sample. NRTK Robustness and XAITK Explainable would take too long to run in a docs build, so this tutorial constructs simulated records to demonstrate the store contract — see the per-tool tutorials for end-to-end real runs.
Setup¶
We create a single AnalyticsStore backed by a temporary directory. Every capability in this tutorial writes into this same store, which is what makes the cross-capability JOINs at the end work.
import tempfile
from pathlib import Path
from checkmaite.core.analytics_store import AnalyticsStore, ParquetBackend
from checkmaite.core.object_detection.dataset_loaders import load_coco_detection_dataset
REPO_ROOT = Path.cwd().parents[1]
COCO_ROOT = REPO_ROOT / "tests/data_for_tests/coco_dataset"
COCO_RESIZED_ROOT = REPO_ROOT / "tests/data_for_tests/coco_resized_val2017"
if not (COCO_ROOT / "ann_file.json").exists():
raise FileNotFoundError(f"missing {COCO_ROOT}")
if not (COCO_RESIZED_ROOT / "instances_val2017_resized_6.json").exists():
raise FileNotFoundError(f"missing {COCO_RESIZED_ROOT}")
store_dir = tempfile.mkdtemp(prefix="analytics_store_tutorial_")
store = AnalyticsStore(ParquetBackend(store_dir))
print("store backed by:", store_dir)
store backed by: /tmp/analytics_store_tutorial_8xizoycb
Load the dataset¶
All five capabilities run against the same coco-example dataset. DataevalShift additionally needs a second dataset to compare against — we use the resized COCO bundle as the evaluation dataset. Passing an explicit dataset_id= keeps the IDs stable across capabilities, which is what makes the JOINs at the end of this tutorial work.
dataset = load_coco_detection_dataset(
root=str(COCO_ROOT),
ann_file=str(COCO_ROOT / "ann_file.json"),
dataset_id="coco-example",
)
shift_eval_dataset = load_coco_detection_dataset(
root=str(COCO_RESIZED_ROOT),
ann_file=str(COCO_RESIZED_ROOT / "instances_val2017_resized_6.json"),
dataset_id="coco-resized",
)
print(f"primary : {dataset.metadata['id']} ({len(dataset)} images)")
print(f"shift_eval: {shift_eval_dataset.metadata['id']} ({len(shift_eval_dataset)} images)")
primary : coco-example (4 images) shift_eval: coco-resized (6 images)
Capability 1 — Dataeval Bias¶
DataevalBiasBase reports dataset-level coverage, balance, and diversity statistics. See the bias tutorial for the full walkthrough — this section just shows how its results land in the store.
from checkmaite.core._common.dataeval_bias_capability import DataevalBiasBase
bias_run = DataevalBiasBase().run(datasets=[dataset], use_cache=False)
bias_records = bias_run.extract()
print(f"extracted {len(bias_records)} bias record(s)")
bias_records[0]
extracted 1 bias record(s)
DataevalBiasRecord(run_uid='7fe7daae69fd0f7230ff1efdf5ebc36ae6003a80122b8cf35f37703f7088ca16', created_at=datetime.datetime(2026, 11, 2, 18, 9, 15, 439371, tzinfo=datetime.timezone.utc), dataset_id='coco-example', coverage_total=4, coverage_uncovered_count=1, coverage_uncovered_ratio=0.25, coverage_radius=1.519218921661377, balance_num_factors=7, balance_mean=0.7644513419696263, balance_max=1.0, balance_factors_above_05=7, diversity_num_factors=7, diversity_mean=0.7689653521422591, diversity_min=0.30238611713665947, diversity_factors_below_04=1)
store.write([bias_run])
store.list_tables()
['dataeval_bias', 'runs']
store.query_sql("SELECT dataset_id, coverage_uncovered_ratio, balance_mean, diversity_mean " "FROM dataeval_bias")
| dataset_id | coverage_uncovered_ratio | balance_mean | diversity_mean |
|---|---|---|---|
| str | f64 | f64 | f64 |
| "coco-example" | 0.25 | 0.764451 | 0.768965 |
Capability 2 — Dataeval Feasibility¶
DataevalFeasibility reports Bayes-error-rate (BER) bounds and object-detection health stats — small-object ratio, truncated-bbox ratio, and any health warnings. See the feasibility tutorial for the full treatment.
from checkmaite.core.object_detection import DataevalFeasibility
feasibility_run = DataevalFeasibility().run(datasets=[dataset], use_cache=False)
feasibility_records = feasibility_run.extract()
print(f"extracted {len(feasibility_records)} feasibility record(s)")
feasibility_records[0]
extracted 1 feasibility record(s)
DataevalFeasibilityRecord(run_uid='0ae13d0c7c3810a0293ec0377817d341ce2f221553107e7b0c7f75cadfe20465', created_at=datetime.datetime(2026, 11, 2, 18, 9, 17, 72159, tzinfo=datetime.timezone.utc), dataset_id='coco-example', ber_upper=0.7058823529411765, ber_lower=0.4690627917182514, num_instances=51, num_classes=19, small_object_ratio=0.3157894736842105, truncated_bbox_ratio=0.12280701754385964, overlap_image_ratio=0.0, health_warning_count=2)
store.write([feasibility_run])
store.query_sql(
"SELECT dataset_id, ber_upper, ber_lower, small_object_ratio, " "health_warning_count FROM dataeval_feasibility"
)
| dataset_id | ber_upper | ber_lower | small_object_ratio | health_warning_count |
|---|---|---|---|---|
| str | f64 | f64 | f64 | i64 |
| "coco-example" | 0.705882 | 0.469063 | 0.315789 | 2 |
Capability 3 — Dataeval Shift¶
Shift compares two datasets
Unlike the single-dataset capabilities above, DataevalShift takes a reference dataset and an evaluation dataset and reports drift (MMD, CVM, KS) and out-of-distribution statistics. Its record stores two dataset IDs — reference_dataset_id and evaluation_dataset_id — which matters when you start writing cross-capability JOINs.
See the shift tutorial for the full walkthrough.
from checkmaite.core.object_detection import DataevalShift
shift_run = DataevalShift().run(
datasets=[dataset, shift_eval_dataset],
use_cache=False,
)
shift_records = shift_run.extract()
print(f"extracted {len(shift_records)} shift record(s)")
shift_records[0]
extracted 1 shift record(s)
DataevalShiftRecord(run_uid='91ff3f99f52ac61ed775b52d6cf0f7a253d0acb09a8dccc36dc58116c4108744', created_at=datetime.datetime(2026, 11, 2, 18, 9, 17, 645070, tzinfo=datetime.timezone.utc), reference_dataset_id='coco-example', evaluation_dataset_id='coco-resized', mmd_drifted=False, mmd_distance=0.05928581953048706, mmd_p_val=0.20000000298023224, mmd_threshold=0.05, cvm_drifted=False, cvm_distance=0.19999995827674866, cvm_p_val=0.34761905670166016, cvm_threshold=0.0125, cvm_feature_drift_count=0, ks_drifted=False, ks_distance=0.5416666865348816, ks_p_val=0.4595238268375397, ks_threshold=0.0125, ks_feature_drift_count=0, ood_count=0, ood_total=6, ood_ratio=0.0, ood_mean_instance_score=0.763585090637207, ood_std_instance_score=0.04156506061553955, ood_max_instance_score=0.8205173015594482)
store.write([shift_run])
store.query_sql(
"SELECT reference_dataset_id, evaluation_dataset_id, " "mmd_drifted, mmd_distance, ood_ratio " "FROM dataeval_shift"
)
| reference_dataset_id | evaluation_dataset_id | mmd_drifted | mmd_distance | ood_ratio |
|---|---|---|---|---|
| str | str | bool | f64 | f64 |
| "coco-example" | "coco-resized" | false | 0.059286 | 0.0 |
Capability 4 — NRTK Robustness (simulated)¶
These records are hand-built for the tutorial
Running NRTK against real perturbers is slow and out of scope here. We construct a small set of NrtkRobustnessRecord objects directly so the JOIN finale below has rows to work with. For an end-to-end NRTK walkthrough, see the nrtk tutorial.
from checkmaite.core._common.nrtk_robustness_capability import NrtkRobustnessRecord
thetas = [0.6, 0.8, 1.0, 1.2, 1.4]
accuracies = [0.62, 0.74, 0.81, 0.69, 0.55]
nrtk_records = [
NrtkRobustnessRecord(
run_uid="tutorial-nrtk-run",
dataset_id="coco-resized",
model_id="resnet50",
metric_id="coco_metrics",
perturber_class="BrightnessPerturber",
perturber_type="Brightness Perturber",
theta_key="factor",
theta_index=i,
theta_value=theta,
metric_key="accuracy",
metric_value=acc,
is_primary=True,
)
for i, (theta, acc) in enumerate(zip(thetas, accuracies, strict=False))
]
len(nrtk_records)
5
# nrtk records are written via the backend directly since they are not
# produced by a capability .run() in this tutorial.
store._backend.write(nrtk_records) # noqa: SLF001 (intentional for the demo)
store.query_sql("SELECT theta_value, metric_value FROM nrtk_robustness " "WHERE is_primary = true ORDER BY theta_index")
| theta_value | metric_value |
|---|---|
| f64 | f64 |
| 0.6 | 0.62 |
| 0.8 | 0.74 |
| 1.0 | 0.81 |
| 1.2 | 0.69 |
| 1.4 | 0.55 |
Capability 5 — XAITK Explainable (simulated)¶
These records are hand-built for the tutorial
XAITK produces per-image (or per-detection) saliency summaries — a different schema shape than the per-dataset capabilities above. We construct a few records directly to show the table layout. For the real walkthrough see the xaitk tutorial.
from checkmaite.core._common.xaitk_explainable_capability import (
XaitkExplainableRecord,
)
xaitk_records = [
XaitkExplainableRecord(
run_uid="tutorial-xaitk-run",
dataset_id="coco-example",
model_id="resnet50",
saliency_generator_type="RISEStack",
image_index=i,
gt_label=label,
mean_saliency=mean,
max_saliency=mx,
std_saliency=std,
positive_saliency_ratio=pos,
)
for i, (label, mean, mx, std, pos) in enumerate(
[
("cat", 0.34, 0.88, 0.18, 0.62),
("dog", 0.41, 0.93, 0.21, 0.71),
("car", 0.28, 0.80, 0.15, 0.55),
]
)
]
len(xaitk_records)
3
store._backend.write(xaitk_records) # noqa: SLF001
store.query_sql(
"SELECT image_index, gt_label, mean_saliency, max_saliency " "FROM xaitk_explainable ORDER BY image_index"
)
| image_index | gt_label | mean_saliency | max_saliency |
|---|---|---|---|
| i64 | str | f64 | f64 |
| 0 | "cat" | 0.34 | 0.88 |
| 1 | "dog" | 0.41 | 0.93 |
| 2 | "car" | 0.28 | 0.8 |
Cross-capability JOINs¶
Every capability above wrote into the same store, keyed by dataset_id. That single design choice is what makes the next three queries possible — we can ask questions that span capabilities without leaving SQL.
First, confirm what landed in the store:
sorted(store.list_tables())
['dataeval_bias', 'dataeval_feasibility', 'dataeval_shift', 'nrtk_robustness', 'runs', 'xaitk_explainable']
JOIN 1 — Everything dataeval knows about coco-example¶
Bias and feasibility both key by dataset_id. Joining them in one query gives a multi-capability profile of the dataset in a single row.
store.query_sql(
"SELECT "
" b.dataset_id, "
" b.coverage_uncovered_ratio, "
" b.balance_mean, "
" b.diversity_mean, "
" f.ber_upper, "
" f.ber_lower, "
" f.small_object_ratio, "
" f.health_warning_count "
"FROM dataeval_bias b "
"JOIN dataeval_feasibility f USING (dataset_id)"
)
| dataset_id | coverage_uncovered_ratio | balance_mean | diversity_mean | ber_upper | ber_lower | small_object_ratio | health_warning_count |
|---|---|---|---|---|---|---|---|
| str | f64 | f64 | f64 | f64 | f64 | f64 | i64 |
| "coco-example" | 0.25 | 0.764451 | 0.768965 | 0.705882 | 0.469063 | 0.315789 | 2 |
JOIN 2 — Drift vs robustness on the same data¶
dataeval_shift doesn't have a single dataset_id — it has reference_dataset_id and evaluation_dataset_id. To JOIN against the single-dataset capabilities, we match on the side we care about. Here we put each dataset's drift verdict side-by-side with the model's robustness curve on the same data, so a reviewer can read drift status and robustness in one query.
store.query_sql(
"SELECT "
" s.evaluation_dataset_id AS dataset_id, "
" s.mmd_drifted, "
" s.mmd_distance, "
" s.ood_ratio, "
" n.perturber_type, "
" n.theta_value, "
" n.metric_value "
"FROM dataeval_shift s "
"JOIN nrtk_robustness n "
" ON s.evaluation_dataset_id = n.dataset_id "
"WHERE n.is_primary = true AND n.metric_key = 'accuracy' "
"ORDER BY n.theta_index"
)
| dataset_id | mmd_drifted | mmd_distance | ood_ratio | perturber_type | theta_value | metric_value |
|---|---|---|---|---|---|---|
| str | bool | f64 | f64 | str | f64 | f64 |
| "coco-resized" | false | 0.059286 | 0.0 | "Brightness Perturber" | 0.6 | 0.62 |
| "coco-resized" | false | 0.059286 | 0.0 | "Brightness Perturber" | 0.8 | 0.74 |
| "coco-resized" | false | 0.059286 | 0.0 | "Brightness Perturber" | 1.0 | 0.81 |
| "coco-resized" | false | 0.059286 | 0.0 | "Brightness Perturber" | 1.2 | 0.69 |
| "coco-resized" | false | 0.059286 | 0.0 | "Brightness Perturber" | 1.4 | 0.55 |
JOIN 3 — Provenance via runs¶
Every store.write() also populates the runs table with one row per (run, capability, entity). That table is your audit trail — joinable back to any capability table by run_uid.
store.query_sql(
"SELECT capability_table, entity_type, entity_id, run_uid, created_at "
"FROM runs ORDER BY capability_table, created_at"
)
| capability_table | entity_type | entity_id | run_uid | created_at |
|---|---|---|---|---|
| str | str | str | str | datetime[μs, UTC] |
| "dataeval_bias" | "dataset" | "coco-example" | "7fe7daae69fd0f7230ff1efdf5ebc3… | 2026-11-02 18:09:15.444994 UTC |
| "dataeval_feasibility" | "dataset" | "coco-example" | "0ae13d0c7c3810a0293ec0377817d3… | 2026-11-02 18:09:17.092367 UTC |
| "dataeval_shift" | "dataset" | "coco-example" | "91ff3f99f52ac61ed775b52d6cf0f7… | 2026-11-02 18:09:17.651098 UTC |
| "dataeval_shift" | "dataset" | "coco-resized" | "91ff3f99f52ac61ed775b52d6cf0f7… | 2026-11-02 18:09:17.651106 UTC |
Why this matters
Each capability's tutorial shows you how to run it. The analytics store is what lets you put those results in one place and ask cross-capability questions later — without re-running anything.
Using results outside Python¶
The Parquet backend writes plain .parquet files — one or more per table. You can point any Parquet-aware tool (DuckDB CLI, pandas, polars from a different process, Spark, …) at the directory and query it without CheckMAITE installed.
for path in sorted(Path(store_dir).rglob("*.parquet")):
print(path.relative_to(store_dir))
dataeval_bias/1793642955445_7d60f902.parquet dataeval_feasibility/1793642957092_3c8f8e0d.parquet dataeval_shift/1793642957651_0dabd8f8.parquet nrtk_robustness/1793642957666_f8e1e0fc.parquet runs/1793642955448_695217ca.parquet runs/1793642957109_c624158c.parquet runs/1793642957654_a959135e.parquet xaitk_explainable/1793642957678_8cd1d055.parquet
Distributed job submission¶
This tutorial writes results locally with store.write([run]). Under distributed job submission the responsibilities split:
- the client chooses the durable store location,
- the worker needs that information to persist results,
- and the client later reads from that same location.
So distributed runs require explicit configuration:
from checkmaite.jobs import configure_job_backend
configure_job_backend(
"ray",
idempotency_scope="team-a-analytics",
analytics_store={"backend": "parquet", "uri": "s3://team-checkmaite/analytics-store"},
artifact_store={"uri": "s3://team-checkmaite/report-artifacts"},
)
checkmaite.jobs._store defines AnalyticsStoreConfig; the job backend forwards this config on submissions so workers write to the intended store. See Analytics store in distributed execution and Job backend configuration.