Key Concepts: Capabilities, Runs, and Caching
This page documents the core abstractions used to define, execute, and cache evaluations: Capabilities, Runs, and the caching layer.
graph LR
subgraph Inputs
D[Dataset]
M[Model]
Me[Metric]
end
subgraph Capability
C[Config] --> R["run()"]
D --> R
M --> R
Me --> R
end
R -.->|use_cache=True| Cache[(Run Cache)]
Cache -.-> Run
R --> O[Outputs]
C --> Run["Run Object"]
O --> Run
Run --> Report["collect_md_report()"]
Run --> Extract["extract()"]
Extract --> Record["Record (flat scalars)"]
Record --> Store["Analytics Store"]
Core Concepts
Capability
A Capability represents a specific evaluation task — for example, running model inference and computing metrics on a dataset. It is the top-level abstraction that users interact with.
A Capability is responsible for:
- Defining the configuration accepted by the evaluation (via a
Configobject) - Knowing how to execute an evaluation (
_run) - Knowing how to check the cache before executing (handled by
run)
Run
A Run is an object that stores everything associated with a specific execution of a Capability. This includes:
- The configuration for that execution (e.g., model, dataset, metric settings)
- The outputs produced (e.g., predictions, metric results)
- An optional
collect_md_report()implementation returning a typed inline or artifact report for those outputs
CheckMAITE uses Pydantic to serialize cache entries. It directly supports common objects such as NumPy arrays and Torch tensors by saving their data in separate binary files. If saving fails, CheckMAITE cleans up partial files. Cache links are only followed while loading the cache, so similar-looking user strings remain strings.
To cache another type, register a codec with
binary_de_serializer.register(...). The codec supplies functions that convert
the value to bytes and restore it. Strict mode uses a custom codec only when the
codec also identifies which values have a lossless round trip.
Implementing a New Capability
Each tool must implement:
Config— A Pydantic model declaring what configuration options the Capability accepts. Can bepassif no configuration is needed.Outputs— A Pydantic model declaring what outputs will be stored and cached.Run— Contains theConfig, theOutputs, and thecollect_md_report()method for report generation.- Capability class — aka the "Runner". Implements
_run(...), the actual execution logic. Calls internal helpers (e.g.,maite_evaluate) to produce outputs.
classDiagram
class CapabilityConfigBase {
Pydantic BaseModel
}
class CapabilityOutputsBase {
Pydantic BaseModel
+serialize/deserialize binary
}
class CapabilityRunBase~TConfig, TOutputs~ {
+capability_id: str
+config: TConfig
+outputs: TOutputs
+dataset_metadata: list
+model_metadata: list
+metric_metadata: list
+run_uid(): str (SHA-256)
+extract() list~BaseRecord~
+collect_md_report() CapabilityReport
}
class Capability~TOutputs, TDataset, TModel, TMetric, TConfig~ {
<<abstract>>
+run() CapabilityRunBase
#_run()* TOutputs
+supports_datasets: Number
+supports_models: Number
+supports_metrics: Number
}
class BaseRecord {
+run_uid: str
+created_at: datetime
scalar fields only
}
Capability --> CapabilityRunBase : creates via run()
CapabilityConfigBase --> CapabilityRunBase : input config
CapabilityOutputsBase --> CapabilityRunBase : input outputs
CapabilityRunBase --> BaseRecord : extract() produces
See the Baseline Evaluation Capability for the simplest implementation.
Caching
The caching layer is designed to avoid redundant computation. There are two levels of caching:
flowchart TD
A["capability.run(model, dataset, ...)"] --> B{use_cache?}
B -->|True| C{Capability Cache Hit?}
C -->|Yes| D[Return cached Run]
C -->|No| E["Execute _run()"]
B -->|False| E
E --> F{use_cache?}
F -->|True| G{Prediction/Eval Cache Hit?}
G -->|Yes| H[Reuse cached predict/evaluate result]
G -->|No| I["Call predict() / evaluate()"]
F -->|False| I
H --> J[Build Outputs]
I --> J
J --> K[Create Run object]
K --> L{use_cache?}
L -->|True| M[Store in Capability Cache]
M --> N[Return Run]
L -->|False| N
1. Capability-level Cache
When a Capability is executed with use_cache=True (the default), it checks
whether a Run with the same configuration and inputs has already been completed.
If a cache hit is found, the stored Run object is returned immediately — no
computation occurs.
2. Prediction/Evaluation Cache
At a lower level, individual predict and evaluate calls (e.g., calls to
maite.evaluate) are also cached globally. If two different Capabilities within
the same pipeline call evaluate with the same model, dataset, and metric
configuration, the second call will reuse the result from the first.
This cache is controlled by the same use_cache flag. When use_cache=False,
both the capability-level and prediction/evaluation-level caches are bypassed.
Note: It is not currently possible to disable the prediction/evaluation cache independently of the capability cache. Both are toggled together via
use_cache.
MAITE Execution and Ownership Semantics
CheckMAITE treats maite.tasks.evaluate() as the fundamental task.
cached_tasks.predict() is the convenience form of
evaluate(metric=None, return_preds=True). On a cold evaluation, MAITE applies
augmentation, calls the model, updates the metric, and optionally retains each
batch for return—all in one pass.
MAITE passes the same batch objects between these steps; it does not make defensive copies. CheckMAITE follows that ownership model and assumes:
- model and augmentation adapters return stable outputs that remain valid after later batches are processed;
metric.update(predictions, targets, metadata)treats all three arguments as read-only; and- callers do not depend on Python alias identity or mutate returned values expecting the cache to change.
CheckMAITE validates batch alignment, but it does not snapshot arrays or tensors before metric updates. A metric that mutates a prediction can therefore also alter the value returned to the caller or published to the prediction cache. A model that repeatedly returns and overwrites the same mutable output buffer can similarly make earlier batches appear to contain the final batch. Such adapters and metrics are outside the supported contract and should copy internally when they need mutable working storage.
Multi-Metric Evaluation
Image-classification and object-detection MaiteEvaluation capabilities accept
one or more metrics. CheckMAITE presents the collection to MAITE as one internal
fanout metric, so augmentation and model inference happen once per batch even
when use_cache=False. Member metrics receive the same prediction, target, and
metadata objects in canonical metric-ID order.
run = MaiteEvaluation().run(
models=[model],
datasets=[dataset],
metrics=[accuracy, recall],
use_cache=False,
)
accuracy_result = run.outputs.metrics[accuracy.metadata["id"]]
print(accuracy_result.result)
print(accuracy_result.overall_metric_value)
Metric metadata IDs must be non-empty and unique. CheckMAITE sorts metrics by ID
before evaluation, run identity generation, reports, and analytics, so reversing
caller order does not create a different run. Any member failure immediately
aborts the complete evaluation with a MaiteEvaluationMetricError identifying
the member and lifecycle stage. Partial capability runs are not returned.
Retained Data and Memory
return_preds=True asks MAITE to retain raw prediction batches in memory;
CheckMAITE returns them after any configured CPU postprocessing. Even when the
caller sets return_preds=False, CheckMAITE may request predictions internally
on a cold call when it needs to publish the prediction cache or perform deferred
CPU postprocessing. The flag controls the public return value, not an
unconditional peak-memory guarantee.
Ordinary MOT calls cache predictions, targets, and metadata, including
track_ids, but store empty input batches instead of decoded video.
return_augmented_data=True is intended primarily for debugging and inspecting
the exact data sent to the model. It returns complete post-augmentation input,
target, and metadata batches. CheckMAITE always runs that request fresh and
publishes nothing to the prediction or evaluation caches.
For a full-data MOT request, CheckMAITE wraps the supplied augmentation and
materializes one-shot video streams after augmentation but before the model
runs. A non-Sequence stream is therefore passed to the model and returned to
the caller as a list of frames rather than as its original wrapper type. This
preserves an inspectable realization after inference but can require substantial
memory. Prefer the default False for normal evaluations, especially with large
videos.
Stochastic Inference
A cache key is a claim that the identified model, dataset, augmentation, batching, and postprocessing define reusable behavior. Runtime RNG state is not inspected or hashed. For stochastic models or augmentations, a cache hit reuses the first published realization rather than drawing another sample.
Use use_cache=False whenever each call must produce a fresh draw. For
reproducible stochastic evaluation, control the RNG and include the seed and all
sampling settings in the relevant model or augmentation metadata ID. Changing a
seed without changing that identity can produce an incorrect cache hit; changing
an ID does not itself seed the implementation.
from checkmaite import cached_tasks
# Reuse one identified, reproducible realization.
results, predictions, _ = cached_tasks.evaluate(
# metadata["id"] includes checkpoint, seed, and sampling settings
model=seeded_model,
metric=metric,
dataset=dataset,
return_preds=True,
use_cache=True,
)
# Draw again on every call.
results, predictions, _ = cached_tasks.evaluate(
model=stochastic_model,
metric=metric,
dataset=dataset,
return_preds=True,
use_cache=False,
)
Cache Key Generation
Cache hits are determined by a SHA-256 hash of:
capability_idconfig(the full configuration object)dataset_id(for each dataset)model_id(for each model)metric_id(for each metric)
Changing the config or using a different capability will produce a different cache key, even with the same datasets and models. The IDs are user-supplied metadata fields and must be unique. The cache does not perform content-based hashing (e.g., checksumming image files) for performance reasons. It is the responsibility of the caller to ensure that IDs accurately reflect the data being passed in.
⚠️ Important: If you run the same model or dataset under the same ID but with different underlying content, you will get incorrect cache hits. When using this library programmatically (e.g., from a notebook), ensure IDs are managed carefully. In a production environment with a model registry or dataset warehouse, these IDs should be derived automatically from versioned artifacts.
Configuring the Cache
The cache behavior is controlled by the use_cache parameter on the
Capability's run method:
# Use cache (default) — will return cached result if available
capability.run(model=my_model, dataset=my_dataset, use_cache=True)
# Bypass cache — always recompute
capability.run(model=my_model, dataset=my_dataset, use_cache=False)
Flexible and Strict Serialization
CheckMAITE provides two serialization options for task artifacts:
- Flexible (potentially lossy) accepts more Python values. Pydantic or a registered codec may change the representation. For example, a tuple may return as a list or a dataclass as a mapping. Use this when those changes do not affect the evaluation.
- Strict (lossless) accepts only values with an explicitly supported round trip. Unsupported artifacts are returned to the caller but are not saved to the cache, so cache limitations do not fail completed computation.
Cached-task APIs select the option with strict_cache_serialization:
from checkmaite import cached_tasks
# Flexible caching (default).
results, predictions, _ = cached_tasks.evaluate(
model=model,
metric=metric,
dataset=dataset,
use_cache=True,
return_preds=True,
strict_cache_serialization=False,
)
# Publish only when every persisted value has an admitted lossless codec.
results, predictions, _ = cached_tasks.evaluate(
model=model,
metric=metric,
dataset=dataset,
use_cache=True,
return_preds=True,
strict_cache_serialization=True,
)
Strict mode supports a conservative tree of exact lists, string-keyed dictionaries, finite scalar values, and explicitly admitted binary codecs. NumPy values require safe dtypes and exact scalar-class round trips. Torch values require exact CPU strided tensors without gradients. Other codecs must explicitly opt into strict admission.
MOT target objects are not currently admitted by strict serialization. A strict MOT call still completes, but CheckMAITE warns and skips prediction artifact publication. Use the default flexible option when MOT predictions, targets, and metadata need to be cached.
Here, lossless means preserving the admitted scientific value and supported
type. It does not cover NumPy or Torch storage topology, array writability, or
arbitrary tensor attributes. Use use_cache=False when unsupported state is
part of the computation.
Flexible serialization is the default for CheckMAITE's built-in capabilities and capability-level caches. The strict option applies to cached-task orchestration.
Input Flexibility (Type Coercion)
CheckMAITE accepts flexible input types at its public API boundary and normalizes them internally. For example, an image can be passed as:
- A file path (
strorPath) - Raw bytes
- A
BufferedIOBaseobject - A PIL
Imageobject
Internally, all images are normalized to PIL Image objects before any
processing occurs. This coercion is handled automatically via Pydantic
validators and follows Postel's
Law: be flexible in what
you accept, strict in what you emit.
For tabular inputs, core capabilities accept pandas DataFrames at the API boundary.
PySpark-dependent paths live in the optional checkmaite-plugins package.
This means internal code never needs to check input types — it can always assume inputs are in the canonical internal format.
Reporting and Visualization
A Run can implement collect_md_report() to return a typed InlineTextReport
or ArtifactReport. The base implementation continues to raise
NotImplementedError for backward compatibility. Inline reports include a media
type, filename, and self-contained textual content; embedded resources should
use forms such as data: URIs rather than local file paths. Job-result metadata
accepts up to 256 KiB inline and can externalize a larger report through its
configured artifact store. Artifact reports include a media type, filename, and
producer-owned durable uri for large, binary, or multi-file reports. Report
models are exported from checkmaite.core.report.
Report generation is handled by utilities located in the report/ submodule:
- Gradient-based reports (legacy, optional dependency) — generates visual outputs using the Gradient library. Will emit a deprecation warning if used.
- Markdown reports — generates a structured
.mdfile summarizing outputs. This is the recommended approach going forward. - PDF reports — converts an inline Markdown report's
contentinto a PDF viacreate_pdf_output(). Requires the optionalreportingextra (pip install ".[reporting]").
These are available as separate functions, so end users can choose the format appropriate to their context.
Analytics Store
The analytics store provides persistent, queryable storage for capability results. While the Run Cache stores full Python objects for reuse, the Analytics Store distills results into flat scalar records that can be queried with SQL.
Each capability can opt in by defining a Record class (inheriting from
BaseRecord) and implementing an extract() method on its Run class.
flowchart LR
subgraph "Capability Runs"
R1["DataevalCleaning Run"]
R2["DataevalBias Run"]
R3["MaiteEvaluation Run"]
R4["DataevalFeasibility Run"]
R5["DataevalShift Run"]
R6["NrtkRobustness Run"]
R7["XaitkExplainable Run"]
end
R1 -->|"extract()"| Rec1["DataevalCleaningRecord<br>(duplicates, outliers, ...)"]
R2 -->|"extract()"| Rec2["DataevalBiasRecord<br>(coverage, balance, diversity)"]
R3 -->|"extract()"| Rec3["MaiteEvaluationRecord<br>(metric key/value pairs)"]
R4 -->|"extract()"| Rec4["DataevalFeasibilityRecord<br>(BER bounds, health stats)"]
R5 -->|"extract()"| Rec5["DataevalShiftRecord<br>(drift tests, OOD stats)"]
R6 -->|"extract()"| Rec6["NrtkRobustnessRecord<br>(per-theta metric values)"]
R7 -->|"extract()"| Rec7["XaitkExplainableRecord<br>(saliency stats per map)"]
subgraph "AnalyticsStore.write()"
Rec1 --> W[Auto-populate RunRecord]
Rec2 --> W
Rec3 --> W
Rec4 --> W
Rec5 --> W
Rec6 --> W
Rec7 --> W
end
W --> P["StorageBackend"]
subgraph "Storage (Parquet by default)"
P --> T1["dataeval_cleaning/"]
P --> T2["dataeval_bias/"]
P --> T3["maite_evaluation/"]
P --> T4["dataeval_feasibility/"]
P --> T5["dataeval_shift/"]
P --> T6["nrtk_robustness/"]
P --> T7["xaitk_explainable/"]
P --> T7["runs/"]
end
Records follow these rules:
- Scalar fields only —
str,int,float,bool,bytes,datetime, orOptionalvariants. No lists, dicts, or nested models. - One table per capability — each
Recordsubclass declares atable_name(e.g.,"dataeval_cleaning"). -
Cross-capability JOINs — single-dataset capabilities include a
dataset_idfield, enabling queries like:SELECT c.exact_duplicate_ratio, f.ber_upper, m.output_value FROM dataeval_cleaning c JOIN dataeval_feasibility f ON c.dataset_id = f.dataset_id JOIN maite_evaluation m ON c.dataset_id = m.dataset_id WHERE m.output_key = 'accuracy'Multi-dataset capabilities (e.g., shift) use descriptive ID fields (
reference_dataset_id,evaluation_dataset_id) and can JOIN on either side. -
Idempotent writes — records are deduplicated by
run_uidacross write calls. - Append-only — run results are historical facts; no updates or deletes.
created_at— auto-populated timestamp on every record; no need to add your own.
To add analytics store support to a new capability, define a BaseRecord
subclass and implement extract() on your Run class. See the reference
notebook for detailed implementation guidance.
For a complete list of available tables and their fields, see the Record Schema Reference (Part 5).
For hands-on usage examples (creating a store, writing runs, querying via SQL), see the Analytics Store Tutorial.