configure_job_backend(...)
This page documents the configure_job_backend(...) contract used by job submission.
configure_job_backend(...) selects the active job backend and captures the
deployment-specific settings that cannot be inferred from an individual
submit_capability(...) call.
Why backend configuration is required
Synchronous capability.run(...) executes in the caller's Python process. That
process already knows its local filesystem, cache, credentials, installed
packages, and result storage location.
Job submission separates those concerns:
- the client submits work and later observes job state,
- the job backend decides where and how that work runs,
- workers execute capability code in a potentially different process, node, container, or cluster,
- and durable outputs must be written somewhere both workers and clients can access.
So the backend needs explicit configuration for the execution environment and coordination boundary. Depending on the backend, that includes:
- which backend implementation to use (
"ray","ray-simple", or future backends), - where to connect for execution (
addressfor Ray clusters), - how workers should be prepared (
runtime_env), - where workers should persist structured analytics results (
analytics_store), - where workers should independently externalize oversized inline reports (
artifact_store), - that job-submission workers run with capability-local caching disabled
(
use_cache=False) because worker-local caches are ephemeral and not shared, - which clients should share job identity, dedupe, and reattach behavior
(
idempotency_scopefor the registry-backed Ray backend), - which native Ray namespace owns the shared backend state
(
registry_namespace; each namespace has one fixed internal registry actor), - actor pending-call limits (
registry_max_pending_calls,controller_max_pending_calls) for handles returned when this client creates an actor; Ray cannot apply them to reattached handles, - and operational settings such as timeouts, retention, cleanup, and actor resource placement.
Provenance fields such as user_id, workspace_id, environment, executor,
cluster_id, and request_id are not backend settings. Set them with
checkmaite.configure_provenance(...). Job backends add job-specific fields such
as job_id, backend name, and submission/completion times when they write run
history.
The goal is to make the backend's execution, coordination, storage, and operational assumptions explicit before jobs are submitted.
API shape
from checkmaite.jobs import configure_job_backend
configure_job_backend(
"ray",
address="ray://cluster-head:10001",
idempotency_scope="my-workspace-or-experiment",
runtime_env={
"working_dir": ".",
"env_vars": {"MODEL_REGISTRY_URL": "https://registry.internal"},
},
analytics_store={
"backend": "parquet",
"uri": "s3://team-checkmaite/analytics-store",
# optional:
# "storage_options": {...},
},
artifact_store={
"uri": "s3://team-checkmaite/report-artifacts",
# optional:
# "storage_options": {...},
},
)
The first positional argument selects the job backend. All other keyword arguments are backend-specific configuration forwarded to that backend's constructor.
analytics_store and artifact_store are deliberately separate. The former
stores structured, queryable records; the latter stores opaque report files.
They may point into the same bucket, but can use different storage systems,
credentials, and retention policies.
artifact_store is required for both Ray backends and is not a file-discovery
service. Its uri must be a concrete directory or object-store prefix such as
s3://team-checkmaite/report-artifacts, not a glob such as
s3://team-checkmaite/report-artifacts/*. Supported protocols are local/file,
S3 (s3/s3a), GCS (gs/gcs), and Azure Gen2 (abfs/az). Local paths
must be absolute and must identify a filesystem mounted at the same path for the
client and every Ray worker. Process-local memory storage is not supported.
Remote stores require their corresponding s3fs, gcsfs,
or adlfs implementation. Put credentials, including Azure SAS tokens, in
storage_options; URI query credentials are rejected so they cannot become part
of an artifact path or returned report URI. The job backend
uses it only to externalize an InlineTextReport that exceeds the 256 KiB
metadata limit.
Report producers must return self-contained inline content: embedded resources
should use forms such as data: URIs, and any external links must already be
reachable by report consumers. An ArtifactReport must likewise contain a URI
to an artifact that the producer has already placed in durable storage. Ray
workers do not inspect Markdown, copy referenced files, or rewrite report links.
If the configured store cannot publish and verify an oversized report, the job
fails rather than recording
successful completion with a missing report. Never return worker-local paths,
relative file references, or file:// URIs from
a capability intended for distributed execution.
Ray job backend choices:
"ray": registry/controller-backed Ray job backend for shared clusters, dedupe, and reattach across client restarts. Requiresidempotency_scope."ray-simple": direct Ray task-based job backend for local or single-driver workflows. It does not use a registry and does not takeidempotency_scope.
Examples
Local development with reattachable jobs
from pathlib import Path
output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
"ray",
address="local",
idempotency_scope="local-dev",
analytics_store={"backend": "parquet", "uri": str(output_root / "analytics")},
artifact_store={"uri": str(output_root / "report-artifacts")},
)
Local development with the simple Ray job backend
from pathlib import Path
output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
"ray-simple",
address="local",
analytics_store={"backend": "parquet", "uri": str(output_root / "analytics")},
artifact_store={"uri": str(output_root / "report-artifacts")},
)
Shared Ray cluster with object storage
configure_job_backend(
"ray",
address="ray://cluster-head:10001",
idempotency_scope="team-a-prod-evals",
runtime_env={
"working_dir": ".",
"env_vars": {"AWS_REGION": "us-east-1"},
},
analytics_store={
"backend": "parquet",
"uri": "s3://team-checkmaite/analytics-store",
"storage_options": {"anon": False},
},
artifact_store={
"uri": "s3://team-checkmaite/report-artifacts",
"storage_options": {"anon": False},
},
)
Shared status + reattach configuration
configure_job_backend(
"ray",
address="ray://cluster-head:10001",
analytics_store={"backend": "parquet", "uri": "s3://team-checkmaite/analytics-store"},
artifact_store={"uri": "s3://team-checkmaite/report-artifacts"},
idempotency_scope="team-a-notebooks",
# Use a different Ray namespace when an independent registry is required.
registry_namespace="checkmaite_jobs",
controller_retention_s=3600,
max_retained_terminal_controllers=1000,
)
registry_namespace uses Ray's built-in named-resource namespace. It does not
create a new Ray cluster, Kubernetes namespace, resource quota, or security
boundary. CheckMAITE's registry actor name is fixed and internal; users select
an independent registry by selecting another Ray namespace. All clients sharing
a namespace must use compatible registry settings.
For detailed Ray runtime behavior, see Ray job backend and Ray simple job backend. For worker image and cluster environment guidance, see Worker environments. For store semantics, provenance, and URI resolution details, see Distributed analytics store.
Common problems
- Report images are missing after a job finishes, or later jobs overwrite
them. Images written to a worker's local cache disappear when the worker
restarts and aren't visible to the client. Configure
artifact_storewith a durable URI, such as object storage or an absolute path shared by the client and every Ray node. A process-localmemorystore or a relative path is rejected. (#737)
Related tutorials
- Ray Simple Job Submission:
configures
ray-simpleand submits a capability. - Ray Job Submission:
configures
rayagainst a shared cluster. - Analytics Store: the store that job backends write results to.
- Object Detection Workflow via API and Image Classification Workflow via API: the capabilities you would submit.