Skip to content

configure_job_backend(...)

This page documents the configure_job_backend(...) contract used by job submission.

configure_job_backend(...) selects the active job backend and captures the deployment-specific settings that cannot be inferred from an individual submit_capability(...) call.

Why backend configuration is required

Synchronous capability.run(...) executes in the caller's Python process. That process already knows its local filesystem, cache, credentials, installed packages, and result storage location.

Job submission separates those concerns:

  • the client submits work and later observes job state,
  • the job backend decides where and how that work runs,
  • workers execute capability code in a potentially different process, node, container, or cluster,
  • and durable outputs must be written somewhere both workers and clients can access.

So the backend needs explicit configuration for the execution environment and coordination boundary. Depending on the backend, that includes:

  • which backend implementation to use ("ray", "ray-simple", or future backends),
  • where to connect for execution (address for Ray clusters),
  • how workers should be prepared (runtime_env),
  • where workers should persist structured analytics results (analytics_store),
  • where workers should independently externalize oversized inline reports (artifact_store),
  • that job-submission workers run with capability-local caching disabled (use_cache=False) because worker-local caches are ephemeral and not shared,
  • which clients should share job identity, dedupe, and reattach behavior (idempotency_scope for the registry-backed Ray backend),
  • which native Ray namespace owns the shared backend state (registry_namespace; each namespace has one fixed internal registry actor),
  • actor pending-call limits (registry_max_pending_calls, controller_max_pending_calls) for handles returned when this client creates an actor; Ray cannot apply them to reattached handles,
  • and operational settings such as timeouts, retention, cleanup, and actor resource placement.

Provenance fields such as user_id, workspace_id, environment, executor, cluster_id, and request_id are not backend settings. Set them with checkmaite.configure_provenance(...). Job backends add job-specific fields such as job_id, backend name, and submission/completion times when they write run history.

The goal is to make the backend's execution, coordination, storage, and operational assumptions explicit before jobs are submitted.

API shape

from checkmaite.jobs import configure_job_backend

configure_job_backend(
    "ray",
    address="ray://cluster-head:10001",
    idempotency_scope="my-workspace-or-experiment",
    runtime_env={
        "working_dir": ".",
        "env_vars": {"MODEL_REGISTRY_URL": "https://registry.internal"},
    },
    analytics_store={
        "backend": "parquet",
        "uri": "s3://team-checkmaite/analytics-store",
        # optional:
        # "storage_options": {...},
    },
    artifact_store={
        "uri": "s3://team-checkmaite/report-artifacts",
        # optional:
        # "storage_options": {...},
    },
)

The first positional argument selects the job backend. All other keyword arguments are backend-specific configuration forwarded to that backend's constructor.

analytics_store and artifact_store are deliberately separate. The former stores structured, queryable records; the latter stores opaque report files. They may point into the same bucket, but can use different storage systems, credentials, and retention policies.

artifact_store is required for both Ray backends and is not a file-discovery service. Its uri must be a concrete directory or object-store prefix such as s3://team-checkmaite/report-artifacts, not a glob such as s3://team-checkmaite/report-artifacts/*. Supported protocols are local/file, S3 (s3/s3a), GCS (gs/gcs), and Azure Gen2 (abfs/az). Local paths must be absolute and must identify a filesystem mounted at the same path for the client and every Ray worker. Process-local memory storage is not supported. Remote stores require their corresponding s3fs, gcsfs, or adlfs implementation. Put credentials, including Azure SAS tokens, in storage_options; URI query credentials are rejected so they cannot become part of an artifact path or returned report URI. The job backend uses it only to externalize an InlineTextReport that exceeds the 256 KiB metadata limit. Report producers must return self-contained inline content: embedded resources should use forms such as data: URIs, and any external links must already be reachable by report consumers. An ArtifactReport must likewise contain a URI to an artifact that the producer has already placed in durable storage. Ray workers do not inspect Markdown, copy referenced files, or rewrite report links. If the configured store cannot publish and verify an oversized report, the job fails rather than recording successful completion with a missing report. Never return worker-local paths, relative file references, or file:// URIs from a capability intended for distributed execution.

Ray job backend choices:

  • "ray": registry/controller-backed Ray job backend for shared clusters, dedupe, and reattach across client restarts. Requires idempotency_scope.
  • "ray-simple": direct Ray task-based job backend for local or single-driver workflows. It does not use a registry and does not take idempotency_scope.

Examples

Local development with reattachable jobs

from pathlib import Path

output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
    "ray",
    address="local",
    idempotency_scope="local-dev",
    analytics_store={"backend": "parquet", "uri": str(output_root / "analytics")},
    artifact_store={"uri": str(output_root / "report-artifacts")},
)

Local development with the simple Ray job backend

from pathlib import Path

output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
    "ray-simple",
    address="local",
    analytics_store={"backend": "parquet", "uri": str(output_root / "analytics")},
    artifact_store={"uri": str(output_root / "report-artifacts")},
)

Shared Ray cluster with object storage

configure_job_backend(
    "ray",
    address="ray://cluster-head:10001",
    idempotency_scope="team-a-prod-evals",
    runtime_env={
        "working_dir": ".",
        "env_vars": {"AWS_REGION": "us-east-1"},
    },
    analytics_store={
        "backend": "parquet",
        "uri": "s3://team-checkmaite/analytics-store",
        "storage_options": {"anon": False},
    },
    artifact_store={
        "uri": "s3://team-checkmaite/report-artifacts",
        "storage_options": {"anon": False},
    },
)

Shared status + reattach configuration

configure_job_backend(
    "ray",
    address="ray://cluster-head:10001",
    analytics_store={"backend": "parquet", "uri": "s3://team-checkmaite/analytics-store"},
    artifact_store={"uri": "s3://team-checkmaite/report-artifacts"},
    idempotency_scope="team-a-notebooks",
    # Use a different Ray namespace when an independent registry is required.
    registry_namespace="checkmaite_jobs",
    controller_retention_s=3600,
    max_retained_terminal_controllers=1000,
)

registry_namespace uses Ray's built-in named-resource namespace. It does not create a new Ray cluster, Kubernetes namespace, resource quota, or security boundary. CheckMAITE's registry actor name is fixed and internal; users select an independent registry by selecting another Ray namespace. All clients sharing a namespace must use compatible registry settings.

For detailed Ray runtime behavior, see Ray job backend and Ray simple job backend. For worker image and cluster environment guidance, see Worker environments. For store semantics, provenance, and URI resolution details, see Distributed analytics store.

Common problems

  • Report images are missing after a job finishes, or later jobs overwrite them. Images written to a worker's local cache disappear when the worker restarts and aren't visible to the client. Configure artifact_store with a durable URI, such as object storage or an absolute path shared by the client and every Ray node. A process-local memory store or a relative path is rejected. (#737)