Skip to content

Ray job backend (kind="ray")

The default Ray job backend uses Ray Core with a detached registry actor and detached per-job controller actors.

This page explains why Ray is a good fit, how the default "ray" job backend maps onto Ray's execution model, and how to use it from CheckMAITE. For the direct process-local Ray task-based job backend, see Ray simple job backend.

Why Ray

Ray is a distributed Python runtime designed for:

  • task-parallel execution,
  • actor-based stateful services, where an actor is a named Python object that keeps state across remote method calls,
  • and dynamic CPU/GPU scheduling.

That lines up well with CheckMAITE's needs:

  • notebook users want to submit work without blocking,
  • capabilities may need CPUs or GPUs,
  • development should work locally while the same API scales to larger infrastructure when available.

Ray's execution model in one paragraph

At the Ray Core level, distributed computation is built from a few primitives:

  • ray.init(...) connects to or starts a Ray runtime,
  • ray.remote(...) turns a Python function or class into distributed work,
  • calling .remote(...) submits that work and returns an ObjectRef,
  • ray.get(...) waits for the value,
  • ray.wait(...) checks whether it is ready,
  • and ray.cancel(...) requests cancellation.

The current CheckMAITE job backend uses one detached per-job controller actor plus one Ray task per submitted capability run. The controller actor owns the live ObjectRef, while the public RayJob handle reads shared lifecycle metadata from the registry.

End-to-end flow

sequenceDiagram
    participant Client
    participant JobBackend as RayJobBackend
    participant Registry as Detached registry actor
    participant Controller as Detached job controller actor
    participant Ray as Ray runtime
    participant Worker as Ray worker
    participant Store as AnalyticsStore

    Client->>JobBackend: submit_capability(...)
    JobBackend->>Registry: register_or_get(scope, scoped_run_key)
    alt existing
        Registry-->>JobBackend: existing job metadata + controller name
        JobBackend-->>Client: RayJob (reattached)
    else new
        Registry-->>JobBackend: new reservation
        JobBackend->>Controller: create/get named detached controller
        JobBackend->>Controller: start(capability, kwargs, resources)
        Controller->>Registry: mark_scheduling(...)
        Controller->>Ray: submit capability worker
        Worker->>Controller: worker_started(node/pod metadata)
        Controller->>Registry: mark_worker_running(...)
        JobBackend-->>Client: RayJob
        Ray->>Worker: execute _execute_capability_ref
        Worker->>Worker: capability.run(..., use_cache=False)
        Worker->>Store: write_with_receipt([run], provenance=...)
        Store-->>Worker: payload URI
        Worker-->>Controller: CapabilityRunRef or exception
        Controller->>Registry: update_terminal(..., result_ref/error)
        Registry->>Registry: retain controller briefly, then cleanup/sweep
    end

The registry/controller split is the key design shown in the flow:

  1. The registry actor is the shared directory for job metadata. register_or_get(...) either reserves a new job ID or returns an existing record for the same scoped run key. Later get_job(...) and list_jobs(...) calls read from this registry, so clients do not need the original submitting process to still be alive.
  2. The controller actor is the per-job owner of live execution. After the registry accepts a new reservation, the backend creates or finds the named detached controller and records SCHEDULING while Ray places the worker. The worker's startup handshake records RUNNING only after execution begins.
  3. The controller owns the Ray task ObjectRef, not the notebook/client. It watches the task complete, handles cancellation requests, and writes terminal status plus the serialized CapabilityRunRef back to the registry.
  4. The public RayJob handle is therefore a lightweight client-side view over shared registry state and, while the controller is retained, controller reconciliation/cancellation methods.

Public usage

1. Configure the job backend

from pathlib import Path

from checkmaite.jobs import configure_job_backend

output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
    "ray",
    address="local",
    analytics_store={
        "backend": "parquet",
        "uri": str(output_root / "analytics"),
    },
    artifact_store={"uri": str(output_root / "report-artifacts")},
    idempotency_scope="team-a-notebooks",
    registry_namespace="checkmaite_jobs",
)

Important:

  • analytics_store=... is required,
  • idempotency_scope=... is required and has no default,
  • capability-local caching is disabled in job submission mode; use_cache=True is rejected because Ray workers are ephemeral and do not share a local cache,
  • the scope should be a stable workspace, project, or experiment identifier,
  • analytics-store configuration is separate from Ray connection/runtime settings,
  • it is forwarded to worker tasks so they know where structured results should be written,
  • artifact_store=... is independently required and must point to supported object storage or an absolute local path mounted identically for the client, every worker, and report consumers,
  • submission is deduplicated by (idempotency_scope, scoped_run_key), and get_job(job_id) / list_jobs() can reattach across client restarts as long as the same Ray cluster, registry_namespace, and scope are reused.
  • worker-side analytics-store writes include provenance resolved on the submitter plus dynamic job metadata: job_id, backend="ray", submitted_at, completed_at, and run_event_id=job_id.

2. Configure provenance defaults, if needed

from checkmaite import configure_provenance

configure_provenance(
    user_id="alice@company.com",
    workspace_id="team-a-notebooks",
    environment="ray-cluster-prod",
    executor="ray",
    cluster_id="cluster-42",
    request_id="req-123",
)

These defaults are optional and process-wide. On submission, the Ray backend reads them from the client process with get_provenance_defaults(), merges in registry-assigned job metadata (job_id, submitted_at) plus backend metadata (backend="ray", run_event_id=job_id), and includes that resolved provenance in the kwargs sent to the controller.

The controller passes that provenance to the worker task with the rest of the submission payload. It may add or carry controller/job metadata, but it does not ask the worker to recompute client defaults. After capability.run(...) finishes, the worker adds completed_at and passes the final provenance explicitly to write_with_receipt(...). The auto-generated runs rows then persist those values as flat columns.

3. Submit work

from checkmaite.jobs import submit_capability

job = submit_capability(
    capability,
    datasets=[dataset],
    models=[model],
    metrics=[metric],
    config=config,
    job_name="September COCO evaluation",
    use_cache=False,
)

use_cache=False is not just a recommendation: job submission rejects use_cache=True. Worker-local caches are ephemeral and are not shared with the client or other workers. Reuse in this backend comes from registry dedupe for the same (idempotency_scope, scoped_run_key) and durable analytics-store writes; future shared-cache support would need an explicit remote cache backend.

4. Inspect lifecycle and retrieve the result reference

print(job.job_id)       # Canonical Checkmaite job ID
print(job.job_name)     # User label, or the capability ID by default
print(job.status)
print(job.wait(timeout=0.1))

ref = job.result(timeout=300)
print(ref.run_uid)
print(ref.store_uri)
print(ref.outputs_uri)  # None today

The returned object is CapabilityRunRef, not a full CapabilityRunBase. The bounded job name is a display label and does not affect run identity or deduplication. It defaults to the capability ID when the user does not provide one. A duplicate submission keeps the original job's label.

5. List jobs

from checkmaite.jobs import JobStatus, list_jobs

recent = list_jobs(limit=100)
completed = list_jobs(limit=50, status_filter=JobStatus.COMPLETED)

list_jobs(...) is intentionally limited/paginated. The registry applies filters and limits before copying records back to the client. Handles returned from listing are attached to the current backend instance, so shutdown(wait=True) treats them as tracked jobs. Use before_submitted_at_ts as a simple cursor for older pages.

Recover after a notebook failure

Reconnect to the same Ray namespace and registry actor with the same stable idempotency_scope, then call list_jobs(). Use job.job_name, job.created_at, and job.status to identify a job whose ID was not saved. When the original logical inputs can be reconstructed, resubmitting them in the same scope also returns the canonical active or completed job through deduplication. get_job(job_id) is the direct path when the ID is known.

job.job_id is the canonical Checkmaite identifier used by the registry, controllers, cancellation, and provenance. A Ray Dashboard driver ID identifies a client session instead: it can change after reconnecting, can own several Checkmaite submissions, and cannot be passed to get_job().

Reconfiguring the process-global backend calls shutdown(wait=False) on the old client object. It changes which backend top-level API calls use, but does not cancel detached jobs or invalidate existing job handles. Avoid force_reinit=True while work is active. Terminating a Ray Client driver also does not mean cancelling each Checkmaite job it submitted; retain or recover the job and call job.cancel() for job-specific cancellation.

Resource scheduling

The job backend distinguishes handing a submission to its controller, waiting for Ray resources, and executing capability code:

PENDING -> SCHEDULING -> RUNNING -> terminal state

SCHEDULING begins when the controller submits the worker request to Ray. RUNNING begins only after the worker has actually started and completed a startup handshake. job.scheduling_info reports requested resources, elapsed scheduling time, worker start time, Ray node identity, and pod placement fields when the deployment exposes them. Configure the Kubernetes downward API as CHECKMAITE_KUBERNETES_NODE_NAME when the Kubernetes node name is required.

The job backend resolves CPU/GPU requirements in the following order:

  1. explicit resources={...} passed at submission time,
  2. hints on the config object (config.num_cpus, config.num_gpus),
  3. capability defaults (default_num_cpus, default_num_gpus),
  4. fallback (num_cpus=1, num_gpus=0).

Example:

job = submit_capability(
    capability,
    datasets=[dataset],
    resources={"num_cpus": 4, "num_gpus": 1, "memory": 8_000_000_000},
)

Registry and controller startup use separate readiness handshakes and registry_startup_timeout_s / controller_startup_timeout_s budgets. Each handshake uses one Checkmaite compatibility version plus typed immutable actor configuration; Ray remains responsible for Python and Ray runtime compatibility. If this client creates a controller that fails its readiness handshake, it best-effort kills that controller before abandoning the registry reservation; a pre-existing controller is never killed merely because this client times out. Submission failures expose their phase through JobSubmissionError.phase, distinguishing validation, registration, controller startup, attachment, and worker scheduling. Capability and analytics/report dependencies are loaded lazily in the capability worker rather than during controller process startup.

By default, scheduling_timeout_s=1800 fails a task that Ray cannot place within 30 minutes. This timer is owned by the detached controller and continues after the submitting notebook exits. It is separate from job.wait(timeout=...), which is only a client-side wait limit. Setting scheduling_timeout_s=None explicitly allows an unbounded resource wait and emits a runtime warning that pending work may prevent idle scale-down, leave cluster nodes running, and cause significant additional compute charges.

Checkmaite does not duplicate Ray's worker-discovery or autoscaling policy. Ray remains the source of truth for whether current or future cluster capacity can satisfy the request; scheduling_timeout_s bounds that wait.

max_active_jobs_per_scope bounds accepted non-terminal work and max_scheduling_jobs_per_scope bounds worker requests waiting in Ray's resource queue. Capacity rejections raise BackpressureError; Ray resources continue to determine actual execution concurrency.

Status mapping

RayJob is a thin protocol wrapper over shared registry lifecycle state.

Primary mapping comes from registry state:

  • SUBMITTING -> PENDING
  • SCHEDULING -> SCHEDULING
  • RUNNING / CANCELLING -> RUNNING
  • COMPLETED -> COMPLETED
  • FAILED -> FAILED
  • CANCELLED -> CANCELLED

RayJob does not own or use the live task ObjectRef directly. The detached controller owns that reference. Client handles resolve terminal payloads from registry metadata (result_ref) and ask the controller to reconcile registry state from its owned task when needed.

Timeouts and cancellation

from checkmaite.jobs import JobTimeoutError

try:
    ref = job.result(timeout=120)
except JobTimeoutError:
    job.cancel()
    raise

Notes:

  • job.result(timeout=...) is a client-side wait timeout,
  • timing out does not automatically cancel the remote task,
  • job.cancel() records shared cancellation intent (CANCELLING) and asks the detached controller actor to call ray.cancel(...) when the controller is reachable,
  • cancellation can be requested while PENDING, SCHEDULING, or RUNNING,
  • CANCELLING still maps to public RUNNING until the controller observes and commits a terminal state,
  • if the cancellation request is dispatched but the acknowledgement times out, cancel() may return True even though final cancellation is still pending,
  • if the registry accepts cancellation while the controller is temporarily unreachable, cancel() returns True; a recovered controller observes the shared intent through its heartbeat, while an expired controller lease eventually finalizes the record as CANCELLED,
  • if the controller has already been cleaned up then the job is already terminal and cancellation is a no-op.

Shutdown semantics

RayJobBackend.shutdown(wait=True) waits for jobs attached to that backend instance and then calls ray.shutdown() for the local client process.

RayJobBackend.shutdown(wait=False) is intentionally a non-blocking local-handle handoff: it does not wait for jobs and does not call ray.shutdown(). This is used by configure_job_backend(...) when replacing the active backend so running detached controllers can continue in the Ray cluster and a new backend can reattach to them. Call ray.shutdown() explicitly, or use shutdown(wait=True), when you really want to disconnect the local Ray runtime.

Registry configuration notes

The registry has two user-visible identity levels:

Ray cluster
└── registry_namespace       selects one independent registry
    └── idempotency_scope    partitions jobs inside that registry

registry_namespace is a native Ray namespace. It is sent to Ray when CheckMAITE creates or looks up its fixed internal checkmaite_job_registry actor. There is no user-configurable registry actor name. Keep the namespace stable across sessions to reconnect to the same registry, and select a different namespace for migration, testing, load sharding, or independent retention and resource settings.

A Ray namespace does not create a separate Ray cluster, Kubernetes namespace, worker pool, quota, or security boundary. Registries in different namespaces on the same cluster still share Ray's scheduler and workers. Use a separate Ray cluster when infrastructure or security isolation is required. In KubeRay, no extra Kubernetes object is needed for a Ray namespace: Ray's GCS records the namespace when the named actor is created.

The registry keys every lookup and deduplication entry by idempotency_scope, so different scopes in one namespace do not list or deduplicate against each other. Scopes are logical partitions, not security boundaries. Controller-count and sweep-batch limits apply to the registry as a whole, not separately to each scope, so activity in one scope can cause terminal controllers from another scope to be cleaned up.

Registry creation settings are immutable for the lifetime of the registry in a namespace. Each client performs a readiness and compatibility handshake before reuse. If settings differ, either make clients agree or choose another registry_namespace; CheckMAITE never silently creates another actor name in the same namespace. A single CheckMAITE compatibility version covers the actor protocol and record/result schemas; incompatible versions are rejected early. Ray remains responsible for Python and Ray runtime compatibility. registry_startup_timeout_s controls this cold-start handshake separately from normal registry update timeouts.

The registry reserves zero Ray CPUs by default. It remains a long-lived detached control-plane actor and should stay small.

Pending-call limits and backpressure

Ray's pending-call limits belong to client-side actor handles, not to the actors themselves. When a client creates an actor, registry_max_pending_calls=1024 bounds the registry handle returned by that creation call and controller_max_pending_calls=64 similarly bounds a newly created controller's initial handle. These limits can protect that client from unbounded queue growth during bursty submits, polling, or cancellation.

A handle later obtained through ray.get_actor() is a distinct object. Ray does not provide a public API for applying max_pending_calls to that handle, so reattached registry and controller handles use Ray's unbounded default. Pending- call limits therefore are not immutable actor settings and do not participate in the registry compatibility handshake.

If Ray rejects a call on a bounded creation handle because its pending-call limit is full, client APIs raise BackpressureError. Treat that as a retryable overload signal: retry with exponential backoff and jitter, reduce client-side concurrency, or tune registry_max_pending_calls / controller_max_pending_calls for the expected creation-client burst size. Passing None leaves a creation handle unbounded. Reattached clients should bound their own submission and polling concurrency; CheckMAITE does not add a separate client-side queue.

Dedupe policy in the current implementation:

  • SUBMITTING, SCHEDULING, RUNNING, and COMPLETED records keep the dedupe key, so a duplicate submit in the same scope returns the existing job.
  • FAILED and CANCELLED records release the dedupe key, so a later submit of the same logical work can create a fresh job.
  • expired SUBMITTING reservations are marked FAILED and also release the dedupe key.
  • stale RUNNING records whose controller heartbeat lease expires are marked FAILED; stale CANCELLING records become CANCELLED. Both release the dedupe key.

Terminal controllers schedule their own retention sweep after successfully publishing terminal state. Consequently, controller_retention_s=0 cleans a controller on a quiet cluster without requiring a later submission or an operator-triggered sweep. Explicit sweep_registry() remains available for maintenance and recovery.

Controller terminal updates are best-effort and bounded by registry_update_timeout_s (default: 5 seconds). If one terminal update times out, the controller keeps the terminal result in its own actor state, retries the registry commit with bounded backoff, and clients can also commit that terminal state on the next observe/reconcile path. While a live controller still has uncommitted terminal state, its heartbeat keeps the registry lease fresh so the job is not swept as stale.

Assumptions

Memory and payload assumptions

The registry actor is intentionally thin. Its Python dictionaries live in the registry actor worker's heap, not in Ray's object store. Registry records should stay small and JSON-like:

  • job id, scope, status,
  • timestamps,
  • controller actor name,
  • short error summary,
  • small terminal CapabilityRunRef metadata.

The registry and controllers must not store large logs, model weights, datasets, checkpoints, full CapabilityRunBase payloads, or unbounded history. Large data belongs in the analytics store, object storage, shared filesystems, databases, or platform logging systems. Terminal job records are retained only for the configured registry retention window/count; once purged, their dedupe entries are also removed and identical future submissions may create fresh jobs. Small, self-contained text reports use CapabilityRunRef.report with kind="inline_text"; inline UTF-8 content is limited to 256 KiB. A larger inline report is published through artifact_store to a run- and job-scoped, path-safe content-addressed key and returned as an artifact report. Retries reuse a verified matching object. A mismatched object is replaced through a supported atomic local rename or object-store commit; if safe replacement is unavailable or fails, publication fails without deleting the existing key. Supported remote implementations are S3, GCS, and Azure through s3fs, gcsfs, and adlfs. The configured artifact_store.uri must be a concrete directory or object-store prefix, not a wildcard or glob. Publication uses a temporary key before the final key becomes visible, so retries cannot truncate an existing artifact. A publication or verification failure fails the job, allowing the logical run to be submitted again after the storage problem is corrected. Optional artifact_store.storage_options are forwarded to the selected supported filesystem implementation.

Report dependencies are a producer concern, not a backend discovery mechanism. Inline reports must be self-contained: embed generated resources with forms such as data: URIs, or link only to locations that report consumers can already reach. Large, binary, or multi-file report producers must publish those products themselves and return an ArtifactReport with a durable URI. The Ray worker does not parse Markdown, copy referenced files, verify producer-owned remote URIs, or rewrite links. A worker-local path, relative file link, or file:// URI will not be usable after execution and must not be returned.

Artifact storage is deliberately independent from the analytics store: deployments may colocate both under one bucket, but they can use different backends, credentials, and retention policies.

During controller.start(...), the controller temporarily receives the capability and run arguments so it can launch the worker task. It does not retain those objects on actor state after submission. If future workloads involve very large submission payloads, a more memory-conscious design can pass a Ray object store reference instead:

client -> ray.put(submission_payload)
client -> controller.start(payload_ref)
controller -> worker task consumes payload_ref

That optimization trades controller heap pressure for Ray object-store pressure and should be introduced only with explicit object-lifetime management.

Cluster-lifetime and durability assumptions

Detached registry/controller actors survive notebook or client-process exits, but they do not survive Ray cluster loss. Current behavior assumes:

  • if a client process exits but the Ray cluster remains alive, running jobs can continue and later clients can reattach using the same Ray cluster, registry namespace, and idempotency scope;
  • if the Ray cluster dies, in-memory registry state, controller actors, and running tasks are lost;
  • durable run data that was successfully written before cluster loss remains in the configured analytics store;
  • jobs that were RUNNING when the cluster died require a future persistent job-metadata store and recovery policy to mark failed, reconcile from the analytics store, or resubmit.

The current in-memory registry is not a long-term job-history database. A production deployment that needs history across cluster restarts should add a persistent registry metadata store (for example Postgres, MongoDB, or object storage) with one small row/document per job. The current record shape is kept small and serializable to leave room for that future durable store.

Scaling assumptions

This design targets relatively heavyweight ML evaluation jobs where each job runs long enough that one controller actor per job is acceptable. It is not a high-throughput system for huge numbers of tiny tasks.

Operational considerations:

  • each detached controller actor has Ray control-plane metadata and Python heap overhead;
  • controller_num_cpus defaults to a small non-zero value to give the Ray scheduler backpressure;
  • controller_memory can reserve Ray actor heap memory when supported by the deployed Ray version;
  • platform deployments can use controller_resources to place controllers on control-plane nodes via custom Ray resources;
  • terminal controller retention must be bounded with controller_retention_s and max_retained_terminal_controllers.

If a deployment needs thousands of concurrent small jobs or very high submit rates, a fixed controller-pool design may be better than one detached controller actor per job. In that model, a bounded set of actors owns many job refs and the registry maps each job to a pool actor. That design reduces actor churn but is more complex and has a larger failure domain per pool actor.

Common problems

  • The notebook crashed before the job ID was saved. The job keeps running in its detached controller. Reconnect to the same Ray namespace with the same idempotency_scope and call list_jobs(), as described in Recover after a notebook failure. (#735)
  • A tutorial's local settings were copied to a shared cluster. Settings such as address="local" and force_reinit=True are for a single-user machine. On a shared cluster, connect to the cluster address, keep a stable idempotency_scope, and avoid force_reinit=True while work is active. (#737)