Ray job backend (kind="ray")
The default Ray job backend uses Ray Core with a detached registry actor and detached per-job controller actors.
This page explains why Ray is a good fit, how the default "ray" job backend
maps onto Ray's execution model, and how to use it from CheckMAITE. For the
direct process-local Ray task-based job backend, see Ray simple job
backend.
Why Ray
Ray is a distributed Python runtime designed for:
- task-parallel execution,
- actor-based stateful services, where an actor is a named Python object that keeps state across remote method calls,
- and dynamic CPU/GPU scheduling.
That lines up well with CheckMAITE's needs:
- notebook users want to submit work without blocking,
- capabilities may need CPUs or GPUs,
- development should work locally while the same API scales to larger infrastructure when available.
Ray's execution model in one paragraph
At the Ray Core level, distributed computation is built from a few primitives:
ray.init(...)connects to or starts a Ray runtime,ray.remote(...)turns a Python function or class into distributed work,- calling
.remote(...)submits that work and returns anObjectRef, ray.get(...)waits for the value,ray.wait(...)checks whether it is ready,- and
ray.cancel(...)requests cancellation.
The current CheckMAITE job backend uses one detached per-job controller
actor plus one Ray task per submitted capability run. The controller actor
owns the live ObjectRef, while the public RayJob handle reads shared
lifecycle metadata from the registry.
End-to-end flow
sequenceDiagram
participant Client
participant JobBackend as RayJobBackend
participant Registry as Detached registry actor
participant Controller as Detached job controller actor
participant Ray as Ray runtime
participant Worker as Ray worker
participant Store as AnalyticsStore
Client->>JobBackend: submit_capability(...)
JobBackend->>Registry: register_or_get(scope, scoped_run_key)
alt existing
Registry-->>JobBackend: existing job metadata + controller name
JobBackend-->>Client: RayJob (reattached)
else new
Registry-->>JobBackend: new reservation
JobBackend->>Controller: create/get named detached controller
JobBackend->>Controller: start(capability, kwargs, resources)
Controller->>Registry: mark_scheduling(...)
Controller->>Ray: submit capability worker
Worker->>Controller: worker_started(node/pod metadata)
Controller->>Registry: mark_worker_running(...)
JobBackend-->>Client: RayJob
Ray->>Worker: execute _execute_capability_ref
Worker->>Worker: capability.run(..., use_cache=False)
Worker->>Store: write_with_receipt([run], provenance=...)
Store-->>Worker: payload URI
Worker-->>Controller: CapabilityRunRef or exception
Controller->>Registry: update_terminal(..., result_ref/error)
Registry->>Registry: retain controller briefly, then cleanup/sweep
end
The registry/controller split is the key design shown in the flow:
- The registry actor is the shared directory for job metadata.
register_or_get(...)either reserves a new job ID or returns an existing record for the same scoped run key. Laterget_job(...)andlist_jobs(...)calls read from this registry, so clients do not need the original submitting process to still be alive. - The controller actor is the per-job owner of live execution. After the
registry accepts a new reservation, the backend creates or finds the named
detached controller and records
SCHEDULINGwhile Ray places the worker. The worker's startup handshake recordsRUNNINGonly after execution begins. - The controller owns the Ray task
ObjectRef, not the notebook/client. It watches the task complete, handles cancellation requests, and writes terminal status plus the serializedCapabilityRunRefback to the registry. - The public
RayJobhandle is therefore a lightweight client-side view over shared registry state and, while the controller is retained, controller reconciliation/cancellation methods.
Public usage
1. Configure the job backend
from pathlib import Path
from checkmaite.jobs import configure_job_backend
output_root = (Path.cwd() / "checkmaite-job-output").resolve()
configure_job_backend(
"ray",
address="local",
analytics_store={
"backend": "parquet",
"uri": str(output_root / "analytics"),
},
artifact_store={"uri": str(output_root / "report-artifacts")},
idempotency_scope="team-a-notebooks",
registry_namespace="checkmaite_jobs",
)
Important:
analytics_store=...is required,idempotency_scope=...is required and has no default,- capability-local caching is disabled in job submission mode;
use_cache=Trueis rejected because Ray workers are ephemeral and do not share a local cache, - the scope should be a stable workspace, project, or experiment identifier,
- analytics-store configuration is separate from Ray connection/runtime settings,
- it is forwarded to worker tasks so they know where structured results should be written,
artifact_store=...is independently required and must point to supported object storage or an absolute local path mounted identically for the client, every worker, and report consumers,- submission is deduplicated by
(idempotency_scope, scoped_run_key), andget_job(job_id)/list_jobs()can reattach across client restarts as long as the same Ray cluster,registry_namespace, and scope are reused. - worker-side analytics-store writes include provenance resolved on the
submitter plus dynamic job metadata:
job_id,backend="ray",submitted_at,completed_at, andrun_event_id=job_id.
2. Configure provenance defaults, if needed
from checkmaite import configure_provenance
configure_provenance(
user_id="alice@company.com",
workspace_id="team-a-notebooks",
environment="ray-cluster-prod",
executor="ray",
cluster_id="cluster-42",
request_id="req-123",
)
These defaults are optional and process-wide. On submission, the Ray backend
reads them from the client process with get_provenance_defaults(), merges in
registry-assigned job metadata (job_id, submitted_at) plus backend metadata
(backend="ray", run_event_id=job_id), and includes that resolved provenance
in the kwargs sent to the controller.
The controller passes that provenance to the worker task with the rest of the
submission payload. It may add or carry controller/job metadata, but it does not
ask the worker to recompute client defaults. After capability.run(...)
finishes, the worker adds completed_at and passes the final provenance
explicitly to write_with_receipt(...). The auto-generated runs rows then
persist those values as flat columns.
3. Submit work
from checkmaite.jobs import submit_capability
job = submit_capability(
capability,
datasets=[dataset],
models=[model],
metrics=[metric],
config=config,
job_name="September COCO evaluation",
use_cache=False,
)
use_cache=False is not just a recommendation: job submission rejects
use_cache=True. Worker-local caches are ephemeral and are not shared with the
client or other workers. Reuse in this backend comes from registry dedupe for the
same (idempotency_scope, scoped_run_key) and durable analytics-store writes;
future shared-cache support would need an explicit remote cache backend.
4. Inspect lifecycle and retrieve the result reference
print(job.job_id) # Canonical Checkmaite job ID
print(job.job_name) # User label, or the capability ID by default
print(job.status)
print(job.wait(timeout=0.1))
ref = job.result(timeout=300)
print(ref.run_uid)
print(ref.store_uri)
print(ref.outputs_uri) # None today
The returned object is CapabilityRunRef, not a full CapabilityRunBase.
The bounded job name is a display label and does not affect run identity or
deduplication. It defaults to the capability ID when the user does not provide
one. A duplicate submission keeps the original job's label.
5. List jobs
from checkmaite.jobs import JobStatus, list_jobs
recent = list_jobs(limit=100)
completed = list_jobs(limit=50, status_filter=JobStatus.COMPLETED)
list_jobs(...) is intentionally limited/paginated. The registry applies
filters and limits before copying records back to the client. Handles returned
from listing are attached to the current backend instance, so shutdown(wait=True)
treats them as tracked jobs. Use before_submitted_at_ts as a simple cursor for
older pages.
Recover after a notebook failure
Reconnect to the same Ray namespace and registry actor with the same stable
idempotency_scope, then call list_jobs(). Use job.job_name,
job.created_at, and job.status to identify a job whose ID was
not saved. When the original logical inputs can be reconstructed, resubmitting
them in the same scope also returns the canonical active or completed job through
deduplication. get_job(job_id) is the direct path when the ID is known.
job.job_id is the canonical Checkmaite identifier used by the registry,
controllers, cancellation, and provenance. A Ray Dashboard driver ID identifies
a client session instead: it can change after reconnecting, can own several
Checkmaite submissions, and cannot be passed to get_job().
Reconfiguring the process-global backend calls shutdown(wait=False) on the old
client object. It changes which backend top-level API calls use, but does not
cancel detached jobs or invalidate existing job handles. Avoid
force_reinit=True while work is active. Terminating a Ray Client driver also
does not mean cancelling each Checkmaite job it submitted; retain or recover the
job and call job.cancel() for job-specific cancellation.
Resource scheduling
The job backend distinguishes handing a submission to its controller, waiting for Ray resources, and executing capability code:
PENDING -> SCHEDULING -> RUNNING -> terminal state
SCHEDULING begins when the controller submits the worker request to Ray.
RUNNING begins only after the worker has actually started and completed a
startup handshake. job.scheduling_info reports requested resources, elapsed
scheduling time, worker start time, Ray node identity, and pod placement fields
when the deployment exposes them. Configure the Kubernetes downward API as
CHECKMAITE_KUBERNETES_NODE_NAME when the Kubernetes node name is required.
The job backend resolves CPU/GPU requirements in the following order:
- explicit
resources={...}passed at submission time, - hints on the config object (
config.num_cpus,config.num_gpus), - capability defaults (
default_num_cpus,default_num_gpus), - fallback (
num_cpus=1,num_gpus=0).
Example:
job = submit_capability(
capability,
datasets=[dataset],
resources={"num_cpus": 4, "num_gpus": 1, "memory": 8_000_000_000},
)
Registry and controller startup use separate readiness handshakes and
registry_startup_timeout_s / controller_startup_timeout_s budgets. Each
handshake uses one Checkmaite compatibility version plus typed immutable actor
configuration; Ray remains responsible for Python and Ray runtime compatibility.
If this client creates a controller that fails its readiness handshake, it
best-effort kills that controller before abandoning the registry reservation; a
pre-existing controller is never killed merely because this client times out.
Submission failures expose their phase through JobSubmissionError.phase, distinguishing
validation, registration, controller startup, attachment, and worker scheduling.
Capability and analytics/report dependencies are loaded lazily in the capability
worker rather than during controller process startup.
By default, scheduling_timeout_s=1800 fails a task that Ray cannot place within
30 minutes. This timer is owned by the detached controller and continues after
the submitting notebook exits. It is separate from job.wait(timeout=...),
which is only a client-side wait limit. Setting scheduling_timeout_s=None
explicitly allows an unbounded resource wait and emits a runtime warning that
pending work may prevent idle scale-down, leave cluster nodes running, and cause
significant additional compute charges.
Checkmaite does not duplicate Ray's worker-discovery or autoscaling policy.
Ray remains the source of truth for whether current or future cluster capacity
can satisfy the request; scheduling_timeout_s bounds that wait.
max_active_jobs_per_scope bounds accepted non-terminal work and
max_scheduling_jobs_per_scope bounds worker requests waiting in Ray's resource
queue. Capacity rejections raise BackpressureError; Ray resources continue to
determine actual execution concurrency.
Status mapping
RayJob is a thin protocol wrapper over shared registry lifecycle state.
Primary mapping comes from registry state:
SUBMITTING->PENDINGSCHEDULING->SCHEDULINGRUNNING/CANCELLING->RUNNINGCOMPLETED->COMPLETEDFAILED->FAILEDCANCELLED->CANCELLED
RayJob does not own or use the live task ObjectRef directly. The detached
controller owns that reference. Client handles resolve terminal payloads from
registry metadata (result_ref) and ask the controller to reconcile registry
state from its owned task when needed.
Timeouts and cancellation
from checkmaite.jobs import JobTimeoutError
try:
ref = job.result(timeout=120)
except JobTimeoutError:
job.cancel()
raise
Notes:
job.result(timeout=...)is a client-side wait timeout,- timing out does not automatically cancel the remote task,
job.cancel()records shared cancellation intent (CANCELLING) and asks the detached controller actor to callray.cancel(...)when the controller is reachable,- cancellation can be requested while
PENDING,SCHEDULING, orRUNNING, CANCELLINGstill maps to publicRUNNINGuntil the controller observes and commits a terminal state,- if the cancellation request is dispatched but the acknowledgement times out,
cancel()may returnTrueeven though final cancellation is still pending, - if the registry accepts cancellation while the controller is temporarily
unreachable,
cancel()returnsTrue; a recovered controller observes the shared intent through its heartbeat, while an expired controller lease eventually finalizes the record asCANCELLED, - if the controller has already been cleaned up then the job is already terminal and cancellation is a no-op.
Shutdown semantics
RayJobBackend.shutdown(wait=True) waits for jobs attached to that backend
instance and then calls ray.shutdown() for the local client process.
RayJobBackend.shutdown(wait=False) is intentionally a non-blocking local-handle
handoff: it does not wait for jobs and does not call ray.shutdown(). This is
used by configure_job_backend(...) when replacing the active backend so running
detached controllers can continue in the Ray cluster and a new backend can
reattach to them. Call ray.shutdown() explicitly, or use shutdown(wait=True),
when you really want to disconnect the local Ray runtime.
Registry configuration notes
The registry has two user-visible identity levels:
Ray cluster
└── registry_namespace selects one independent registry
└── idempotency_scope partitions jobs inside that registry
registry_namespace is a native Ray namespace. It is sent to Ray when
CheckMAITE creates or looks up its fixed internal checkmaite_job_registry
actor. There is no user-configurable registry actor name. Keep the namespace
stable across sessions to reconnect to the same registry, and select a different
namespace for migration, testing, load sharding, or independent retention and
resource settings.
A Ray namespace does not create a separate Ray cluster, Kubernetes namespace, worker pool, quota, or security boundary. Registries in different namespaces on the same cluster still share Ray's scheduler and workers. Use a separate Ray cluster when infrastructure or security isolation is required. In KubeRay, no extra Kubernetes object is needed for a Ray namespace: Ray's GCS records the namespace when the named actor is created.
The registry keys every lookup and deduplication entry by idempotency_scope, so
different scopes in one namespace do not list or deduplicate against each other.
Scopes are logical partitions, not security boundaries. Controller-count and
sweep-batch limits apply to the registry as a whole, not separately to each
scope, so activity in one scope can cause terminal controllers from another
scope to be cleaned up.
Registry creation settings are immutable for the lifetime of the registry in a
namespace. Each client performs a readiness and compatibility handshake before
reuse. If settings differ, either make clients agree or choose another
registry_namespace; CheckMAITE never silently creates another actor name in the
same namespace. A single CheckMAITE compatibility version covers the actor
protocol and record/result schemas; incompatible versions are rejected early.
Ray remains responsible for Python and Ray runtime compatibility.
registry_startup_timeout_s controls this cold-start handshake
separately from normal registry update timeouts.
The registry reserves zero Ray CPUs by default. It remains a long-lived detached control-plane actor and should stay small.
Pending-call limits and backpressure
Ray's pending-call limits belong to client-side actor handles, not to the actors
themselves. When a client creates an actor, registry_max_pending_calls=1024
bounds the registry handle returned by that creation call and
controller_max_pending_calls=64 similarly bounds a newly created controller's
initial handle. These limits can protect that client from unbounded queue growth
during bursty submits, polling, or cancellation.
A handle later obtained through ray.get_actor() is a distinct object. Ray does
not provide a public API for applying max_pending_calls to that handle, so
reattached registry and controller handles use Ray's unbounded default. Pending-
call limits therefore are not immutable actor settings and do not participate in
the registry compatibility handshake.
If Ray rejects a call on a bounded creation handle because its pending-call limit
is full, client APIs raise BackpressureError. Treat that as a retryable overload
signal: retry with exponential backoff and jitter, reduce client-side concurrency,
or tune registry_max_pending_calls / controller_max_pending_calls for the
expected creation-client burst size. Passing None leaves a creation handle
unbounded. Reattached clients should bound their own submission and polling
concurrency; CheckMAITE does not add a separate client-side queue.
Dedupe policy in the current implementation:
SUBMITTING,SCHEDULING,RUNNING, andCOMPLETEDrecords keep the dedupe key, so a duplicate submit in the same scope returns the existing job.FAILEDandCANCELLEDrecords release the dedupe key, so a later submit of the same logical work can create a fresh job.- expired
SUBMITTINGreservations are markedFAILEDand also release the dedupe key. - stale
RUNNINGrecords whose controller heartbeat lease expires are markedFAILED; staleCANCELLINGrecords becomeCANCELLED. Both release the dedupe key.
Terminal controllers schedule their own retention sweep after successfully
publishing terminal state. Consequently, controller_retention_s=0 cleans a
controller on a quiet cluster without requiring a later submission or an
operator-triggered sweep. Explicit sweep_registry() remains available for
maintenance and recovery.
Controller terminal updates are best-effort and bounded by
registry_update_timeout_s (default: 5 seconds). If one terminal update times
out, the controller keeps the terminal result in its own actor state, retries the
registry commit with bounded backoff, and clients can also commit that terminal
state on the next observe/reconcile path. While a live controller still has
uncommitted terminal state, its heartbeat keeps the registry lease fresh so the
job is not swept as stale.
Assumptions
Memory and payload assumptions
The registry actor is intentionally thin. Its Python dictionaries live in the registry actor worker's heap, not in Ray's object store. Registry records should stay small and JSON-like:
- job id, scope, status,
- timestamps,
- controller actor name,
- short error summary,
- small terminal
CapabilityRunRefmetadata.
The registry and controllers must not store large logs, model weights, datasets,
checkpoints, full CapabilityRunBase payloads, or unbounded history. Large data
belongs in the analytics store, object storage, shared filesystems, databases,
or platform logging systems. Terminal job records are retained only for the
configured registry retention window/count; once purged, their dedupe entries are
also removed and identical future submissions may create fresh jobs. Small,
self-contained text reports use CapabilityRunRef.report with
kind="inline_text"; inline UTF-8 content is limited to 256 KiB. A larger
inline report is published through artifact_store to a run- and
job-scoped, path-safe content-addressed key and returned as an artifact report.
Retries reuse a verified matching object. A mismatched object is replaced through
a supported atomic local rename or object-store commit; if safe replacement is
unavailable or fails, publication fails without deleting the existing key. Supported
remote implementations are S3, GCS, and Azure through s3fs, gcsfs, and adlfs.
The configured artifact_store.uri must be a concrete directory or object-store
prefix, not a wildcard or glob. Publication uses a temporary key before the
final key becomes visible, so retries cannot truncate an existing artifact. A
publication or verification
failure fails the job, allowing the logical run to be submitted again after the
storage problem is corrected. Optional artifact_store.storage_options are
forwarded to the selected supported filesystem implementation.
Report dependencies are a producer concern, not a backend discovery mechanism.
Inline reports must be self-contained: embed generated resources with forms such
as data: URIs, or link only to locations that report consumers can already
reach. Large, binary, or multi-file report producers must publish those products
themselves and return an ArtifactReport with a durable URI. The Ray worker does
not parse Markdown, copy referenced files, verify producer-owned remote URIs, or
rewrite links. A worker-local path, relative file link, or file:// URI will not
be usable after execution and must not be returned.
Artifact storage is deliberately independent from the analytics store: deployments may colocate both under one bucket, but they can use different backends, credentials, and retention policies.
During controller.start(...), the controller temporarily receives the
capability and run arguments so it can launch the worker task. It does not retain
those objects on actor state after submission. If future workloads involve very
large submission payloads, a more memory-conscious design can pass a Ray object
store reference instead:
client -> ray.put(submission_payload)
client -> controller.start(payload_ref)
controller -> worker task consumes payload_ref
That optimization trades controller heap pressure for Ray object-store pressure and should be introduced only with explicit object-lifetime management.
Cluster-lifetime and durability assumptions
Detached registry/controller actors survive notebook or client-process exits, but they do not survive Ray cluster loss. Current behavior assumes:
- if a client process exits but the Ray cluster remains alive, running jobs can continue and later clients can reattach using the same Ray cluster, registry namespace, and idempotency scope;
- if the Ray cluster dies, in-memory registry state, controller actors, and running tasks are lost;
- durable run data that was successfully written before cluster loss remains in the configured analytics store;
- jobs that were
RUNNINGwhen the cluster died require a future persistent job-metadata store and recovery policy to mark failed, reconcile from the analytics store, or resubmit.
The current in-memory registry is not a long-term job-history database. A production deployment that needs history across cluster restarts should add a persistent registry metadata store (for example Postgres, MongoDB, or object storage) with one small row/document per job. The current record shape is kept small and serializable to leave room for that future durable store.
Scaling assumptions
This design targets relatively heavyweight ML evaluation jobs where each job runs long enough that one controller actor per job is acceptable. It is not a high-throughput system for huge numbers of tiny tasks.
Operational considerations:
- each detached controller actor has Ray control-plane metadata and Python heap overhead;
controller_num_cpusdefaults to a small non-zero value to give the Ray scheduler backpressure;controller_memorycan reserve Ray actor heap memory when supported by the deployed Ray version;- platform deployments can use
controller_resourcesto place controllers on control-plane nodes via custom Ray resources; - terminal controller retention must be bounded with
controller_retention_sandmax_retained_terminal_controllers.
If a deployment needs thousands of concurrent small jobs or very high submit rates, a fixed controller-pool design may be better than one detached controller actor per job. In that model, a bounded set of actors owns many job refs and the registry maps each job to a pool actor. That design reduces actor churn but is more complex and has a larger failure domain per pool actor.
Common problems
- The notebook crashed before the job ID was saved. The job keeps running
in its detached controller. Reconnect to the same Ray namespace with the
same
idempotency_scopeand calllist_jobs(), as described in Recover after a notebook failure. (#735) - A tutorial's local settings were copied to a shared cluster. Settings
such as
address="local"andforce_reinit=Trueare for a single-user machine. On a shared cluster, connect to the cluster address, keep a stableidempotency_scope, and avoidforce_reinit=Truewhile work is active. (#737)
Related tutorials
- Ray Job Submission: submit, wait, and clean up a tracked Ray job end to end.