Kubernetes and KubeRay deployment notes
This page collects Kubernetes-specific guidance for running CheckMAITE job
submission on KubeRay. General worker image and Ray runtime_env guidance lives
in Worker environments.
The exact cluster YAML belongs to the platform repository, not to CheckMAITE.
The important point is that the platform image and RayCluster definition live
outside CheckMAITE, while the job backend connects to that cluster and
submits work into it.
KubeRay-style deployment model
A typical RayCluster separates the head pod from one or more worker groups:
apiVersion: ray.io/v1
kind: RayCluster
spec:
headGroupSpec:
template:
spec:
containers:
- name: ray-head
image: registry.example.com/checkmaite-ray:2026-04-14
workerGroupSpecs:
- groupName: cpu-workers
replicas: 2
template:
spec:
containers:
- name: ray-worker
image: registry.example.com/checkmaite-ray:2026-04-14
resources:
limits:
cpu: "8"
memory: "32Gi"
- groupName: gpu-workers
replicas: 1
template:
spec:
containers:
- name: ray-worker
image: registry.example.com/checkmaite-ray-gpu:2026-04-14
resources:
limits:
nvidia.com/gpu: "1"
cpu: "8"
memory: "64Gi"
Use deployment-specific images, resources, node selectors, tolerations, secrets, and autoscaling settings in your platform configuration.
Detached actors and scale-down
The default ray backend uses a detached registry actor and detached per-job
controller actors. Detached actors can remain alive after the submitting notebook
or driver exits, so they affect scheduling, autoscaling, pod placement, and
cleanup.
A pod that hosts a retained terminal controller actor generally cannot scale down until that actor is killed or forgotten. The registry actor is intentionally long-lived; if it lands on an autoscaled worker pod, that pod may stay alive for as long as the registry exists.
For Kubernetes deployments, consider aggressive terminal-controller cleanup when
reattach-through-controller is not needed after terminal state is committed, for
example controller_retention_s=0.0 and
max_retained_terminal_controllers=0. Terminal controllers now request their
own registry cleanup after retention expires, so quiet-cluster scale-down does
not depend on a later submission. Keep an operator sweep as a recovery mechanism
for actors that fail before scheduling that request.
Head node placement
Ray head pods have extra memory and control-plane pressure from GCS, dashboard,
and cluster services. Unless intentionally using the head for lightweight
control-plane actors, configure the Ray head with num-cpus: "0" so nonzero-CPU
user tasks and actors do not land there.
Controller actors should normally reserve a small nonzero CPU amount, such as
the default controller_num_cpus=0.01, or use a custom placement resource. A
fractional reservation can fragment a one-CPU worker and leave only 0.99 CPU
for a capability requesting one full CPU. Avoid controller_num_cpus=0.0 in
production Kubernetes unless placement is otherwise controlled.
A clean production layout is often a small dedicated control-plane worker group
advertising a custom resource such as checkmaite-control, while normal worker
groups run capability tasks. Configure controller_resources so controllers
consume that control resource and cannot occupy capability-only workers.
Registry actor placement and resources
The registry actor should have an explicit small resource reservation or custom
resource placement in production KubeRay deployments. Use registry_num_cpus,
registry_memory, and registry_resources to make placement explicit.
For example, a custom resource such as {"checkmaite-control-plane": 1} can
force the registry onto a dedicated control-plane worker group. Avoid allowing
the registry to land on arbitrary autoscaled worker groups if that would prevent
scale-down.
The registry remains a single serialized coordination point, so keep records
small and list operations bounded. registry_max_pending_calls defaults to
1024 for the registry handle returned when this client creates the actor, and
controller_max_pending_calls similarly defaults to 64 for newly created
controller handles. If those bounded handles fill, client calls raise
BackpressureError; retry with exponential backoff and jitter or tune the limits
for the expected burst size. Ray cannot apply these handle-local limits when a
client reattaches through ray.get_actor(), so reattached registry and controller
handles use Ray's unbounded default and clients should bound their own request
concurrency. Passing None also leaves creation handles unbounded.
Durability boundary and workload fit
Detached actors survive notebook or driver termination, but they do not survive RayCluster deletion, full cluster recreation, or loss of actor memory. The in-memory registry is sufficient for reattach across client restarts while the RayCluster is alive. It is not durable job history across RayCluster replacement.
A production Kubernetes architecture that needs durable history should treat detached actors as the live control plane, an external database or object store as durable truth, and the Ray object store as transient data plane.
The per-job detached-controller design is intended for long-running capability jobs. Workloads with thousands of concurrent jobs or very short tasks may need a future design with a supervisor actor, controller pool, sharded registries, batched task tracking, or DB-backed coordination.