Ray Core Object Store Spill Tuning: Fixing OOM Workers

Ray Core object store spill tuning prevents sudden node evictions when worker memory spikes during bulk vector chunking operations. When distributed workers process massive payloads without active object store management, Plasma shared memory fills up, forcing the Linux Out-Of-Memory killer to abruptly terminate raylet processes and corrupt job progress. Production workloads executing chunk tokenization and tensor transformations often stall indefinitely due to unmonitored object lifecycle references and unoptimized spill disk targets.

To restore stability across large multi-node clusters processing enterprise documents, architecture teams must transition from default in-memory assumptions to explicit spilling strategies. Implementing structured object eviction policies, isolating temporary NVMe scratch directories, and releasing Python reference handles immediately after computation prevents worker death while maintaining maximum vector processing throughput.

Why Ray Core Spills Plasma Shared Memory to Disk During Large Payload Swarms

Ray relies on Plasma, an in-memory shared object store deployed across every cluster node. When a task generates outputs or puts objects into memory via ray.put(), those artifacts reside inside the node local Plasma store. Because Plasma operates as a fixed-size shared memory segment—typically defaulting to 30% of system RAM—bursty downstream tasks can consume available space much faster than garbage collection routines can reclaim it.

When memory demand exceeds the allocated Plasma capacity, Ray executes its spilling protocol. If no external spill location is configured, or if the local disk I/O hits a bandwidth bottleneck, worker processes experience severe backpressure. In heavy streaming data architectures, such as those powering our RAG Knowledge Base Pipeline, unmanaged Plasma saturation leads to catastrophic worker evictions and cascaded re-executions across the cluster.

Unlike traditional distributed SQL engines discussed in our post on DuckDB Out-of-Core Memory Spills, Ray manages arbitrary Python object graphs, intermediate NumPy arrays, and Arrow tables. These complex object graphs create pinning risks. If a single active Python task holds a reference handle to an ObjectRef, Plasma marks that memory block as unevictable. When unevictable pinned objects accumulate alongside incoming stream chunks, Plasma has no choice but to push remaining unpinned objects to secondary storage or crash when disk spill capacity fails.

Diagnosing Object Store Spills with Prometheus Metrics and Ray Dashboard

Identifying whether pipeline slowdowns stem from Plasma memory pressure or network transport requires tracking precise operational metrics. Ray exposes telemetry through OpenTelemetry and Prometheus endpoints, providing direct visibility into object lifecycle transitions and disk write latencies.

The critical metrics to observe during heavy ingestion swarms include:

1. ray_object_store_memory: Measures current active bytes stored in Plasma across local node limits.
2. ray_object_store_spilled_bytes_total: A monotonically increasing counter tracking total bytes offloaded to secondary storage.
3. ray_memory_usage_bytes: Real-time RSS memory tracked across Python worker processes.
4. ray_component_cpu_percentage: Identifies if I/O serialization threads spent executing object spilling block task scheduling.

When ray_object_store_spilled_bytes_total spikes rapidly while task throughput drops, the cluster is experiencing spill-induced disk I/O thrashing. This condition occurs when workers attempt to read spilled objects back from slow EBS volumes or remote cloud storage faster than the storage layer can deliver them. Visualizing these metrics alongside node-level disk IOPS highlights whether your instance store configuration can support the current concurrency model.

Configuring Hybrid Storage Policy for Local NVMe and Cloud Object Store Spills

Resolving Plasma saturation requires configuring explicit object spilling directories within the Ray system settings. By default, Ray attempts to spill objects to /tmp/ray/spill, which usually resides on the root file system. Root EBS volumes on cloud provider instances rarely possess the IOPS or write throughput needed to sustain concurrent multi-gigabyte object dumps.

High-performance streaming environments should configure a multi-tiered spilling strategy: primary spills land on local ephemeral NVMe SSD drives attached to the instance, with secondary fallback pointing to Amazon S3 or Google Cloud Storage for deep spill spillover. The following configuration demonstrates how to programmatically initialize Ray with custom JSON system configs specifying hybrid disk and object storage spill targets.