Support result buffers backed by caller-provided shared memory for zero-copy cross-process consumption #26909

Is your feature request related to a problem or challenge?

DataFusion is frequently embedded as an in-process engine that serves data to other processes: the engine runs in a dedicated "database" process, while consumers read results over the Arrow C Data Interface (directly, or through an ADBC driver or a custom FFI layer).

That path cannot be zero-copy today, and the failure mode is hard to diagnose.

The C Data Interface is pointer-based. ArrowSchema/ArrowArray handed to a consumer contain absolute addresses (format, children, buffers, …) and a release callback. DataFusion produces those buffers on its own heap, so:

• a consumer in another process cannot dereference buffers[i], children[i], format — the addresses are not mapped there, which results in an access violation or, worse, in silently dangling pointers that only fail when the first value is read;
• the release callback is a function pointer into the producer's code, so the consumer must not call it either.

As a result, the only correct way to move a result set across a process boundary is to copy every buffer, or to serialize the data (Arrow IPC). Both cost a full pass over the data, which is exactly what users of an embedded engine are trying to avoid.

DataFusion currently offers no way to influence where result buffers live:

• MemoryPool performs accounting and spilling only; it does not select an address space.
• RuntimeEnv/SessionConfig have no allocator hook.

The allocation itself happens in arrow-rs MutableBuffer, which goes through the process-global allocator and cannot be redirected afterwards.
So "run DataFusion in one process and read its results in another without copying" is not expressible with the current API, even though DataFusion's own execution model (partitioned, streaming, RecordBatch-based) would fit such a deployment well.

Describe the solution you'd like

An opt-in way to have a query's output buffers allocated in a caller-provided memory region, plus a descriptor that lets a consumer reconstruct the batches in its own address space.

Sketch of the API surface:

// Region is created by the application: named shared memory / memory-mapped file.
let region = SharedRegion::create("df_results", 1 << 30)?;

let config = SessionConfig::new()
    .with_buffer_allocator(region.allocator());   // new

let df = ctx.sql("SELECT ... FROM parquet_scan(...)").await?;
let mut stream = df.execute_stream().await?;

while let Some(batch) = stream.next().await {
    // Batch buffers live inside `region`; publish() returns only offsets/lengths,
    // so the consumer never needs producer addresses.
    let descriptor = region.publish(batch?)?;
    send_descriptor_to_consumer(descriptor)?;
}

Two properties matter:

Allocator threading. A BufferAllocator handle carried by TaskContext/RuntimeEnv and used at buffer creation sites, defaulting to the current global allocator so behavior is unchanged unless opted in. Because operators run on multiple threads and also spawn work (repartition, joins, parquet prefetch), a thread-local set at the call site is not sufficient — the allocator has to travel with the task.
Descriptor-based export. The consumer should receive (region handle, buffer offsets, lengths, offsets/validity layout) rather than absolute pointers, then build its own ArrowArray with buffers[i] = local_base + offset. This avoids the requirement that the region be mapped at the same base address in both processes, which is fragile in practice (address-space collisions, ASLR, needing VirtualAlloc(MEM_RESERVE) / mmap(MAP_FIXED_NOREPLACE)).
With that, the data is written once into shared memory, the consumer maps the region and reads zero-copy, and release stays a local no-op on the consumer side.

Describe alternatives you've considered

• Deep copy on the consumer side. Walk the schema/array graph and copy every buffer into the consumer's memory (e.g. ReadProcessMemory on Windows, process_vm_readv on Linux). Works, but it is a full memcpy of the result set — the same cost as IPC, without the portability.
• Arrow IPC. Standard and supported, but serializes the data and adds framing/deserialization overhead.
• A custom driver/FFI layer that copies each batch once into a memory-mapped region. This is workable today and is essentially the feature requested here, just implemented outside DataFusion — it keeps DataFusion untouched but duplicates the logic in every embedder.
• Replacing the global allocator in the host process. A custom #[global_allocator] that routes allocations with Arrow's buffer alignment (ALIGNMENT = 128 on x86_64) into a shared arena. It affects the whole process (tokio, parsers, catalog), misses buffers created from Vec/bytes::Bytes, and depends on an implementation detail of arrow-rs rather than on a contract.
• Patching arrow-rs. Adding a per-buffer allocator to arrow-buffer and threading it through the execution context. Correct in principle, but it means maintaining a fork of arrow-rs and DataFusion and pinning versions.
• ADBC partitions / ConnectionReadPartition. Designed to be passable across processes, but the partition payload is serialized, so it is again a copy.

Additional context

• The allocation site is arrow-buffer's MutableBuffer → std::alloc with Layout { align: 128 }. A hook in DataFusion can only be effective if it reaches those allocation sites; that is why the "allocator travels with the task" part is the essential piece, not just a config flag.
• A partial solution is still useful: even if some buffers (e.g. those built from Vec by the parquet reader) end up outside the region, exposing the mechanism plus a per-batch "is this batch fully region-backed?" flag would let callers fall back to a copy only when necessary.
• Precedent for the descriptor approach: the Arrow C Device Data Interface separates the device representation from the arrays themselves (ArrowDeviceArray + ArrowMemoryManager). A host shared-memory device type there would be the natural long-term home for this, but DataFusion-side support for placing batches in a region is a prerequisite.
• This is an API/design request, not a crash report; the current behavior is expected given a private heap, but it is undocumented and easy to hit when embedding DataFusion as a separate service process.

Willingness to contribute

None