Is Arrow Streaming end-to-end copy-free - apache-arrow

Question

I am confused about Arrow Streaming. Many sources describing Arrow just paraphrase the following
The Arrow memory format supports zero-copy reads
and say that Arrow is zero-copy tool.
However, as far as I understand these paragraphs:
The primitive unit of serialized data in the columnar format is the “record batch”. Semantically, a record batch is an ordered collection of arrays, known as its fields, each having the same length as one another but potentially different data types. A record batch’s field names and types collectively form the batch’s schema.
In this section we define a protocol for serializing record batches into a stream of binary payloads and reconstructing record batches from these payloads without need for memory copying.
the description of the IPC Streaming Format and, my limited understanding, the source, data is serialized, and only deserialization is zero-copy.
In other words - systems that use Arrow Streaming actually copy the data on the way.
Is that correct?

Answer 1

As you said, Arrow is always zero-copy on the receiver side.
systems that use Arrow Streaming actually copy the data on the way.
It depends what you mean by "copy". Is data duplicated in-memory in the same process? No. The bytes have to be transported somehow from one virtual address space to another, whether you believe that technically constitutes a "copy" or not depends on your application (and point of view, perhaps).
Here is the actual C++ code (as of this writing) where the data written by the sender into an OutputStream which is a proxy for the receiver
// Now write the buffers
for (size_t i = 0; i < payload.body_buffers.size(); ++i) {
const std::shared_ptr<Buffer>& buffer = payload.body_buffers[i];
int64_t size = 0;
int64_t padding = 0;
// The buffer might be null if we are handling zero row lengths.
if (buffer) {
size = buffer->size();
padding = BitUtil::RoundUpToMultipleOf8(size) - size;
}
if (size > 0) {
RETURN_NOT_OK(dst->Write(buffer));
}
if (padding > 0) {
RETURN_NOT_OK(dst->Write(kPaddingBytes, padding));
}
}
Nothing is being forcibly copied here. If dst is standing in front of a socket-like interface then the bytes go on the wire to the receiver immediately (or with buffering, or whatever the OutputStream is doing). If dst is a file handle then the bytes are written to the file, etc.