Phase 2 of OTel-Arrow explores using Apache Arrow not just as a wire transport format but as the internal representation for telemetry pipeline processing. The team built the OTel-Arrow Dataflow Engine, a Rust runtime using a thread-per-core, shared-nothing architecture with OTAP (OpenTelemetry Arrow Protocol) as its primary data path. Benchmarks show that keeping telemetry in Arrow columnar batches dramatically reduces CPU overhead for common operations like attribute renaming — the OTAP path achieves 10–20x higher throughput than the OTLP path on the same hardware by eliminating protobuf decode/encode cycles. The engine also scales near-linearly to 16 cores (14.6x speedup) and applies backpressure to keep memory bounded under overload. The project is currently incubation-stage and not recommended for production workloads, but the team is seeking community feedback on runtime semantics, processing APIs, and real-world pipeline designs.
