A developer describes debugging a tshark-based PCAP analysis CLI tool that ground to 6-7 hours and crashed with out-of-memory errors on a 2.5 GB capture file (1.9 million packets). The root causes were three separate full passes over the file, tshark's single-threaded nature, and full JSON dissection held entirely in memory by both tshark and the Go program. Consolidating the three queries into one pass cut runtime to 1-2 hours, and piping tshark's stdout directly into the Go program (streaming, with narrowed -T fields/-T ek output instead of full JSON) eliminated OOM crashes and dropped processing to about 70 minutes. Throughput remained single-threaded and unsolved, setting up a follow-up post on concurrency.
Table of contents
The tool that workedThen came 2.5 GBWhat was actually wrongCutting three passes down to oneStreaming the outputWhat it didn't fixQuestions this post answers
How do I avoid out-of-memory crashes when using tshark to analyze large PCAP files in Go?
Stream tshark's output instead of buffering it. Pipe tshark's stdout directly into a Go program's stdin using narrowed output flags like -T fields or -T ek instead of full JSON dissection, so packets are processed line by line as they arrive rather than held entirely in memory. On a 2.5 GB file with 1.9 million packets, this eliminated OOM crashes and cut processing from 6-7 hours to about 70 minutes. Anyone piping large PCAP captures through tshark can compare streaming approaches like this on daily.dev.
Why does running multiple tshark queries against the same PCAP file cause slow processing?
Each separate query forces a full pass over the entire file, so three queries mean three full traversals of the capture. Consolidating three queries answering analytics, rows, and full dissection needs into a single optimized query that returns everything in one pass reduced processing time on a 2.5 GB file from 6-7 hours down to 1-2 hours, before any streaming changes were made. Developers tuning packet-analysis pipelines can track techniques like query consolidation on daily.dev.