Processing massive PCAP files: from slow scripts to high-performance network analysis with Tshark, Editcap, and Zeek

Anyone investigating network traffic or analyzing packet captures knows the frustration of opening a multi-gigabyte .pcap file only to watch their system grind to a halt.

A common initial reaction is writing a quick Python script to parse packets and dump summaries to CSV. However, as file size scales, memory usage skyrockets, CPU load peaks, and execution crawls indefinitely.

Below is an overview of why pure script approaches bottleneck and how to use native tooling to process captures cleanly and rapidly.

1. The pure Python parsing bottleneck (scapy)#

Python libraries like Scapy are fantastic for crafting custom packets and rapid prototyping. However, they are not designed for high-throughput stream processing on large files.

Processing large packet dumps in pure Python causes severe issues:

• Memory consumption explodes as the runtime retains packet objects in RAM.
• Layer-by-layer interpretation burdens the CPU, turning a simple extraction into a multi-hour bottleneck.

When speed matters in terminal workflows, compiled C utilities are dramatically superior.

2. The golden rule: Tshark for direct CSV exports#

The fastest and most direct CLI utility to convert raw .pcap files into structured tabular data is Tshark, Wireshark's command-line interface.

Written in C, Tshark operates with minimal memory footprint compared to interpreted runtimes. The optimal command structure for clean CSV exports:

tshark -r traffic.pcap -T fields \
  -e frame.number \
  -e ip.src \
  -e ip.dst \
  -e _ws.col.Protocol \
  -e frame.len \
  -E header=y -E separator=, -E quote=d > output.csv

Practical optimization tip: Request only the exact fields (-e) required for your investigation. The fewer dissectors Tshark executes per frame, the faster the output generates.

3. Handling multi-gigabyte captures: split and parallelize#

When working with massive packet captures, single-threaded execution on a monolithic file can still saturate disk I/O and pin a single CPU core.

The cleanest strategy is chunking the file before processing using Editcap (packaged natively with Wireshark/Tshark):

# Split the input capture into 100,000-packet chunks
editcap -c 100000 input.pcap chunk.pcap

Once chunked, convert files in parallel across available CPU cores using xargs. Because Tshark operates in a streaming fashion without loading entire captures into RAM, chunking primarily serves to distribute decoding across multiple CPU cores:

# 1. Convert each chunk into its own CSV in parallel (4 concurrent workers)
ls chunk_*.pcap | xargs -P 4 -I {} sh -c 'tshark -r "{}" -T fields -E header=n -E separator=, -e ip.src -e ip.dst > "{}.csv"'

# 2. Merge all chunks into a unified CSV with header
echo "ip.src,ip.dst" > merged_data.csv
cat chunk_*.pcap.csv >> merged_data.csv
rm -f chunk_*.pcap.csv

4. High-performance event engines: Zeek and arkime#

If your workflow involves continuous security monitoring or recurrent querying rather than ad-hoc CSV exports, flat files become obsolete:

• Zeek (formerly Bro): Rather than raw packet dumps, Zeek extracts structured network events into lightweight, categorized logs (conn.log, http.log, dns.log). These files are lightning-fast to filter with grep, awk, or ingest into a SIEM.
• Arkime (formerly Moloch): Designed for full-packet capture indexing at enterprise scale. Arkime indexes packet metadata into OpenSearch (formerly Elasticsearch), enabling sub-second web searches and session reconstructions over terabytes of PCAP data.

Practical workflow summary#

• Medium captures (up to hundreds of MB): Run tshark with explicit field flags for instant CSV parsing.
• Massive captures (multiple GB): Slice with editcap -c and process parallel chunks across multiple CPU cores.
• Continuous traffic analysis: Transition from raw CSV parsing to event-driven architectures with Zeek or indexed search clusters with Arkime.

Conclusion#

In this post, I shared the layered workflow I use for handling large PCAPs: Tshark with strictly minimal fields for fast CSV exports, editcap chunking with parallel decoding for massive files, and Zeek or Arkime when packet inspection becomes a regular operational routine — always prioritizing native C engines over interpreted scripts.

See also

Cybersecurity Troubleshooting Fail2ban status 255 exception: migrating to systemd backend and Nginx modular includes

Cybersecurity Incident anatomy: mitigating simultaneous L7 attacks, Dovecot DoS, and CPU bottlenecks

Cybersecurity Forensic analysis of a CloudLinux and cPanel compromise: from the sshd_config trap to PAM and IPMI persistence

This post is licensed under CC BY-NC.