joincap

Merge multiple pcap files together, gracefully.

Installation

• Download a precompiled binary from https://github.com/assafmo/joincap/releases

• Or... Use go get:

  go get -u github.com/assafmo/joincap
  

• Or use Ubuntu PPA:

  curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add -
  sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list
  sudo apt update
  sudo apt install joincap
  

Basic Usage

Usage:
  joincap [OPTIONS] InFiles...

Application Options:
  -v, --verbose  Explain when skipping packets or entire input files
  -V, --version  Print the version and exit
  -w=            Sets the output filename. If the name is '-', stdout will be used (default: -)

Help Options:
  -h, --help     Show this help message

Why?

I believe skipping corrupt packets is better than failing the entire merge job.
When using tcpslice or mergecap sometimes pcapfix is needed to fix bad input pcap files.

1. One option is to try and run merge (mergecap/tcpslice), if we get errors then run pcapfix on the bad pcaps and then run merge again.
   • Adds complexity (run -> check errors -> fix -> rerun)
   • (If errors) Demands more resources (pcapfix processes)
   • (If errors) Extends the total run time
2. Another option is to run pcapfix on the input pcap files and then merge.
   • Extends the total run time by a lot (read and write each pcap twice instead of once)
   • Demands more storage (for the fixed pcaps)
   • Demands more resources (pcapfix processes)
3. We can use pcapfix "in memory" with process substitution: mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap).
   • Adds complexity (build a complex command line)
   • Demands more resources (pcapfix processes)
   • Harder for us to use pathname expansion (e.g. tcpslice -w out.pcap *.pcap)
   • We have to mind the command line character limit (in case of long pathnames)
   • Doesn't work for tcpslice (seeks the last packets to calculate time ranges - cannot do this with pipes)

Error handling: joincap vs mergecap vs tcpslice

Results

Error outputs

How to reproduce

Benchmarks

• Merging 3 files with total size of 2.99994GiB.
• Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.