@tdole/pcap-tcp_reassemble v8.10.4

With this program you can reorder TCP sessions in a pcap file.

Sometimes TShark or Wireshark cannot reorden TCP stream in the right way and this will result in corrupted TCP stream content. Processes which work with the TCP content, like TLS, will then also fail.

As an input source you can now also read data from a RabbitMQ. The pcap data must have been uploaded with the "@todle/pcaptorabbit" program.

Requirements:

• NodeJS (v14.6.1 or higher).
• Capture file must be in pcap or pcap-ng (data in 'Enhanced Packet Block' blocks) format.

Installation:

This programm also exists on the online NPM repository. Install from there with 'npm install @todle/pcap-tcp_reassemble -g'. This will install de program 'pcap-tcp_reassemble' globally.

To install the checkedout git repository version use the command 'npm install -g' will install the program 'pcap-tcp_reassemble' globally.

Usage

Usage: node ./pcap-tcp_reassemble.js [options]
Show this help message
    --help
When connection to AMQP server is lost. Retry this many times before stopping program. (Default: 100)
    --amqpFailAfterReconnects <value>
When connection to AMQP server is lost. Wait this many milliseconds before reconnect. (Default: 5000)
    --amqpReconnectInterval <value>
When proces is asked to stop save a list of open session files to pick up on next start.
    --saveBeforeStop
Before starting check if we have a list generated by to continueAfterStop. Read the content of these files to continue.
    -c, --continueAfterStop
Allow only one TCP session per source ip. Close existing when new session is seen.
    --onlyOneSessionPerSource
IP address of server.
    --serverIp <value>
Make one pcap file per tcp session.
    --savePerTCPSession
Next to TCP session pcap file generate a forensic log file including dropped packts and original source files.
    --generateForensicLog
Drop/do not create pcap files when payload of tcp session is zero byte.
    --onlyStoreCleanSessions
Drop/do not create pcap files when payload of tcp session is zero byte.
    --dropZeroPayloadSession
Splity boundary. String with hexadecimal values. For example 0A0D for CrLf.
    --splitboundary <value>
Splity size. Depends on splittype if it is number of packet or seconds. Default: 100
    --splitsize <value>
Type of splitting (1=number of packet, 2=time interval in seconds, 3=time interval and both incoming and outgoing data stream need to end on boundary). Default: 1
    --splittype <value>
Split tcp session data over multiple files.
    --split
Only ouput packets to/from specified port(s) (comma separated list of port numbers)
    --portfilter <value>
Folder used for final destination of [tcp session] pcap files
    --destfolder <value>
RabbitMq queue
    -q, --amqpQueue <value>
AMQP Routing key
    -l, --amqpRoutingKey <value>
RabbitMQ exchange
    -e, --amqpExchange <value>
URI to amqp (RabbitMQ) host
    --amqpHost <value>
TCP session timeout to use in seconds. (Default: 86400
    --tcpSessionTimeout <value>
Save all packets also non tcp.
    -a, --all
Show version
    -v, --version
Write reassembled and cleand data to pcap file
    -o, --output <value>
read from pcap file
    -f, --file <value>
Debug generate a lot of output to stdout and stderr
    -d, --debug

The program can read from stdin and write to stdout when using either "-f" or "-o" command line arguments.

Example local files:

node pcap-tcp_reassemble.js -f test.pcap -o test_cleaned.pcap -a

cat test.pcap | node pcap-tcp_reassemble.js -f - -o - -a > test_cleaned.pcap

Example splitting tcp sessions into individual pcap files. One pcap per tcp session:

node pcap-tcp_reassemble.js -f test.pcap --destfolder /data/out/

This will generate per tcp session a pcap file with the following path: /data/out/<ip1>/<ip2>/<port1>-<port2>/YYYYMMDDhhmmss.SSSSSSs|c_<frame#>_<version>.pcap. The reported session states can be found in Flowchart of TCP connections and their definition

• ip1 and ip2: ip1 will be the string compared lowest of source and destination ip of the session.
  • So source == 10.1.1.1 and destination == 205.1.1.1 ip1 wil become 10.1.1.1 and ip2 will be 205.1.1.1
  • but when source == 205.1.1.1 and destination == 10.1.1.1 will also result in ip1 will be 10.1.1.1 and ip2 will be 205.1.1.1
• port1 and port2 will be set respectively to the ports used by ip1 and ip2.
• YYYYMMDD_hhmmss.SSSSSS will be set to the timestamp of the first pcap packet in the tcp session. This will be in your local timezone.
• s|c: Will tell you if this is a clean tcp session which started with a SYN packet or a SYN/ACK packet.
• frame: will be set to the last frame received from the source when this file was created. This to make the filename unique.
• version: will be the version of the pcap-tcpreassemble software. It is the version specified in the package.json file and the dots (.) have been replaced by underscores (). So 5.2.1 becomes 5_2_1

If you add the command line argument "--generateForensicLog" next to each tcp session pcap file a ".*.log.json" will be create which contains forensic information about the tcpsession pcap. In it the following information is recorderd:

• sourceFiles: Array of filename(s) which are the source of this tcp session pcap. When pcap-tcp_reassemble is used with the "-f" argument it is the name of this source file. When reading from a RabbitMQ it is the name of the source files read by pcaptorabbit program.
• connection:
  • hasInitialSyn: Booolean where a SYN or SYN/ACK packet was seen to start the session,
  • starttime: Start time of this session in unix timestamp including milliseconds. For Example: 1662842458.226251.
  • endtime: End time of session in unix timestamp including milliseconds. For example: 1662842921.333337.
  • duration: Duration of session in seconds.
  • cleanSession: Session started with a clean SYN or SYN/ACK packet.
  • established: Both ends of TCP session went through the ESTABLISHED state.
  • source: You can see this as the client or initiator of the tcp session.
    • ip: ip number
    • port: tcp port number
    • transmittedBytes: Number of bytes send from this ip.
    • missingBytes: Total number of bytes missing in pcap. Based on sequence number one or more packets are missing. The missing packets/size are reported in the 'gap' property of this connection.
    • finalState: This is the final state of this side of the connection
    • gap: This is a key/value pair list of packets/segments missing in the source data. Each key is the a string with "<sequence from> -> <sequence to>" of the missing packet and the value is the number of bytes missing in the packet.
    • dropped-duplicate-data: This is a key/value pair list of packet for which the content was already received. So this packet was retransmitted. The key is the frame number of the packet in the source and the value is the sequence number of the packet.
    • truncated: Sometimes data in a tcp session can be retransmitted. And sometimes the payload of a retransmitted packet fills a complete gap but overlaps the payload of other packet(s). So the software truncates the end and/or start of the payload to fit the gap and this truncated packet is written in place of the gap. This property contains a key/value pair list of truncated packets. The key is the frame number of the source packet truncated and the value is an object with the following properties:
      • old: Information of original packet. this is an object with the following properties:
        • sequence: Sequence number of original packet.
        • length: Length of original packet.
      • new: Information of new truncated packet. this is an object with the following properties:
        • sequence: New sequence number of packet. When only end it truncated then this is the same as the old sequence number.
        • length: Length of new packet.
  • destination: You can see this as the server/listening side of the tcp session. This object has the same properties as the source property.

{
	"sourceFiles": [
		"/data/in/test.pcap"
	],
	"connection": {
		"source": {
			"ip": "10.0.0.1",
			"port": 1234,
			"gap": {
				"3065721720 -> 3065722450": 730
			},
			"dropped-duplicate-data": {
				"65001295": 3065936480,
				"65001298": 3065936480,
			},
            "truncated": {
                "old": {
                    "sequence": 30123456,
                    "length": 512
                },
                "new": {
                    "sequence": 30123456,
                    "length": 200
                }
            }
			"transmittedBytes": 2078750,
			"missingBytes": 2190,
			"finalState": "TRANSITION_TO_TIME_WAIT"
		},
		"destination": {
			"ip": "192.168.1.1",
			"port": 80,
			"gap": {
				"1734786526 -> 1734792672": 6146,
				"1754514400 -> 1754525504": 11104
			},
			"dropped-duplicate-data": {
				"65007570": 1740352772,
			},
			"transmittedBytes": 31833310,
			"missingBytes": 17250,
			"finalState": "CLOSE_WAIT"
		},
		"hasInitialSyn": true,
		"starttime": 1662842458.226251,
		"cleanSession": true,
		"endtime": 1662842921.333337,
		"duration": 463.107086,
		"established": true
	}
}