@innerworks-me/flow-extractor-node-dev v1.0.1
Introduction
This tool is designed to efficiently capture network flows and compute a range of statistics and features for each flow. These features include packet count, byte count, inter-arrival time (IAT), protocol type, TCP flags, and packet length.
Requirements
All of these prerequisites are usually installed togehter with nodeJS as the suggested required tools, bu we advice going over the following steps to make sure everything needed is present:
On Linux:
- libpcap-dev: Library crucial for network traffic capture
sudo apt-get install libpcap-dev
- Network Permissions: ensure these privileges are enabled in your program execution environment
NET_ADMIN
: For manipulating network interfaces.NET_RAW
: For raw socket access, essential for flow capture.
However, this command only sets these permission once while you actually need to programmatically incorporate thesudo setcap 'cap_net_admin,cap_net_raw=eip' $(which node)
cap_net_admin
andcap_net_raw
capabilities into your deployment (this will vary depending on your environment and should be always fairly easy to set up; it is not explained here due to the impossibility to cover the countless number of different setup possible). As an example, below is the configuration for a contenerized environment setup:
or with docket-compose with:docker run --cap-add=NET_ADMIN --cap-add=NET_RAW -d your_image_name
version: '3.8' services: your_service: image: your_image_name ***cap_add:** **- NET_ADMIN** **- NET_RAW***
On Windows:
In a terminal with Administrative Privileges, install:
Python ^3.x
: Releases for WindowsVisual Studio Build Tools
This is used to compile binary specific dependencies of pcap
Adjust version of Tools (debending of your OS, you might use x32 instead of x64, etc …) and WindowsSDK(switch to windows11SDK instead of 10 if needed) accordingly depending on your setup. if you are running this on a Windows server you can use the —quiet or —passive flags to automatically do any required interaction without needing a GUI. Remember the installation might take a while and usually requires a restart of the machine after for the changes to be applied. These tools are used to compile the node pcap binding for windows.curl -o vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe .\vs_buildtools.exe --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.20348
Npcap
On the npcap GUI installer select "Install Npcap in WinPcap API-compatible Mode" option.curl -o npcap-1.78.exe https://[npcap.com/dist/npcap-1.78.exe](https://npcap.com/dist/npcap-1.78.exe) npcap-1.78.exe
- Network Permissions: ensure these privileges are enabled in your program execution environment
NET_ADMIN
: For manipulating network interfaces.NET_RAW
: For raw socket access, essential for flow capture. Make sure to run your application with a user that has the right Network Administrative permissions and ensure your firewall settings allow for packet capturing.
Installation
Instructions for installing the application and its dependencies.
npm i --save @innerworks/flow-extractor-node
Usage
Integrating the Network Flow Extractor in Node.js:
- Import the Module:
At the beginning of your service file, import the Network Flow Extractor:
```jsx import NetworkFlowExtractor from "@innerworks/flow-extractor-node"; ```
Initialize Once:
To minimize performance overhead, instantiate the Network Flow Extractor only once where needed. Ideally, do this as a class field within a service constructor or similar scope where it can be easily accessed later:
const iwFlowExtractor = new NetworkFlowExtractor();
Usage During User Login: When processing a user login request, retrieve the specific network flow features for that user using the flow extractor instance initialised earlier passin the srcIp and srcport for the current request:
```jsx const flowFeatures = await this.flowExtractor.getFlowFeaturesWhenReady(req.socket.remoteAddress, req.socket.remotePort); ``` As outlined above, ensure the request object, or just the values **`req.socket.remoteAddress`** and **`req.socket.remotePort`** , is passed from its initial access point (usually the endpoint controller) through to the flow extractor function call. When triggered, **`getFlowFeaturesWhenReady()`** immediately retrieves a flow from the cache for that id if available; otherwise, it awaits a 'flowDataUpdated' event signaling a new flow's arrival. This asynchronous function times out after a default of 1.5 seconds if no flow is found, timeout is adjustable via the function third optional parameter.
Optional Parameters Tuning
- Constructor
new NetworkFlowExtractor( networkInterface?: string, flowLengthThreshold: number = 3, flowsCacheCleanupInterval: number = 60000; )
networkInterface?: string
: The flow extractor auto-detects by default the network interface used by your backend. If this does not work as expected, you can manually specify the interface to listen to in this parameter.flowLengthThreshold: number = 3
: Defines the minimum packet count for a valid flow. It's advised to keep this default value, as a valid flow usually comprises at least 3 packets. Increase this threshold only if you're certain that your endpointd requests will consistently involve more packets, which can enhance flow feature accuracy.flowsCacheCleanupInterval: number = 60000
: Determines the interval (in milliseconds) for purging flows older than 1 minute from memory.
Testing Network Flow Extractor Locally
To test the flow extractor locally using your laptops's network interface follow below steps
generate openssl sertificates server.cert and server.key:
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -x509 -key server.key -out server.cert -days 365
make your own copy of index.js from scripts/index.js.example
- adjust network interface
- adjust correct flow id
- run
sudo node index.js
Offline Pcap file processing
- this is needed for debugging & feature engineering purposes
- in the moment, we need this to analyze feature importance of our extractor on https://www.unb.ca/cic/datasets/iotdataset-2023.html dataset
run the example Pcap=>CSV feature extraction
sudo node scripts/pcap2csv.js
output: flow_features.csv
trouble shooting
there could be issue with pcap package on Mac M1
nvm install 20.5.0
nvm use 20.5.0
npm rebuild pcap --update-binary