Skip to main content

NetFlow

Synopsis​

Creates a NetFlow collector that accepts flow data over UDP connections. Supports high-volume collection with multiple workers and configurable buffer sizes.

warning

The collector only supports legacy NetFlow types, such as NetFlow v5. For NetFlow v9, use the ipfix collector type.

For details, see Appendix.

Schema​

- id: <numeric>
  name: <string>
  description: <string>
  type: netflow
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    protocol: <enum>
    framing: <enum>
    max_connection_age: <numeric|string>
    workers: <numeric>
    reuse: <boolean>

Configuration​

The following fields are used to define the device:

Device​

FieldRequiredDefaultDescription
idYUnique identifier
nameYDevice name
descriptionN-Optional description
typeYMust be netflow
tagsN-Optional tags
pipelinesN-Optional pre-processor pipelines
statusNtrueEnable/disable the device

Connection​

FieldRequiredDefaultDescription
addressN"0.0.0.0"Listen address
protocolNudpTransport: udp or tcp. See Transport below
framingNipfix over TCPTCP message framing. See Transport below
max_connection_ageN0 (off)Age at which a TCP connection is recycled. Not used on UDP. See Transport below
portN2055Listen port
workersNOne per 8 CPU coresNumber of worker goroutines, never below 1 and reduced further on memory-constrained hosts. Forced to 1 when reuse is false.
reuseNfalseEnable socket address reuse

Ingest Buffer​

FieldRequiredDefaultDescription
max_buffer_sizeN128MBIngest buffer admission cap. Also sizes this listener's ingest buffers: 3/8 of this value at start, growing to at most 3x
note

max_buffer_size is the ingest-buffer admission cap, and is distinct from any buffer_size field above, which sizes the network read buffer. Left unset it is 128MB, whose buffers take 48 MiB of heap per listener worker at start and grow to at most 384 MiB as bursts need the room — and with reuse: true a device runs one worker per listener. Size strings are binary, and 32MB and 32MiB are exact synonyms.

See Listener Memory Sizing for the arithmetic, the sizing table, and what happens when the cap is exhausted.

Details​

Transport​

The listener runs over UDP by default, which is how flow exporters almost always send. Setting protocol: tcp binds a TCP listener instead, for an exporter configured to use it or a path that needs the delivery guarantees.

FieldRequiredDefaultDescription
protocolNudpTransport to bind: udp or tcp. network is accepted as an alias and takes precedence when both are set
framingNipfix over TCPHow message boundaries are found on a TCP stream: delimiter, octet or ipfix. Not used on UDP
max_connection_ageN0Age at which a TCP connection is recycled, in seconds or a short form such as "1h". Each connection is closed at a message boundary once it is older than its own jittered lifespan, up to 10% shorter than this value, so the exporter reconnects. The age is checked only as records arrive, so an idle connection is not recycled. 0 turns it off. Not used on UDP. See TCP for how the recycle behaves

Framing​

A UDP datagram is one message, so nothing has to be framed. A TCP stream does, and the default framer looks for newlines — which never appear cleanly in a binary flow stream.

So over TCP the listener selects the ipfix framer automatically, reading the total length from bytes 2-3 of each message header as RFC 7011 §3.1 specifies. Setting framing explicitly overrides that choice.

note

The automatic selection applies to IPFIX and NetFlow v9 only, which share that header layout. sFlow over TCP would need a different framer and is not auto-selected — if you run it, set framing yourself.

TLS over TCP​

TLS is available on the TCP transport, configured with the standard tls block. It has no effect on UDP.

TLS Fields​

FieldRequiredDefaultDescription
tls.statusNfalseEnable TLS encryption
tls.cert_nameY*cert.pemTLS certificate
tls.key_nameY*key.pemTLS private key
tls.passphraseN-Passphrase for an encrypted private key
tls.min_tls_versionNtls1.2Minimum accepted TLS version (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_versionN-Maximum accepted TLS version. When unset, the highest mutually supported version is negotiated.
tls.client_ca_nameN-CA bundle used to verify client certificates (mTLS)
tls.client_auth_requiredN**falseRequire connecting clients to present a valid certificate. When false, a client certificate is verified only if one is presented.
tls.insecure_skip_verifyNfalseHas no effect on a listener. It applies only where the Director connects to a server, never to the connections it accepts: a client certificate is checked only through tls.client_ca_name and tls.client_auth_required. Changing it restarts the listener and nothing else.

* = Required when tls.status is true.

** = Requires tls.client_ca_name. If tls.client_auth_required is true or tls.client_ca_name is set and the named CA cannot be loaded, the configuration is rejected and the device fails to start.

note

tls.min_version is a deprecated alias for tls.min_tls_version, honored only when tls.min_tls_version is unset. Use tls.min_tls_version in new configurations.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

  • File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
  • Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
  • Inline PEM content — used verbatim when the value contains -----BEGIN.
  • Environment variable — ${ENV_VAR}.
  • Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

NetFlow, sFlow, and IPFIX devices share a common flow collection backend (backend/module/listener/flow/). The thin per-protocol controller sets the flow type and default port.

When reuse is enabled, the collector spawns multiple workers which maintain their own UDP listeners, process flows independently, and write to dedicated queue files. The count scales to one worker per 8 CPU cores; without reuse it is forced to 1.

The collector supports fixed format NetFlow v5 records, application identification, port-based protocol mapping, flow state tracking, and statistical aggregation.

Examples​

The following are commonly used configuration types.

Basic​

Creating a simple NetFlow v5 collector on the default port...

devices:
  - id: 1
    name: basic_netflow
    type: netflow
    properties:
      port: 2055

High-Volume​

Optimizing for high flow volumes using multiple workers...

devices:
  - id: 2
    name: performant_netflow
    type: netflow
    properties:
      address: "0.0.0.0"
      port: 2055
      reuse: true
      workers: 4

Application Protocols​

The collector supports application-based identification, TCP port mapping, and UDP port mapping. The definition files for these respectively are placed in three locations under <vm_root>:

  • /user/definitions/app-definitions-{device-id}.csv (device-specific)
  • /user/definitions/app-definitions.csv (user-defined)
  • /package/definitions/app-definitions.csv (system defaults)
  • /user/definitions/tcp-definitions-{device-id}.csv (device-specific)
  • /user/definitions/tcp-definitions.csv (user-defined)
  • /package/definitions/tcp-definitions.csv (system defaults)
  • /user/definitions/udp-definitions-{device-id}.csv (device-specific)
  • /user/definitions/udp-definitions.csv (user-defined)
  • /package/definitions/udp-definitions.csv (system defaults):::warning

Definition files must be in CSV format with exactly two columns per row. :::

The contents of these files are:

app-definitions.csv:

SSH,Secure Shell
RDP,Remote Desktop
HTTP,Web Browsing
HTTPS,Secure Web

tcp-definitions.csv:

22,SSH
3389,RDP
80,HTTP
443,HTTPS

udp-definitions.csv:

53,DNS
67,DHCP
123,NTP
161,SNMP
note

The collector will fall back on the system defaults if the custom definition files are not found.

NetFlow collector with application identification enabled...

devices:
  - id: 3
    name: app_aware_netflow
    type: netflow
    properties:
      port: 2055
      reuse: true