File

Pull

Synopsis

Director polls glob patterns on its own filesystem (or any mounted path) at a configured interval and forwards matched log lines through an optional pipeline. No Agent is required.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: file
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    path: <string>
    pipeline_name: <string>
    poll_interval: <numeric>
    scan_interval: <numeric>
    idle_timeout: <numeric>
    file_log_concurrency: <numeric>
    start_date: <numeric>
    ignore_cache: <boolean>
    ignore_old_date: <boolean>
    ignore_retention: <boolean>
    ignore_time: <boolean>
    date_format: <string>
    line_parser: <string|map>
    encoding: <string>
    filter_mode: <string>
    filter_rules: <map[]|string[]>
    compute_pattern: <boolean>
    compute_level: <boolean>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be file
tags	N	-	Optional tags
status	N	true	Enable/disable the device

File Source

Field	Required	Default	Description
path	Y	-	Comma-separated glob pattern(s) to scan. Each entry is treated as an independent glob; whitespace around commas is trimmed and empty segments are dropped. Supports ** for recursive directory matches.
pipeline_name	N	-	Name of the pipeline that pre-processes matched lines. Empty string passes lines through unprocessed.

Polling

Field	Required	Default	Description
poll_interval	N	0	Fallback stat-poll cadence in milliseconds for per-file workers. 0 disables the stat poll; reads are driven by fsnotify events. Must be >= 0. Changing this value restarts the collector.
scan_interval	N	10000	Directory glob cadence in milliseconds. Discovers files newly matching the path glob and prunes entries that no longer match. Must be greater than 0. Changing this value restarts the collector.
idle_timeout	N	10000	Per-file worker idle shutdown threshold in milliseconds. A worker that receives no fsnotify events within this window exits; the scanner spawns a replacement on its next tick. Must be greater than 0. Changing this value restarts the collector.
file_log_concurrency	N	8	Maximum number of files read in parallel. Higher values increase throughput at the cost of memory. Changing this value restarts the collector.
start_date	N	300	Lookback window in seconds. It chooses which files are read, by their modification time, and on a file read from its start, which lines are kept. 0, or leaving it out, uses the default of 300 seconds (5 minutes). -1 (or any negative value) disables time-based filtering entirely.

Reader Options

Field	Required	Default	Description
ignore_cache	N	false	Skip the persisted file-position cache and re-read from the beginning of each file.
ignore_old_date	N	false	Skip the reader's old-date filter.
ignore_retention	N	true	Skip retention-based filtering. Defaults to true because the built-in 1-day retention window silently drops historical logs; set to false to re-enable retention filtering.
ignore_time	N	false	Skip per-line time filtering.
date_format	N	-	Custom log timestamp format. Uses Go's reference time layout (2006-01-02T15:04:05Z07:00), not strftime.

All reader options are hot-reloaded on the next poll tick without restarting the collector.

Line Parser

Controls how individual lines are grouped into log entries.

Field	Required	Default	Description
line_parser	N	-	Line parser definition. Accepts either a map (preferred) or a bare string shorthand.
line_parser.type	N*	-	Parser mode: regex, newline (alias new_line), string, or prefix. Numeric aliases: 1 = regex, 2 = newline, 3 = string.
line_parser.regex	N*	-	Regex pattern that detects the start of a new log entry. Alias: value.
line_parser.date_based	N	false	Use date-based multiline merging.
line_parser.has_space	N	false	Treat leading whitespace as a line-continuation marker.

* Required when using the map form with type: regex.

A bare string value for line_parser is treated as a regex pattern equivalent to type: regex with that pattern.

Encoding

Field	Required	Default	Description
encoding	N	-	Character encoding of the source files. Accepts an alias (case-insensitive; -, _, spaces, and dots are stripped) or a numeric decoder ID.

Supported aliases:

Alias	Encoding
utf8	UTF-8
utf8bom	UTF-8 with BOM
utf16be	UTF-16 Big Endian
utf16le	UTF-16 Little Endian
utf16bebom	UTF-16 BE with BOM
utf16lebom	UTF-16 LE with BOM
gbk	GBK (Simplified Chinese)
latin1, iso88591	ISO 8859-1 / Latin-1
windows1250, cp1250	Windows-1250 (Central European)
windows1251, cp1251	Windows-1251 (Cyrillic)
windows1252, cp1252	Windows-1252 (Western European)
windows1256, cp1256	Windows-1256 (Arabic)

The encoding applies to text files and to JSON files alike, and to each member of a .zip archive, which is read as its own source. Lines are split on the encoding's own line terminator, so a UTF-16 file breaks on the two-byte newline rather than on a bare 0x0A byte.

Upgrading

Earlier versions accepted encoding but applied it only to JSON files. Text files were read as raw bytes, so a UTF-16 or code-page source arrived garbled, and UTF-16 lines kept a stray NUL byte at their start. If you compensated downstream -- a processor stripping NULs, a gsub, or a re-encode at the target -- remove the workaround when you upgrade. The Director now decodes those files correctly, and the workaround will corrupt the correctly decoded text.

note

The EBCDIC code pages (decoder IDs 40--42) end lines with NEL (0x15), not LF, so they are not split into lines correctly. No alias maps to them; they are reachable only by numeric ID.

Filtering

Field	Required	Default	Description
filter_mode	N	-	Filter direction: include keeps only matching lines; exclude drops matching lines.
filter_rules	N	-	List of filter rules. Accepts map form or a bare list of strings (treated as regex rules).
filter_rules[].type	N*	-	Rule type: regex or string.
filter_rules[].regex	N*	-	Regex pattern to match against each line. Required when type: regex.
filter_rules[].source	N*	-	Substring or wildcard pattern to match. Alias: value. Required when type: string.

* Required for each rule entry.

A bare list of strings is accepted as shorthand and treated as regex rules.

Enrichment

Field	Required	Default	Description
compute_pattern	N	false	Eagerly compute a log-pattern field (SystemD9/S9/S8) per record during file reading. When false, pattern computation can be delegated to the pipeline.
compute_level	N	false	Eagerly compute a severity-level field (SystemD10/S10) per record during file reading. When false, level computation can be delegated to the pipeline.

Structural Decoding

Field	Required	Default	Description
decoder	N	-	Structural (binary) decoder to apply to matched files instead of the text/line reader. sap_audit decodes SAP Security Audit Log (.AUD) records into one ECS JSON document per record, using the compact audappend layout (see the layout note below). Files matched by path with a .aud extension (case-insensitive) select this decoder automatically; set explicitly to decode files under another extension.

Ingest Buffer

Field	Required	Default	Description
max_buffer_size	N	128MB	Ingest buffer admission cap. Also sizes this listener's ingest buffers: 3/8 of this value at start, growing to at most 3x

note

max_buffer_size is the ingest-buffer admission cap, and is distinct from any buffer_size field above, which sizes the network read buffer. Left unset it is 128MB, whose buffers take 48 MiB of heap per listener worker at start and grow to at most 384 MiB as bursts need the room — and with reuse: true a device runs one worker per listener. Size strings are binary, and 32MB and 32MiB are exact synonyms.

See Listener Memory Sizing for the arithmetic, the sizing table, and what happens when the cap is exhausted.

Details

Path Resolution

path accepts a single string that may contain comma-separated glob expressions. Each entry is processed as an independent glob after whitespace trimming; empty segments (e.g., trailing commas) are discarded. Each path is normalized via filepath.Clean before globbing. The ** double-star pattern matches recursively across directory levels.

Hot Reload vs Restart

Most configuration changes take effect on the next poll tick without interrupting the collector:

• Hot-reload (no restart): path, start_date, ignore_cache, ignore_old_date, ignore_retention, ignore_time, date_format, line_parser, encoding, filter_mode, filter_rules, compute_pattern, compute_level, decoder
• Restart required: poll_interval, scan_interval, idle_timeout, file_log_concurrency

SAP Audit Log (.AUD) Decoding

A file matched by path with a .aud extension, matched case-insensitively (.AUD, .Aud, .aud), is read with the SAP Security Audit Log binary decoder instead of the text/line reader — no decoder property is required. Set decoder: sap_audit explicitly to decode files under another extension. The extension check is always case-insensitive; glob matching in path itself follows the host filesystem's case sensitivity instead — case-insensitive on Windows, case-sensitive on Linux and macOS.

Each fixed-width, UTF-16 record is decoded into one ECS JSON line. Byte order is auto-detected from a BOM, else inferred from the first record (little-endian when ambiguous); set encoding to utf16be or utf16le to pin it. The rendered message, event.code (the 3-character SAP audit message ID), and sap.audit_class / sap.event_class come from an SM20 message catalog (sm20.csv) resolved in order: a copy in the Director's user config directory, then the shipped copy in its package directory, then an embedded default catalog. sap.terminal, sap.transaction, sap.report, sap.instance, sap.counter, and sap.flags are populated from every record; sap.slot, sap.slot_class, sap.slot_severity, sap.slot_user, and sap.slot_client are populated only for the audit-configuration events (AUF/AUI) that describe an audit slot's filter scope, not the user who triggered the event. sap.username and sap.client are part of the ECS output shape but are not populated by this decoder — see the layout note below.

warning

The decoder reads the compact audappend layout only. The backend also implements the real SAP NetWeaver SAL format-2 layout, but nothing selects it: the decoder is constructed without a format option, so it always falls back to the compact layout. Per the field offsets in the backend source, the compact layout mis-reads a genuine NetWeaver .AUD file, and the fields carried only by format-2 — sap.username, sap.client, sap.process_id, sap.task, sap.process_type, sap.dialog_mode — stay empty.

Treat .AUD support as covering files produced by the bundled audappend generator. Verify against your own data before relying on it for production SAP audit logs.

Time-Based Filtering

start_date is applied against each file's modification time before the file is read:

• Positive value (e.g., 300): Only files modified within the last N seconds are processed.
• 0: Uses the default of 300 seconds.
• Negative value (e.g., -1): Disables time-based filtering; all matched files are processed regardless of modification time.

Startup Behavior

At startup the collector sleeps for a random interval of 0–20 seconds to spread load when multiple file devices start simultaneously. An initial directory scan runs immediately after this delay; subsequent scans repeat at scan_interval. Per-file workers are driven by fsnotify events; poll_interval provides an optional fallback stat poll for environments where fsnotify is unreliable (network mounts, some container setups).

A heartbeat monitor checks that the collector reports progress within 120 seconds. If the heartbeat threshold is exceeded the collector is stopped and the device connection state is set to error.

Security

Symlink containment and allow-listed root path enforcement are not implemented. The Director follows symlinks without restriction.

warning

Operators are responsible for ensuring that configured paths do not expose unintended parts of the filesystem.

Examples

Single Glob

Collecting all .log files under /var/log/app/ with a 5-minute lookback window and a 60 ms fallback poll...

- id: 1 name: app-logs type: file properties: path: "/var/log/app/*.log" poll_interval: 60 start_date: 300

Multiple Globs with Pipeline

Scanning two directory trees with a single comma-separated path and routing matched lines through a preprocessing pipeline...

- id: 2 name: app-and-archive-logs type: file properties: path: "/var/log/app/*.log,/var/log/app/archive/*.log" pipeline_name: "normalize-timestamps" poll_interval: 120 file_log_concurrency: 4 start_date: 600

Multiline Log Entries

Merging Java-style stack traces into single log entries using a date-prefix regex to detect the start of each entry...

- id: 3 name: java-app-logs type: file properties: path: "/opt/app/logs/*.log" line_parser: type: regex regex: '^\d{4}-\d{2}-\d{2}' date_based: true has_space: true date_format: "2006-01-02T15:04:05.000Z07:00"

Filter Rules

Including only ERROR and WARN lines while excluding health-check entries using mixed filter rule types...

- id: 4 name: filtered-app-logs type: file properties: path: "/var/log/app/*.log" filter_mode: include filter_rules: - type: regex regex: '^(ERROR|WARN)' - type: string source: '*healthcheck*'

Full Historical Scan

Re-reading all matched files from the beginning by disabling time filtering and resetting the position cache, useful for reprocessing after a pipeline change...

- id: 5 name: historical-scan type: file properties: path: "/mnt/archive/logs/**/*.log" start_date: -1 ignore_cache: true file_log_concurrency: 8 pipeline_name: "reprocess-pipeline"

SAP Audit Log

Reading SAP NetWeaver Security Audit Log files; the .AUD extension auto-selects the binary decoder, no decoder property needed. Compact audappend layout only — see the layout note under Details...

- id: 6 name: sap-audit-log type: file properties: path: "/var/log/sap/*.AUD" start_date: -1