Authoring Guides
The server ships six guidance documents that teach an agent DataStream's pipeline-authoring conventions and the mistakes that are easy to make without them. They are embedded in the server itself, so a connected agent has them whether or not anything is installed locally.
An agent discovers them with list_skills, which returns each guide's name and description, and retrieves one in full with get_skill. Both tools are described in Tool Reference.
The guides matter because several of DataStream's conventions fail silently when they are broken — a misparsed condition does not raise an error, it simply never matches. An agent that has read the relevant guide before writing a pipeline avoids a class of bug that testing alone is slow to localize.
The Guides
| Guide | Covers |
|---|---|
writing-normalization-pipelines | The umbrella guide: the tester tools, how a pack is laid out, and how a pack's name prefix selects its target schema |
building-a-pack | Creating a content pack for a new source from nothing — the directory, metadata.yml, the pipeline, and the sample-log and expected-output fixtures |
building-a-pipeline | Adding, fixing, or updating a pipeline inside an existing pack — processor selection, the expression and conditional rules, templates, grok, and date handling |
migrating-from-cribl | Converting a Cribl Stream pipeline, the function-to-processor mapping, and what the conversion cannot carry across |
migrating-from-kql | Converting an ASIM parser or KQL query, the pipe-operator mapping, and which constructs degrade to comment stubs |
migrating-from-logstash | Converting a Logstash configuration, the filter-plugin mapping, and conditional translation |
Choosing Between Them
The three authoring guides form a hierarchy. writing-normalization-pipelines is the entry point and routes to the other two: building-a-pack when a source has no pack yet, building-a-pipeline when the pack exists and a pipeline inside it needs work. An agent that reads only the umbrella guide has the taxonomy but not the detail; an agent that reads only a leaf guide has the detail but may have skipped the schema decision that shapes it.
What the Authoring Guides Teach
writing-normalization-pipelines establishes the target-schema taxonomy — ASIM, OCSF, ECS, or a custom schema — and the rule that a pack's name prefix selects its target schema. It also states the completion criteria a pack is expected to meet.
building-a-pack walks the creation of a pack as a sequence: the four files that constitute it, the metadata, the pipeline, then the expected-output fixture built up iteratively against real output rather than written speculatively in advance, and finally validation.
building-a-pipeline is the guide that concentrates on the traps. It covers choosing processors from the catalog rather than guessing at names, the expression and conditional rules — including the reserved words and non-identifier field names that misparse silently unless referenced through the documented escape form — grok pattern syntax, template and field-reference forms, and date handling. It closes by requiring that every edit is tested rather than assumed.
What the Migration Guides Teach
The three migration guides share a shape: the workflow to follow, a mapping table from the source platform's constructs to DataStream processors, an explicit account of what does not convert, and instructions for handling the comment stubs the converters leave where a construct has no native equivalent.
That last part is the reason to read them. Every converter degrades unsupported constructs into comments rather than failing, so a conversion that returns cleanly may still be incomplete; the guides describe how to find and resolve those places.
The same ground is covered from the operator's perspective in the migration documentation:
Those pages describe the migration as a task you are directing. The guides describe it as a procedure the agent follows, in the detail an agent needs to follow it correctly. See Workflows for how the two fit together.