Workflows

The tools are designed to be used in sequence rather than individually. Three sequences cover most of what the server is for: authoring a pipeline from scratch, migrating one from another platform, and exporting one back to KQL.

Each is described as the sequence of tool calls an agent makes. You direct the agent in your own words; what follows is what it does on your behalf, and what the intermediate results mean when you review them.

Authoring and Verifying a Pipeline

The path from a raw log to a pipeline you trust runs through four tools in a fixed order, because each one answers a question the next depends on.

The agent begins by reading the relevant authoring guide with get_skill — building-a-pack for a source with no pack yet, building-a-pipeline for a pipeline inside an existing one. It then locates the processors it needs with list_processors and reads their option schemas with get_processor, rather than writing a processor name from memory.

With a draft pipeline written, test_pipeline runs a sample log through it and returns the normalized output. This first call is exploratory: you are looking at what the pipeline actually produces, not comparing it to anything. When the output is wrong in a way that is not obvious, setting verbose to true returns the input and output of each processor in turn, which localizes the step that broke.

Once the output is right, it becomes the expected fixture. validate_pipeline then turns that fixture into a regression check that answers pass or fail on every subsequent edit. While a change is still being worked out, diff_pipeline takes the same arguments but returns the two documents and a unified diff, which is the form that identifies the offending field.

Finally, validate_schema checks the normalized event against the schema the pipeline is meant to produce — an ASIM table, an OCSF class, a UDM reference, or a custom schema. This is a different question from the previous two: validate_pipeline asks whether the pipeline produces what you expected, while validate_schema asks whether what you expected is what the target requires. A pipeline can pass the first and fail the second.

Tighten the schema check by degrees. The default reports only required fields; once those pass, setting recommended and then optional raises the bar without changing anything else about the call.

Migrating a Pipeline From Another Platform

Migration inverts the order: the pipeline is produced by a converter, and your work is verifying it rather than writing it.

The agent reads the relevant migration guide, then calls the matching converter — convert_cribl_pipeline, convert_kql_query, or convert_logstash_pipeline — with the source configuration inline. What comes back is a native DataStream pipeline, and it is a draft: every converter degrades constructs it cannot translate into comments rather than failing the call.

Reviewing those comments is the substance of the migration. A conversion that returns without error may still be missing behavior, and the comments mark exactly where. In a converted Logstash pipeline, warning lines are also added to the pipeline's description; in a converted Cribl pipeline, functions that were disabled in the original arrive as disabled processors, so the result matches the original rather than quietly enabling something.

Verification then follows the authoring sequence above: test_pipeline against a sample of the source's real logs, an expected fixture, validate_pipeline to lock it in, and validate_schema if the pipeline targets a schema.

The one thing conversion cannot do is tell you whether the original was correct. Comparing the converted pipeline's output against output from the platform you are leaving is what establishes that the migration preserved behavior, and it is worth doing on a sample of genuinely representative events rather than a synthetic one.

For the operator-facing account of each migration, see Migration.

Exporting a Pipeline Back to KQL

convert_pipeline_to_kql runs the conversion in the opposite direction, producing KQL from a DataStream pipeline. It is the tool to use when a pipeline needs to exist as an ASIM parser — for instance, when the same normalization has to run inside Microsoft Sentinel as well as in DataStream.

By default it emits a bare parser query. Setting envelope to true wraps it in the full ASIM parser YAML envelope, which is the form Sentinel expects for a parser you intend to deploy; parser_name sets the function name, falling back to the pipeline's own name when it is omitted.

The export is lossy in a specific and visible way. Processors with no faithful KQL equivalent — reroute, stateful and external processors, and calls into another pipeline — are emitted as // UNSUPPORTED comments. The result is valid KQL, but wherever such a comment appears it does less than the pipeline it came from. Read each one and decide whether the gap matters for the use the query is being put to; there is no general workaround, because the constructs in question have no KQL form.

Exporting is available through the MCP server only. There is no equivalent route in the web interface, whereas the import direction has one — see Pipeline Management.

Working Within the Constraints

Two properties of the server shape all three sequences.

Because arguments are inline and nothing persists between calls, iteration is cheap and stateless: each call carries the current version of the pipeline, and there is no workspace to keep in sync. An agent can revise and re-test a pipeline as many times as it takes without any setup between attempts.

Because the processor guard rejects pipelines that reach the network, execute scripts, or touch shared cluster state, a production pipeline that uses one of those cannot be tested here as it stands. Verify the normalization logic remotely with that processor removed, then confirm the complete pipeline in the Pipeline Debugger. See Limits and Safety for the full set.