Codeless Connector Framework (CCF)

Pull

Synopsis

Director polls the source system's API at configured intervals to retrieve log and telemetry data rather than receiving pushed events. The poller is driven by a Microsoft Sentinel Codeless Connector Framework (CCF) PollingConfig.json file — either a single connector object or the as-published array of connectors, in which case each connector runs as its own poller. Secrets and environment-specific values referenced as {{placeholder}} tokens are supplied as separate device properties, keeping credentials out of the polling config. Credentials shared by every connector can be defined once at the device level via ccf_auth.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: ccf
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    ccf_config: <string>
    ccf_auth: <string>
    <placeholder_key>: <string>
    tls:
      insecure_skip_verify: <boolean>
    poll_interval: <numeric>
    ingestion_delay: <numeric>
    workers: <numeric>
    reuse: <boolean>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be ccf
tags	N	-	Optional tags
pipelines	N	-	Optional preprocessing pipelines
status	N	true	Enable/disable the device

CCF Config

Field	Required	Description
ccf_config	Y	Contents of the connector's PollingConfig.json — a single connector object, or a multi-connector array pasted verbatim from a Sentinel solution. In the array form every connector with a request.apiEndpoint runs as its own poller; sibling resources bundled in the same array (DCRs, workspaces) are skipped automatically. Any {{placeholder}} token is resolved from a sibling device property with the same name.
ccf_auth	N	A single CCF auth block (same JSON shape as a connector's properties.auth, {{placeholder}} tokens allowed) used as the default credentials for every connector in ccf_config that does not carry its own auth. A connector-level auth block always takes precedence, so verbatim Sentinel files — where auth is repeated per poller — import unchanged.
<placeholder_key>	N*	Any property whose name matches a {{placeholder}} token in ccf_config or ccf_auth. Typically used for secrets such as API keys and passwords. A placeholder used in a credential position is withheld from the device's read and keeps its stored value when a save sends it blank. See Placeholder Resolution. A {{name|list}} token reads the property name and takes several values; see List Placeholders.

* Required for each {{placeholder}} token present in ccf_config or ccf_auth.

note

A CCF connector package contains multiple files. Only the PollingConfig.json is used here — the connector definition, DCR, and workbook files are not required.

The PollingConfig.json top-level fields connectorDefinitionName, dataType, and dcrConfig are ignored. auth, request, paging, and response drive the poller; the resource name labels the connector in logs and poller state, and disabled turns individual connectors off.

[[concat(...)]] ARM template expressions found in some polling configs are not evaluated. Replace them with {{placeholder}} tokens and supply the resolved value as a sibling property.

TLS

Field	Required	Default	Description
tls.insecure_skip_verify	N	false	Skip TLS certificate verification (useful for dev/test environments or APIs with self-signed certificates)

Performance

These fields are device-level controls and are not part of the connector JSON.

Field	Required	Default	Description
poll_interval	N	10	Polling interval in seconds
ingestion_delay	N	0	Shifts every poll window's end back from now by this many seconds. Use for APIs that make events queryable only some time after they occur (ingestion lag), so late-arriving events are not skipped when the window advances. 0 keeps windows ending at the current time.
workers	N	1	Accepted but not applied. A change restarts the device, and nothing else follows from it
reuse	N	true	Accepted but not applied. A change restarts the device, and nothing else follows from it

Details

Placeholder Resolution

Placeholder resolution runs on the raw JSON string before parsing. If any placeholder cannot be resolved the device fails to start and reports all missing keys in the error message. The same resolution applies to ccf_auth.

A placeholder property is treated as a secret when its {{placeholder}} token appears in a credential position in ccf_config (single-connector or array form) or in ccf_auth. JSON keys are matched case-insensitively, as the importer reads them.

Position	Fields
An auth block's credential field	Basic or Session Password; JwtToken Password pair's value, and flat passwordValue; ApiKey; ClientSecret; AuthorizationCode; UserToken
A header value	The request's headers; an auth block's headers, TokenEndpointHeaders, and JwtTokenEndpointHeaders

Any non-blank value of such a property counts, a number included, since it is substituted into the request as text. This applies to the ccf and ccfreplay device types:

Action	Result
Device read (console, GET /device/{id})	Value omitted; the property name is listed in redactedProperties
Save with the property blank or left out	Stored value kept
Save with a new value	Value replaced
Save naming the property in clearedProperties	Value removed
Value is a ${ENV} or $secret{} reference	Published as usual, since it is a reference rather than the secret

Stored configurations are unchanged, so there is nothing to migrate on upgrade.

warning

A placeholder used only in a URL (apiEndpoint, TokenEndpoint), a query parameter, or a user name or client ID ({{Domain}}, {{tenant}}, {{userid}}, {{ClientId}}) is not a credential position and stays readable. Do not put a credential in a URL or query string if it must stay hidden — put it in a header or an auth credential field.

List Placeholders

A {{name|list}} token stands for several values. |list is a modifier, not part of the property name: {{account_ids|list}} reads the device property account_ids, and the Placeholders editor in the console names it account_ids. This is a VirtualMetric extension, so a connector that uses it does not run on Microsoft Sentinel.

Write the token as the only content of a JSON string that is an element of an array. The string, quotes included, is replaced by one JSON string per value:

"variables": { "accountIDs": ["{{account_ids|list}}"] }

With account_ids set to 1234, 5678, the request carries "accountIDs": ["1234","5678"].

The value is split on commas and line breaks. Spaces around an item are trimmed and empty items are dropped. The | belongs to the token in the connector JSON. It is never a separator in the value:

Value of account_ids	Substituted as
10745	["10745"]
123,456,789	["123","456","789"]
123, 456, 789	["123","456","789"]
123,,456,	["123","456"]
123|456|789	["123|456|789"], one value rather than three

In a Director configuration file the property can also be a YAML list. Each item is one value and is not split further:

properties:
  account_ids:
    - "1234"
    - "5678"

The following rules apply:

• Empty is missing. A value with no items left after splitting, such as an empty string or , ,, counts as a missing placeholder. The device fails to start, and the error names the token, account_ids|list.

• The token is the whole string. Anything else inside the same JSON string fails the import with an error that begins placeholder "account_ids|list" must be the only content of a JSON string. Outside an array, several values do not form valid JSON, and the import fails as well.

• list is the only modifier. Any other, for example {{account_ids|lst}}, fails the import with placeholder "account_ids|lst" has an unknown modifier "lst".

• A queryParametersTemplate that holds one must be a JSON object. In Sentinel's string form, which escapes every quote, the token cannot be the whole content of a JSON string, so the import refuses it. Write the template as an object instead. The poller accepts both forms and compacts the object into the same template text:

  "queryParametersTemplate": {
    "query": "query Feed($accountIDs: [ID!]) { feed(accountIDs: $accountIDs) { marker } }",
    "variables": { "accountIDs": ["{{account_ids|list}}"] }
  }

  Keep the string form for a scalar placeholder that has to be a JSON number, as in "{\"limit\": {{limit}}}". The console saves connector JSON only when it parses with its placeholders still in place, and a bare {{limit}} in the object form does not.

${ENV} and $secret{...} references are resolved differently for a list and for a text value:

Property holds	Resolution	Example
A YAML list	Each item on its own	- ${ACCOUNT_A} and - $secret{id=9} are each resolved
Text	Once, as a whole, before splitting	${ACCOUNT_IDS}, with the variable set to 123,456, gives two values

A reference that is only part of a text value is not resolved. 123,$secret{id=9} sends the literal $secret{id=9} as the second value, and ${A},${B} fails the import. To keep several items behind references of their own, give the property as a YAML list.

Paging Parameters in the Request Body

By default a paging token travels in the query string as nextPageParaName=<token>. On a POST connector it can travel in the JSON request body instead. This applies to the token of NextPageToken and PersistentToken paging:

Connector sets	Paging request carries
nextPageParaName written as a JSONPath, such as "$.variables.after", on a POST	The original body with the token set at that path. With pagingQueryParamOnly: true, a body holding only that path
NextPageToken on a POST with isPostPayloadJson: true	A body holding only {"<nextPageParaName>": "<token>"}
pagingInfoPlacement: "RequestBody" on a POST	The original body with the token set at its top level. With pagingQueryParamOnly: true, a body holding only the token
Anything else, including every GET	The original URL with nextPageParaName=<token> added. With pagingQueryParamOnly: true, every other query parameter is dropped

The first row that matches applies.

Microsoft's CCF reference defines pagingInfoPlacement as QueryString or RequestBody, and pagingQueryParamOnly as omitting every query parameter except the paging parameters. Two behaviours here are VirtualMetric extensions that Microsoft's reference does not describe, so a connector that relies on them is not expected to behave the same on Microsoft Sentinel:

• A JSONPath nextPageParaName sets the token inside the body and keeps the rest of it.
• pagingQueryParamOnly also empties the body when the token is placed there.

pagingInfoPlacement: "RequestBody" on a GET is ignored, since there is no body. A body that is not a JSON object cannot take the token, so the token goes to the query string instead. A JSONPath nextPageParaName on a GET, or on a body that is not a JSON object, is used as a plain name without its leading $..

A Cursor Inside the Query

A GraphQL cursor is one of the query's variables, and the query has to travel with every page. Write nextPageParaName as $. followed by dot-separated keys:

"paging": {
  "pagingType": "NextPageToken",
  "nextPageParaName": "$.variables.after",
  "nextPageTokenJsonPath": "$.data.events.pageInfo.endCursor",
  "hasNextFlagJsonPath": "$.data.events.pageInfo.hasNextPage"
}

With the body {"query": "…", "variables": {"first": 100}}, the second page sends {"query":"…","variables":{"after":"MTAw","first":100}}.

The following rules apply:

• Missing objects are created. A path whose objects the body does not have yet gets them.
• A later page replaces the token. The body never carries two.
• The rest of the body is kept. It is re-encoded compactly with its object keys in alphabetical order, and numbers keep the digits they were written with.
• The same path serves a REST search API that wants the same filter resent beside the cursor, for example "nextPageParaName": "$.cursor".
• A PersistentToken is set at the same path on the first request of the next poll.

A Body That Carries Only the Token

Some APIs issue a cursor that stands for the whole query and refuse a body that carries anything beside it. With pagingQueryParamOnly: true, a body-placed token is sent alone. For a PersistentToken, this applies to the next page and equally to the first request of the next poll:

"paging": {
  "pagingType": "PersistentToken",
  "nextPageParaName": "cursor",
  "pagingInfoPlacement": "RequestBody",
  "pagingQueryParamOnly": true,
  "nextPageTokenJsonPath": "$.cursor",
  "hasNextFlagJsonPath": "$.has_more"
}

The first request sends the configured body, for example {"limit": 100}. Every request after a cursor exists sends {"cursor":"<token>"}.

Has-Next Comparison

hasNextFlagJsonPath names the response field that says whether more pages are left. On its own it is read as a boolean: paging continues while the value is true, "true", "True" or 1.

Some APIs leave "is there more" to a count the caller compares. hasNextFlagOperator and hasNextFlagValue turn the flag into a comparison. They are VirtualMetric extensions that Microsoft's CCF reference does not define, so a connector that uses them does not behave the same on Microsoft Sentinel:

"hasNextFlagJsonPath": "$.meta.returned",
"hasNextFlagOperator": "GreaterThanOrEqual",
"hasNextFlagValue": 1000

This asks for another page while a page returns 1000 or more records.

Operator	Also written
Equals	eq, ==, =
NotEquals	ne, !=
GreaterThan	gt, >
GreaterThanOrEqual	gte, >=
LessThan	lt, <
LessThanOrEqual	lte, <=

The following rules apply:

• Equals is the default. Setting hasNextFlagValue without an operator compares for equality. Operators are matched without regard to case.
• An unknown operator fails the import with an error that reads unsupported hasNextFlagOperator "<operator>".
• The value is a JSON scalar. 3000, "3000" and true are all accepted. An object or an array fails the import.
• Numbers compare as numbers, anything else as text. When both sides read as numbers they are compared numerically. Otherwise they are compared as text, ignoring case, and an ordering operator such as GreaterThan is false, which ends the paging.
• An array is compared by its length. "The page came back full" can be written against the events themselves: "hasNextFlagJsonPath": "$.items", "hasNextFlagOperator": ">=", "hasNextFlagValue": 100.
• A path that cannot be read ends the paging. A missing or null value is compared as empty text against hasNextFlagValue. With a non-empty hasNextFlagValue, that ends the paging for every operator except NotEquals, which goes on paging.
• A PersistentToken is still stored. On the last, short page the token is kept for the next poll: the flag decides whether to ask again now, not which position is remembered.

The flag, boolean or compared, is read by NextPageToken, PersistentToken, NextPageUrl and CountBasedPaging. Token paging also ends when no token comes back.

Response Checks

Some APIs report a failure inside an HTTP 200 response. GraphQL APIs return a top-level errors array, and an API can answer a throttled or unauthorized call with the requested field set to null. Read as a page, that looks like "no new events": the poll succeeds, the window moves on, and nothing is collected.

Microsoft's CCF reference offers successStatusJsonPath and successStatusValue, which accept a page only when the value at the path equals the given value. Two further lists under response check a page before any of its events are shipped. They are VirtualMetric extensions that Microsoft's CCF reference does not define:

"response": {
  "eventsJsonPaths": ["$.data.events.nodes"],
  "requiredJsonPaths": ["$.data.events"],
  "errorJsonPaths": ["$.errors"]
}

Field	A page passes when
requiredJsonPaths	Every path resolves to a value that is not null
errorJsonPaths	Every path resolves to nothing: null, "", 0, false, {}, [], or an array holding only those. A path that is not in the response passes

A wildcard error path such as $.results[*].error resolves to ["", ""] on a good page, and passes.

A failed check fails the poll:

• Nothing from the failed page is shipped.
• The window cursor does not move, so the next poll asks again from the same start. Pages shipped earlier in the same poll are shipped again by that retry.
• A PersistentToken connector keeps the token of the last page it shipped.

The error in the log reads response reports an error at '<path>': <value> for an error path, or response has no value at required path '<path>': <response> for a required path. The value, or the head of the response, is cut to 500 characters, which is usually enough to show the API's own message.

tip

Add "$.errors" to errorJsonPaths on every GraphQL connector. A non-empty errors array beside usable data is a partial result. Shipping it lets the cursor or the window move past the records that failed, while failing the poll loses nothing. Leave it out only for an API known to report non-fatal notices there.

Fractional Rate Limit

rateLimitQPS is the number of requests per second. Microsoft's CCF reference documents it as an integer that applies to the initial request, with a separate PaginatedCallsPerSecond for paging requests. VirtualMetric differs in both:

• A fraction is allowed. An API limited per minute needs less than one request a second.
• One setting paces every request. The poller waits 1 / rateLimitQPS seconds before the first request of a poll and before every further page. PaginatedCallsPerSecond is not read. A retry waits its own back-off instead.

rateLimitQPS	Wait before each request	Requests a minute, at most
0.08	12.5 seconds	about 5
0.5	2 seconds	30
1.5	about 0.67 seconds	90
10	0.1 seconds	600

Keep poll_interval above the wait, or polls run back to back. 0 or a value left out disables rate limiting.

Multi-Connector Configs

Sentinel solutions often publish several pollers in one PollingConfig.json array (for example 1Password ships signinattempts, auditevents, and itemusages connectors in a single file). Pasting that array into ccf_config runs one poller per connector under the same device. Each connector polls independently with its own HTTP client, retry/rate-limit settings, and cursor state, keyed by the connector's resource name (falling back to its array position).

Individual connectors can be switched off without editing them out of the file by adding "disabled": true at the connector's top level or inside its properties. A config in which every connector is disabled is a configuration error.

Shared Authentication

When all connectors of a device authenticate the same way, define the credentials once in ccf_auth instead of repeating the auth block per connector. Connectors without their own auth inherit it; a connector-level auth block always wins.

For refreshable auth types (OAuth2, JwtToken, Session), the acquired token is cached per device and shared by all of its connectors: the first poller fetches the token, the others reuse it, and an expiry or invalidation triggered by any connector refreshes it for all of them. Static credentials (Basic, APIKey) involve no token exchange.

Poller State

Each connector's poll cursor (and persistent paging token or link, where the paging type uses one) is stored in Director's cluster-replicated persistent store. After a restart the poller resumes from its stored cursor — records already collected are not re-fetched, and events that occurred during the downtime are picked up by the catch-up window. On a first start (no stored cursor) the poller looks back one queryWindowInMin, not the full history.

Ingestion Delay

Poll windows normally end at the current time. Some APIs ingest events with a delay — an event with timestamp T only becomes queryable at T + lag — which can cause a window that has already advanced past T to miss it permanently. Setting ingestion_delay to a value larger than the source's worst-case lag (for example 120 for the 1Password Events API) shifts every window back by that amount so late-arriving events are always still inside a future window. The trade-off is that delivery lags real time by the configured delay.

Defaults

When optional numeric fields such as retryCount, timeoutInSeconds, queryWindowInMin, and pageSize are omitted from the connector JSON, the following defaults are applied:

Field	Default
queryWindowInMin	5
retryCount	3
timeoutInSeconds	20
pageSize	100

If these fields are present in the connector JSON they are used as-is. An explicit value of 0 or negative causes the device to silently fall back to the default value listed above.

rateLimitQPS and maxPages are exceptions: 0 is a valid value for both. A rateLimitQPS of 0 disables rate limiting; a maxPages of 0 allows unlimited pages. rateLimitQPS also takes a fraction; see Fractional Rate Limit.

Replay

The ccfreplay device type runs the configured connectors over a bounded historical window instead of on a continuing poll. Connection, authentication and connector properties are unchanged; the window, chunking and run controls are described in Replay Devices.

Four behaviours differ from the streaming device:

• The window and the schedule are device-level. poll_interval and each connector's own cursor do not apply, because one run walks every connector's pending chunks in a single pass.
• The chunk defaults to the connector's own query window. Rather than the usual one hour, a replay chunk defaults to each connector's queryWindowInMin — the window size that connector was written against — and window_chunk overrides it.
• Only time-bounded connectors take part. A connector binds the chunk through startTimeAttributeName and endTimeAttributeName, or through the _QueryWindowStartTime and _QueryWindowEndTime placeholders in its parameters. A connector paging by PersistentToken or PersistentLinkHeader cannot be resumed at an arbitrary point in time and is skipped; the discovery preview says so before the run starts.
• The streaming device's state is never touched. Its cursor, token and link state are left as they are.

Field	Required	Default	Description
start_date	Y	-	Lower bound of the replay window
end_date	N	-	Upper bound, exclusive. Unset means an open window that follows the present
timezone	N	UTC	Zone used to read an offset-less bound
window_chunk	N	The connector's queryWindowInMin	Seconds requested per call. Minimum 60
interval	N	60	Seconds between runs while the window still has work
cron	N	-	Cron expression for the run, taking precedence over interval

Examples

Basic Auth (Confluence Audit Logs)

Polling Confluence audit logs using Basic authentication with offset-based pagination...

- id: 1 name: confluence_audit type: ccf properties: userid: "alice@example.com" apikey: "my-secret-key" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "Basic", "UserName": "{{userid}}", "Password": "{{apikey}}" }, "request": { "apiEndpoint": "https://myorg.atlassian.net/wiki/rest/api/audit", "httpMethod": "GET", "retryCount": 3, "timeoutInSeconds": 60, "queryTimeFormat": "UnixTimestampInMills", "startTimeAttributeName": "startDate", "endTimeAttributeName": "endDate", "headers": { "Accept": "application/json" } }, "paging": { "pagingType": "Offset", "offsetParaName": "start", "pageSizeParaName": "limit", "pageSize": 1000 }, "response": { "eventsJsonPaths": ["$.results"], "format": "json" } } }

OAuth2 (Auth0)

Polling Auth0 logs using OAuth2 client credentials with persistent token pagination across poll cycles...

- id: 2 name: auth0_logs type: ccf properties: ClientId: "my-client-id" ClientSecret: "my-client-secret" Domain: "https://myorg.auth0.com" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "OAuth2", "ClientId": "{{ClientId}}", "ClientSecret": "{{ClientSecret}}", "GrantType": "client_credentials", "TokenEndpoint": "{{Domain}}/oauth/token", "TokenEndpointQueryParameters": { "audience": "{{Domain}}/api/v2/" } }, "request": { "apiEndpoint": "{{Domain}}/api/v2/logs", "httpMethod": "GET", "headers": { "Accept": "application/json" } }, "Paging": { "pagingType": "PersistentToken", "nextPageParaName": "from", "nextPageTokenJsonPath": "$.[-1:].log_id", "PageSizeParameterName": "take", "PageSize": 100 }, "response": { "eventsJsonPaths": ["$"], "format": "json" } } }

API Key (Illumio)

Polling Illumio Insights using an API key header with a rate limit and extended query window...

- id: 3 name: illumio_insights type: ccf properties: apiToken: "my-api-token" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "APIKey", "ApiKey": "{{apiToken}}", "ApiKeyName": "X-api-secret" }, "request": { "apiEndpoint": "https://gw.console.illum.io/api/v1/insights-summary", "httpMethod": "GET", "rateLimitQPS": 10, "queryWindowInMin": 360, "retryCount": 3, "timeoutInSeconds": 60, "headers": { "Accept": "application/json" }, "queryParameters": { "persona": "threat_hunter", "report_interval": "daily" } }, "response": { "eventsJsonPaths": ["$"], "format": "json" } } }

Multi-Connector with Shared Auth (1Password)

Polling all three 1Password Events API connectors from one device: the verbatim Sentinel array runs one poller per connector, credentials are defined once in ccf_auth and inherited by every connector, the item-usage stream is switched off with disabled, and ingestion_delay compensates for the API's ingestion lag...

- id: 5 name: onepassword_events type: ccf properties: ApiToken: "my-events-api-bearer-token" BaseUrl: "https://events.1password.com" poll_interval: 10 ingestion_delay: 120 ccf_auth: | { "type": "APIKey", "ApiKey": "{{ApiToken}}", "ApiKeyName": "Authorization", "ApiKeyIdentifier": "Bearer" } ccf_config: | [ { "name": "OnePasswordSignInEvents", "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/signinattempts", "httpMethod": "Post", "queryWindowInMin": 5, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } }, { "name": "OnePasswordAuditEvents", "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/auditevents", "httpMethod": "Post", "queryWindowInMin": 5, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } }, { "name": "OnePasswordItemUsageEvents", "disabled": true, "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/itemusages", "httpMethod": "Post", "queryWindowInMin": 1, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } } ]

GraphQL with a Cursor in the Body

Polling a GraphQL API: the query travels as an object-form queryParametersTemplate, the cursor is set inside its variables on every page, a non-empty errors array or a null result fails the poll instead of reading as an empty page, and the connector is held to 30 requests a minute...

- id: 6 name: graphql_events type: ccf properties: apiToken: "my-api-token" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "APIKey", "ApiKey": "{{apiToken}}", "ApiKeyName": "Authorization", "ApiKeyIdentifier": "Bearer" }, "request": { "apiEndpoint": "https://api.example.com/graphql", "httpMethod": "POST", "isPostPayloadJson": true, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "rateLimitQPS": 0.5, "queryParametersTemplate": { "query": "query Events($since: String!, $until: String!, $first: Int!, $after: String) { events(since: $since, until: $until, first: $first, after: $after) { nodes { id time type } pageInfo { endCursor hasNextPage } } }", "variables": { "since": "{_QueryWindowStartTime}", "until": "{_QueryWindowEndTime}", "first": 100 } } }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "$.variables.after", "nextPageTokenJsonPath": "$.data.events.pageInfo.endCursor", "hasNextFlagJsonPath": "$.data.events.pageInfo.hasNextPage" }, "response": { "eventsJsonPaths": ["$.data.events.nodes"], "requiredJsonPaths": ["$.data.events"], "errorJsonPaths": ["$.errors"] } } }

With Pipeline Processing

Attaching preprocessing pipelines to normalize timestamps and enrich fields before events reach the route...

- id: 4 name: ccf_with_pipeline type: ccf pipelines: - timestamp_normalizer - field_enrichment properties: apiKey: "my-key" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "APIKey", "ApiKey": "{{apiKey}}", "ApiKeyName": "Authorization", "ApiKeyIdentifier": "Bearer" }, "request": { "apiEndpoint": "https://api.example.com/v1/events", "httpMethod": "GET" }, "response": { "eventsJsonPaths": ["$.events"], "format": "json" } } } poll_interval: 30