Amazon Security Lake

Amazon AWS Security Lake

Synopsis

Creates a target that writes security and compliance log data to Amazon Security Lake, AWS's purpose-built data lake for security data. A target is one Security Lake custom source: it writes Parquet files with the schema of one OCSF (Open Cybersecurity Schema Framework) event class under that source's path in the Security Lake bucket, which is what Security Lake ingests. A custom source is registered for one event class, so a second class is a second target.

Schema

- name: <string>
  description: <string>
  type: awssecuritylake
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    part_size: <numeric>
    timeout: <numeric>
    bucket: <string>
    source: <string>
    account: <string>
    schema: <string>
    format: <string>
    compression: <string>
    name: <string>
    max_size: <numeric>
    batch_size: <numeric>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be awssecuritylake
pipelines	N		Optional pipeline for OCSF normalization
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region where Security Lake is configured (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom endpoint URL (for testing or alternate configurations)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure. The target signs with the pair only: a key without a secret, or the other way round, is ignored and the role is used. The console asks for the second one as soon as the first is set.

Security Lake Configuration

Field	Required	Default	Description
bucket	Y	-	The S3 bucket Security Lake stores its data in. The target checks that it can reach it when it starts
source	Y	-	Custom source name, as registered in Security Lake
account	Y	-	AWS account ID where Security Lake is configured
schema	Y	-	The OCSF event class the custom source is registered with, e.g. OCSF4001 for Network Activity. Every file is written with this class's schema. Required for parquet and avro
format	N	-	File format. Set parquet: Security Lake ingests nothing else. Left unset in a YAML, the objects are written as JSON lines. The console always sends parquet and has no field for it
compression	N	zstd	Compression codec inside the Parquet file
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File name template. See Templates. An explicit empty value is refused; leave the key out for the default

Connection

Field	Required	Default	Description
part_size	N	5	Multipart upload part size in megabytes (minimum 5MB)
timeout	N	30	Connection timeout in seconds

File Rotation

Field	Required	Default	Description
max_size	N	33554432	Maximum file size in bytes before rotation (32 MB)
batch_size	N	100000	Maximum number of messages per file

note

When max_size is reached, the current file is uploaded to Security Lake and a new file is created. Setting max_size to 0 does not disable rotation — an explicit 0 is replaced by the 32 MB default (MustInt64), so there is no unlimited setting. Raise the value instead.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Amazon Security Lake target integrates with Amazon Security Lake, providing a centralized security data lake for collecting, normalizing, and analyzing security logs across your AWS environment and third-party sources. This target organizes data using Security Lake's partitioning structure automatically, and writes it in the OCSF-compliant Parquet format of the event class you configure.

Console

In the console, the target's Security Lake settings step holds Bucket and Account ID, Custom source and OCSF event class, File name template and Compression, and the file rotation fields Maximum file size and Batch size. The event class is picked from a list that names each class with its number, such as Network Activity (4001). There is no format field: the console saves format: parquet for every target. The detail page shows the same fields.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

At startup the target probes bucket with a head-bucket call, which checks the credentials and the access to that bucket in one step. The credentials therefore need s3:ListBucket on the bucket, and nothing on the account as a whole.

IAM Permissions

The Amazon Security Lake target uses the S3 upload implementation internally. The following permissions are required:

IAM Action	Purpose
s3:ListBucket	Probe the bucket at startup (HeadBucket)
s3:PutObject	Upload Parquet files to Security Lake bucket
s3:CreateMultipartUpload	Initiate multipart upload for large files
s3:UploadPart	Upload individual parts of multipart upload
s3:CompleteMultipartUpload	Complete multipart upload
s3:AbortMultipartUpload	Abort failed multipart uploads

Minimum IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketProbe",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::SECURITY_LAKE_BUCKET"
    },
    {
      "Sid": "S3Upload",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:CreateMultipartUpload",
        "s3:UploadPart",
        "s3:CompleteMultipartUpload",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::SECURITY_LAKE_BUCKET/*"
    }
  ]
}

note

Objects are uploaded with a Security Lake path prefix: ext/{source}/region={region}/accountId={account}/eventDay={date}/{file}. Ensure the IAM policy covers the full object path.

Security Lake Integration

Amazon Security Lake provides a centralized repository for security data, automatically normalizing logs into the Open Cybersecurity Schema Framework (OCSF) format. The target handles the required partitioning structure (ext/{source}/region={region}/accountId={accountId}/eventDay={YYYYMMDD}/) automatically, ensuring data is properly organized for Security Lake ingestion and analysis.

Parquet is the only format Amazon Security Lake accepts. The schema names the OCSF event class the custom source is registered with, and determines the structure of every file the target writes. compression names the codec used inside the Parquet file.

One Event Class per Target

A Security Lake custom source is registered for one OCSF event class, and every file of a target is written under its source with the schema of its schema. To deliver a second class, add a second target with its own custom source, and route each class's events to its target in the pipeline.

A buckets list under properties, with bucket, name and schema per entry, is still read, but an entry receives only the events a pipeline routes to it by bucket name. The aws_lake pack routes nothing that way, so a target that has such a list and no top-level bucket refuses every event as an unknown destination. A target saved from an earlier console version carries that shape: open it in the console and save it, and the first entry becomes the target's bucket, name and schema.

VirtualMetric Amazon Security Lake Pack

VirtualMetric provides the Amazon Security Lake Automation and Normalization Pack, which normalizes and routes data for Amazon Security Lake. This automation pack transforms diverse security data sources into OCSF-compliant format, including:

• Syslog messages (native, CEF, LEEF formats)
• Windows Security Events and Windows Firewall logs
• Firewall logs from major vendors (Fortinet, Palo Alto Networks, Check Point, Cisco ASA, SonicWall, WatchGuard, Cisco Meraki)
• Windows DNS logs

The pack implements a sophisticated multi-stage processing pipeline with intelligent source detection, vendor-specific optimization, and automatic OCSF schema compliance. It supports all OCSF schema classes and handles the complete transformation from source format to OCSF without requiring manual schema configuration in the pipeline.

When using the VirtualMetric Amazon Security Lake Pack (pipeline: aws_lake), data is automatically normalized to OCSF format before reaching the target. The pack handles all intermediate transformations, including CEF/LEEF to CommonSecurityLog, ECS to ASIM, and ASIM to OCSF conversions. You only need to give the target the custom source and the OCSF event class it is registered with.

OCSF Schema Identifiers

Amazon Security Lake uses OCSF schema classes to categorize security events. Common schema identifiers include:

Schema ID	Description	Event Types
OCSF1001	File Activity	File access, creation, deletion, modification
OCSF1002	Kernel Extension Activity	Kernel module operations
OCSF1003	Kernel Activity	System calls, kernel events
OCSF1004	Memory Activity	Memory allocation, access patterns
OCSF1005	Module Activity	Library loading, dynamic linking
OCSF1006	Scheduled Job Activity	Cron jobs, task scheduler
OCSF1007	Process Activity	Process creation, termination
OCSF2001	Security Finding	Vulnerability findings, security issues
OCSF3001	Account Change	User account modifications
OCSF3002	Authentication	Login, logout, authentication events
OCSF3003	Authorize Session	Session authorization, access control
OCSF3004	Entity Management	Identity and entity operations
OCSF3005	User Access Management	Permission changes, role assignments
OCSF4001	Network Activity	Network connections, traffic flows
OCSF4002	HTTP Activity	Web requests, API calls
OCSF4003	DNS Activity	DNS queries and responses
OCSF4004	DHCP Activity	DHCP lease operations
OCSF4005	RDP Activity	Remote desktop connections
OCSF4006	SMB Activity	File sharing, SMB sessions
OCSF4007	SSH Activity	SSH connections and commands
OCSF4008	FTP Activity	File transfer operations
OCSF4009	Email Activity	Email sending, receiving
OCSF4010	Network File Activity	Network file operations
OCSF4011	Email File Activity	Email attachment handling
OCSF4012	Email URL Activity	Links in emails
OCSF5001	Inventory Info	Asset inventory updates
OCSF5002	Config State	Configuration changes
OCSF6001	Web Resources Activity	Web resource access
OCSF6002	Application Lifecycle	App deployment, updates
OCSF6003	API Activity	API endpoint usage
OCSF6004	Web Resource Access Activity	Web content access

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Files are automatically uploaded to the correct Security Lake partition path based on the current date and configured source, region, and account parameters.

Set format: parquet and the OCSF schema, as required by Amazon Security Lake. The schema determines the structure and field types within the Parquet files. compression names the codec used inside the file.

Templates

The following template variables can be used in file names:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	parquet
{{.Extension}}	File extension	parquet
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	security_logs
{{.TargetType}}	Target type	awssecuritylake
{{.Table}}	Bucket name	security-data
{{.Thread}}	Writer thread index. The sender also appends this automatically when two threads would otherwise produce the same path, so an explicit token is only needed to control WHERE it lands	3
{{.ServiceRoot}}	The service root directory	/opt/vmetric

Multipart Upload

Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage for security data workloads.

Examples

Network Activity

A custom source registered for Network Activity, fed by the VirtualMetric pack:

targets:
  - name: security_lake_network
    type: awssecuritylake
    pipelines:
      - aws_lake
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "aws-security-data-lake-us-east-1-example"
      source: "virtualmetric-network"
      account: "123456789012"
      schema: "OCSF4001"
      format: "parquet"
      name: "network-{{.Timestamp}}.{{.Extension}}"

Two Event Classes

Authentication events and network events go to two targets, one custom source each:

targets:
  - name: security_lake_auth
    type: awssecuritylake
    pipelines:
      - aws_lake
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "aws-security-data-lake-us-east-1-example"
      source: "virtualmetric-authentication"
      account: "123456789012"
      schema: "OCSF3002"
      format: "parquet"
      name: "auth-{{.Year}}{{.Month}}{{.Day}}-{{.Timestamp}}.{{.Extension}}"
  - name: security_lake_network
    type: awssecuritylake
    pipelines:
      - aws_lake
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "aws-security-data-lake-us-east-1-example"
      source: "virtualmetric-network"
      account: "123456789012"
      schema: "OCSF4001"
      format: "parquet"
      name: "network-{{.Year}}{{.Month}}{{.Day}}-{{.Timestamp}}.{{.Extension}}"

High Reliability Configuration

Security findings with a longer timeout, larger parts and smaller files:

targets:
  - name: critical_security_findings
    type: awssecuritylake
    pipelines:
      - aws_lake
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "aws-security-data-lake-us-east-1-example"
      source: "virtualmetric-findings"
      account: "123456789012"
      schema: "OCSF2001"
      format: "parquet"
      compression: "snappy"
      timeout: 60
      part_size: 10
      batch_size: 50000

Managed Credentials

On AWS infrastructure, with the instance role instead of a key pair, and larger files before rotation:

targets:
  - name: security_lake_process
    type: awssecuritylake
    pipelines:
      - aws_lake
    properties:
      region: "us-west-2"
      bucket: "aws-security-data-lake-us-west-2-example"
      source: "virtualmetric-process"
      account: "123456789012"
      schema: "OCSF1007"
      format: "parquet"
      max_size: 268435456

Debug Configuration

Configuration with debugging enabled for testing:

targets:
  - name: debug_security_lake
    type: awssecuritylake
    pipelines:
      - aws_lake
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      bucket: "aws-security-data-lake-us-east-1-example"
      source: "virtualmetric-test"
      account: "123456789012"
      schema: "OCSF4001"
      format: "parquet"
      debug:
        status: true
        dont_send_logs: true

note

All configurations must include the aws_lake pipeline for automatic OCSF normalization, and name the custom source and the OCSF event class it is registered with. The VirtualMetric Amazon Security Lake Pack handles all data transformation automatically.

Troubleshooting

The awssecuritylake target behaves the same way as the Amazon S3 target and reports the same errors for the same causes: refused credentials, a bucket that does not exist, upload timeouts, throttling, and buckets that stay empty. It uses AWS credentials, so everything on that page about key, secret, session, and the fallback to an instance role applies here unchanged. Use the Troubleshooting section of Amazon S3 for the full list of errors, causes, and fixes. See Target Delivery Errors for how Director logs and retries target failures.

Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples there.

What differs for Amazon Security Lake

• Startup probes the bucket. With bucket set, the target makes one HeadBucket call on it, as the Amazon S3 target does, so the credentials need s3:ListBucket on that bucket. A bucket that does not exist, or that the credentials may not reach, fails the start with api error NotFound: Not Found or api error Forbidden: Forbidden after HeadBucket, and nothing is sent until it is corrected. Only a target with no top-level bucket, which an earlier console version saved, probes the account with ListBuckets instead, which needs s3:ListAllMyBuckets on *.

• Parquet is not applied for you in a YAML. Set format: parquet beside the OCSF schema. Left at the defaults, the objects are written as JSON lines, the upload succeeds, the target reports healthy, and Amazon Security Lake typically does not ingest them. A target saved from the console always carries parquet.

• One event class per target. Every event the target receives is written with its one schema. Give each class its own target and custom source, and route each class to its target in the pipeline.

• The partition prefix is part of the object key. Objects are written under ext/{source}/region={region}/accountId={account}/eventDay={date}/, so s3:PutObject has to be allowed on that path and not only on the bucket root.

"invalid AWS Security Lake configuration: source, region and account are required"

[Error] [director] [target-<target id>] [security_lake_network] Failed to reinitialize target "security_lake_network" (attempt 1). Reason: invalid AWS Security Lake configuration: source, region and account are required

Cause: one of source, region, or account is missing. All three build the partition prefix, so the target cannot form an object key without them.

Fix: set all three. account is the AWS account id that Security Lake is configured in, region is the region it runs in, and source is the custom source name registered there. Nothing is sent while this lasts, and it is retried until you fix the cause.

"invalid bucket configuration: schema is required for parquet format"

Cause: format is parquet (or avro) and schema is empty. The file writer needs the event class's schema to build the file.

Fix: set schema to the OCSF event class the custom source is registered with, such as OCSF4001. In the console the OCSF event class field is required, so a target saved there always has one.