Parquet

Apache Parquet is a column-oriented binary storage format optimized for analytical workloads. Originally developed within the Apache Hadoop ecosystem, Parquet provides efficient compression and encoding schemes for large-scale data processing.

The writer targets the Parquet 2.9 format. Any schema conforming to it is written correctly; the tables below describe which parts VirtualMetric actively understands.

Binary Layout

Section	Internal Name	Description	Possible Values / Format
File Header	magic	4-byte magic number identifying Parquet files	ASCII: PAR1 (hex: 50 41 52 31)
Row Group	row_group_metadata	Metadata for each row group	Contains column chunk metadata and statistics
	column_chunk	Data for each column in the row group	Compressed and encoded column data
File Footer	metadata	File-level metadata including schema and row groups	Thrift-encoded metadata structure
	metadata_length	Length of metadata section	4-byte little-endian integer
	magic	Footer magic number	ASCII: PAR1 (hex: 50 41 52 31)

Column Storage Example

Row-based storage (traditional):

id,name,last_name,age
1,John,Buck,35
2,Jane,Doe,27
3,Joe,Dane,42

Column-based Storage (Parquet):

id: [1, 2, 3]
name: [John, Jane, Joe]
last_name: [Buck, Doe, Dane]
age: [35, 27, 42]

JSON Schema Definition

A Parquet schema is declared in VirtualMetric as a single JSON object whose keys are field names and whose values are field definitions. Definitions nest via the fields property to describe records and lists of records. The schema is parsed into a Parquet schema tree and used by the writer when producing .parquet files.

{
  "user_id": { "type": "STRING", "requirement": "required" },
  "score": { "type": "INT", "bitWidth": 32, "signed": true },
  "metadata": {
    "fields": {
      "region": { "type": "STRING" },
      "tier": { "type": "STRING" }
    }
  }
}

The top level is always an object — there is no wrapping fields, name, or schema key.

A record is declared by omitting type and giving fields, as metadata does above. There is no GROUP type: a definition carrying any type at all is treated as a leaf, and an unrecognized value fails the schema with unknown type.

Field Definition Properties

Property	Type	Applies to	Description
type	string	every field	Logical type (see below); required for every leaf field
fields	object	groups, complex lists	Nested fields; presence promotes the field to a record/group
requirement	string	every field	required, optional, or recommended. Default: optional
repeated	boolean	every field	If true, the field is repeated (array-like); equivalent to type: LIST
optional	boolean	every field	Marks the field nullable in the internal mapping; does not change repetition
bitWidth	integer	INT	Integer bit width: 8, 16, 32, or 64. Required when type is INT
signed	boolean	INT	true for signed, false for unsigned. Default: unsigned
logicalType	string	INT64	TIMESTAMP_NANOS, TIMESTAMP_MICROS, or TIMESTAMP_MILLIS to mark a timestamp
unit	string	INT64 ts, TIME	Time unit NANOS, MICROS, or MILLIS; inferred from logicalType if unset
precision	integer	DECIMAL	Total number of digits
scale	integer	DECIMAL	Number of digits after the decimal point
typeLength	integer	FIXED_LEN_BYTE_ARRAY	Length in bytes; must be greater than zero
compression	string	every field	Accepted but not used at parse time (see Compression Codecs)
adjustedToUtc	boolean	timestamp fields	Accepted but not used; reserved for future use

Any property not in this table is ignored on parse.

Repetition is decided in order: if type is LIST or repeated is true, the field is repeated; otherwise if requirement is required, the field is required; otherwise it is optional (the default, covering optional, recommended, and unset).

Supported Types

type	Notes
STRING	UTF-8 string
BOOLEAN	Boolean
INT	Generic integer; requires bitWidth; honors signed. Only bitWidth 32 and 64 survive the validator
INT32	Fixed 32-bit signed integer
INT64	Fixed 64-bit signed integer; with logicalType TIMESTAMP_* becomes a timestamp
INT96	Legacy 96-bit integer (decoded as INT64 internally). Dropped by the validator
FLOAT	32-bit floating point
DOUBLE	64-bit floating point
BYTE_ARRAY	Variable-length binary. Dropped by the validator
FIXED_LEN_BYTE_ARRAY	Fixed-length binary; requires typeLength greater than zero. Dropped by the validator
DECIMAL	Fixed-point decimal; requires precision, scale recommended. Dropped by the validator
DATE	Calendar date (no time). Dropped by the validator
DATETIME	Timestamp at microsecond precision
TIME	Time of day; honors unit, defaults to MILLIS. Dropped by the validator
JSON	Stored as a UTF-8 string, logically tagged as JSON. Dropped by the validator
LIST	Repeated field; see Lists

For DECIMAL, the storage backing is chosen automatically from precision: up to 9 digits uses INT32, up to 18 uses INT64, and beyond 18 uses FIXED_LEN_BYTE_ARRAY.

warning

The schema validator — the stage that runs before encoding, both in the Enforce Schema processor and in the file sender when a target declares a schema — recognizes a narrower set of types than the schema parser accepts. It carries through STRING, BOOLEAN, INT, INT32, INT64, FLOAT, DOUBLE, DATETIME, and the list forms of those scalar types.

Any other declared type — BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY, DECIMAL, DATE, TIME, JSON, INT96, and lists whose element is a record — reaches the validator's default branch, which removes the field from the record. The schema itself is accepted, no error is raised, and the column is simply absent from the output.

INT is only carried at bitWidth 32 or 64. A bitWidth of 8 or 16 produces the internal types INT8 and INT16, which the validator does not recognize either, so those columns are dropped as well.

Keep fields you need on the wire to the recognized types until this is addressed.

Lists

A list takes one of three shapes:

{
  "tags": { "type": "LIST" },
  "scores": {
    "type": "LIST",
    "fields": { "element": { "type": "INT", "bitWidth": 32, "signed": true } }
  },
  "events": {
    "type": "LIST",
    "fields": {
      "ts": { "type": "INT64", "logicalType": "TIMESTAMP_MILLIS" },
      "name": { "type": "STRING" }
    }
  }
}

tags is the shorthand form (a list of strings); scores names a single element type; events is a list of records. Writing "repeated": true on any scalar produces a repeated field without type: LIST.

Requirement Filtering

The Enforce Schema processor's requirement_filter field selects which fields a schema emits: an empty value or all includes every field, while a comma-separated list of required, optional, and recommended includes only matching fields. This lets one schema produce either a required-only projection or a full projection without rewriting it.

Reference Example

{
  "id": { "type": "STRING", "requirement": "required" },
  "received_at": { "type": "INT64", "logicalType": "TIMESTAMP_MICROS", "requirement": "required" },
  "severity": { "type": "INT", "bitWidth": 32, "signed": false },
  "score": { "type": "DOUBLE" },
  "tags": { "type": "LIST" },
  "payload": { "type": "STRING" },
  "user": {
    "fields": {
      "id": { "type": "STRING", "requirement": "required" },
      "email": { "type": "STRING" }
    }
  }
}

Field mapping uses dot notation (user.id, user.email). Schemas in this format can be registered as reusable Library entries and referenced by name from the Enforce Schema and Check Schema processors, or supplied inline.

How Targets Write Parquet

The targets that write Parquet files write them column by column: File, Amazon S3, Azure Blob Storage, Google Cloud Storage and IBM Cloud Object Storage, and the Snowflake, Databricks, Amazon Redshift and Azure Data Explorer targets for the files they stage. Nothing has to be configured to get this writer. These are the differences from the older writer that you can see.

Values in the Files

• 0, false and empty text are kept. The older writer wrote NULL for a field whose value was 0, false or empty text in an optional column. The file now holds the value, so queries that count NULLs, or test for 0 or false, give different results on new files.
• A number too large to store is left out, and the record is kept. The older writer refused the whole record when any field held a number too large for a 64-bit floating-point value, such as 1e400. That field is now left out: an optional column holds NULL, and a required column holds its type's zero value. The rest of the record is written.

Everything else is unchanged: the same columns, in the same order, with the same types and values. A required column with no value holds its type's zero value, as before.

File Format

Values are dictionary-encoded, and data pages are version 1. Every Parquet reader reads these files, and they are usually smaller.

Writer Settings

Field	Effect
data_page_version	V1, or no value, uses the column-by-column writer, unless the schema or the compression keeps the older writer (see below). V2 keeps the older writer, which writes version 2 data pages. Before, the default was V2
buffer_size	The page buffer size of the older writer only. The column-by-column writer does not use it
max_rows_per_rowgroup, compression, metadata	Work as before

Outputs That Keep the Older Writer

The older writer still writes the file when:

• the schema has a list, a group of nested fields or a JSON column, or a field whose name has a dot or starts with _vmetric;
• compression is brotli or lz4;
• data_page_version is V2.

An output that keeps the older writer for one of the first two reasons writes version 2 data pages unless data_page_version is V1.

Encoding Types

Encoding is selected automatically per column by the writer. It is not a configuration option—the table below is a reference for reading the files DataStream produces.

Encoding	Internal Name	Description	Use Case
Plain	PLAIN	Values encoded back to back without compression	Default fallback for all data types
Dictionary	RLE_DICTIONARY	Values replaced with dictionary indices using RLE	Repeated string values, low-cardinality columns
Run Length / Bit-Packing Hybrid	RLE	Combination of bit-packing and run length encoding	Repetition/definition levels, dictionary indices, booleans
Delta Binary Packed	DELTA_BINARY_PACKED	Delta encoding with binary packing for integers	INT32, INT64 with sequential or clustered values
Delta Length Byte Array	DELTA_LENGTH_BYTE_ARRAY	Delta-encoded lengths followed by concatenated data	Variable-length byte arrays
Delta Byte Array	DELTA_BYTE_ARRAY	Incremental/front compression storing prefix lengths	BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY with common prefixes
Byte Stream Split	BYTE_STREAM_SPLIT	Scatters bytes to separate streams for better compression	FLOAT, DOUBLE, INT32, INT64 (added in Parquet 2.8)

Deprecated Encodings

Encoding	Internal Name	Description	Replacement
Plain Dictionary	PLAIN_DICTIONARY	Legacy dictionary encoding in data pages	Use RLE_DICTIONARY in data pages
Bit Packed	BIT_PACKED	Fixed-width bit-packing without padding	Use RLE hybrid encoding

Compression Codecs

Set with the compression field on a target. The following values are recognized:

Value	Codec	Best For
zstandard	ZSTD	Best balance of speed and compression ratio
zstd	ZSTD. An alias for zstandard	Best balance of speed and compression ratio
snappy	SNAPPY	General-purpose, balanced performance
gzip	GZIP	Storage-constrained environments
brotli	BROTLI	High compression ratio needs
lz4	LZ4_RAW	Low-latency applications

Omitting the compression field applies zstandard.

Note that lz4 selects LZ4_RAW, not the deprecated LZ4 codec, and that lz4_raw is not itself a recognized value.

warning

An unrecognized value is not an error. The file is written uncompressed and no warning is emitted, so a misspelled codec is silent. The six values above are the only ones that compress.