Library: Sensitive Data Rules

Standard and Enterprise

Synopsis

A sensitive data rule finds one kind of sensitive value and is named after it, such as "Credit card number". The rules work like sensitive information types in Microsoft Purview, and they are kept in the Library, on the Sensitive Data Rules tab.

When live data, console log or Search captures are masked, they hide the values that the enabled rules find. On the Enterprise edition, a target can also protect these values before it receives them.

Pipelines don't use these rules yet. The Mask and Redact processors work as before.

The platform ships 27 built-in rules, one for each kind of value. You can change, switch off or hide a built-in rule, but you can't delete it. You can also create rules of your own.

Requirements

• Edition - Standard and Enterprise, like the rest of the Library.
• Directors - Version 2.5.1 or later receives the tenant's rules. An older director keeps using its built-in rules. For users who can see directors, the tab names the older directors and director clusters in an alert, Some directors use the built-in rules only, which ends with "Upgrade them to version 2.5.1 or later to use your rules."
• Permissions - The rules use the Library permissions described under Roles:

To do this	You need	Roles that have it
See the tab, a rule's drawer and its page, and test a rule	Library read (LIBRARY_READ)	User, Contributor, Admin, Owner
Create a rule, or clone one	Library create (LIBRARY_CREATE)	Contributor, Admin, Owner
Change a rule, enable or disable it, reset it to platform, or restore it	Library edit (LIBRARY_EDIT)	Contributor, Admin, Owner
Hide a built-in rule, or delete one of your own	Library delete (LIBRARY_DELETE)	Admin, Owner

A rule's Activity logs tab also needs the audit log read permission (AUDIT_READ).

Accessing Sensitive Data Rules

To open the rules:

1. Click the hamburger menu in the top left corner.
2. Select Content Management > Library.
3. Open the Sensitive Data Rules tab.

The Rules Table

The table lists each rule with the following columns:

• Name - The rule's name. Click it to open the drawer.
• Category - What kind of value the rule finds.
• Region - Global, or the country the value belongs to.
• Validator - The check a found value must pass, if any.
• Origin - Platform or Custom. A built-in rule you changed also shows Modified, and one you hid shows Hidden.
• Status - Enabled or Disabled.

Above the table, the search field filters by name, the Origin filter narrows the list to All, Platform, Custom or Hidden rules, the Category filter narrows it by category, and the Create new sensitive data rule button opens the create wizard.

Clicking a rule's name opens a drawer with its main settings. The drawer offers Manage rule and Clone rule, with Reset to platform on a built-in rule you changed and Restore rule on a hidden one.

The row menu depends on the rule:

Rule	Row menu
A built-in rule	Manage rule, Clone rule, Enable rule or Disable rule, Reset to platform (only after you changed the rule), Hide rule
A rule of your own	Manage rule, Clone rule, Enable rule or Disable rule, Delete rule
A hidden rule	See rule details, Restore rule

A user without Library edit sees See rule details instead of Manage rule. Each item appears only for users with its permission.

To act on several rules at once, see Library > Bulk Actions.

Built-in Rules

The 27 built-in rules, grouped by category. The Region column of the console shows a country's name in your language; the table below uses its two-letter code. Each rule's own description is on its page.

Rule	Key	Category	Region	Validator	On at first	Notes
AWS access key ID	aws-access-key-id	Credentials and secrets	Global	None	Yes	Starts with AKIA, ASIA, AROA or AIDA.
AWS secret access key	aws-secret-key	Credentials and secrets	Global	None	Yes	The value after aws_secret_access_key. The name stays readable.
Azure client secret	azure-client-secret	Credentials and secrets	Global	None	Yes	The value after client_secret or clientSecret. The name stays readable.
Azure storage key	azure-storage-account-key	Credentials and secrets	Global	None	Yes	The value after AccountKey= in a connection string. The name stays readable.
Google Cloud API key	gcp-api-key	Credentials and secrets	Global	None	Yes	Starts with AIza.
Password or token assignment	generic-assignment	Credentials and secrets	Global	None	Yes	The value after password=, pwd=, secret=, api_key=, token= and similar names. The name stays readable. It can also catch values that are not secrets.
GitHub token	github-token	Credentials and secrets	Global	None	Yes	Starts with ghp_, gho_, ghu_, ghs_, ghr_ or github_pat_.
HTTP Basic authorization header	http-basic-auth	Credentials and secrets	Global	None	Yes	The encoded user name and password after Authorization: Basic.
HTTP Bearer token	http-bearer-token	Credentials and secrets	Global	None	Yes	The token after Authorization: Bearer.
JSON Web Token	jwt	Credentials and secrets	Global	None	Yes	Three parts separated by dots, the first two starting with ey.
PEM private key	pem-private-key	Credentials and secrets	Global	None	Yes	The whole key block, from its BEGIN line to its END line.
Slack token	slack-token	Credentials and secrets	Global	None	Yes	Starts with xoxa-, xoxb-, xoxp-, xoxr- or xoxs-.
Password in a URL	url-password	Credentials and secrets	Global	None	Yes	The password in a URL such as postgres://user:password@host. The user name and the host stay readable.
Credit card number	credit-card-number	Payment and banking	Global	Luhn checksum	Yes	Visa, Mastercard, American Express and Discover. A masked card keeps its first 6 and last 4 digits.
Credit card number with spaces or dashes	credit-card-separated	Payment and banking	Global	Luhn checksum	No	Card numbers written in groups, such as 4111 1111 1111 1111.
IBAN	iban	Payment and banking	Global	IBAN checksum	Yes	IBANs of 29 countries, the Gulf states among them.
UAE Emirates ID number	ae-emirates-id	National IDs	AE	Luhn checksum	Yes	In the form 784-YYYY-NNNNNNN-C, with or without dashes.
German ID card number	de-id-card	National IDs	DE	German ID card check digit	Yes	Only after a word such as Personalausweis or Ausweisnummer.
Saudi national ID or Iqama number	sa-national-id	National IDs	SA	Luhn checksum	Yes	Only after a word such as national ID or Iqama. The word stays readable.
Turkish ID number (TCKN)	tr-kimlik-no	National IDs	TR	Turkish ID number checksum	Yes	Only after a word such as TCKN or TC Kimlik No. The word stays readable.
UK National Insurance number	uk-nino	National IDs	GB	None	Yes	Written without spaces, such as AB123456C.
US Social Security number	us-ssn	National IDs	US	None	Yes	Written with dashes. A masked number keeps its last 4 digits.
Email address	email	Contact details	Global	None	Yes	Only when the domain has a dot, so a syslog user@host is left alone.
Phone number (international format)	phone-e164	Contact details	Global	None	No	Such as +447700900123. It is off at first because many other numbers look the same.
IPv4 address	ipv4-address	Network	Global	None	No	Off at first because operators need addresses to find problems.
IPv6 address	ipv6-address	Network	Global	None	No	Off at first, for the same reason.
MAC address	mac-address	Network	Global	None	No	Off at first, for the same reason.

Creating a Rule

Click Create new sensitive data rule and complete three steps.

General settings

• Name - Required. Between 3 and 64 characters.
• Description (Optional)
• Category - Required. One of Credentials and secrets, Payment and banking, National IDs, Contact details and Network.
• Region (Optional) - Global by default, or a country.
• Status - Enabled by default. Only enabled rules mask values.

Detection settings

This step has four groups.

• Matching
  • Pattern - Required. A regular expression in RE2 syntax.
  • Ignore case - Off by default.
  • Value group - 0 by default, which protects the whole match. Another number protects only that capture group.
  • Validator - None, Luhn checksum, IBAN checksum, Turkish ID number checksum or German ID card check digit.
• Context
  • Keywords (Optional) - With keywords, the rule masks a value only when one of them comes shortly before it. Case doesn't matter.
  • Keyword distance - Appears once you add a keyword. From 1 to 1000 characters, and 30 by default.
  • Exclusions (Optional) - Exact values that are never masked, such as test card numbers.
• Masking
  • Keep first and Keep last - How many characters stay readable at each end, from 0 to 32. With both at 0, the whole value becomes [masked:<rule key>].
• Test
  • Should match and Should not match - Samples to test the rule with. A rule takes up to 20 samples in all, each up to 4 KB. A rule of your own needs at least one Should match sample. See Testing a Rule.

Review and create

Review the three sections, each with Change to return to it, then click Create rule.

The Rule's Key

The platform makes a rule's key from its name when you create the rule. It joins the lowercase letters and digits with hyphens, such as employee-number. When that key is taken, it adds -2, -3 and so on. The key never changes, even when you rename the rule. Masked values and protected targets show the key.

Cloning a Rule

Clone rule opens the wizard with a copy of the rule's settings and samples, named "Copy of" and the rule's name. The copy is always a rule of your own, with its own key.

Testing a Rule

The console checks the samples as you type:

• In each sample, it highlights the values that the rule masks. It underlines a value that the pattern found but the rule left alone, and gives the reason: the value fails the validator, no keyword comes before it, or it is an exclusion.
• Each sample shows its text After masking and a Result, Passed or Failed. A summary says how many samples behave as expected.
• A pattern error shows under Pattern.

The wizard won't leave Detection settings, and the rule's page won't save, while the pattern has an error or a sample fails. The message names the sample.

The test runs the same code as the platform and the directors, so it shows exactly what a director from version 2.5.1 will do with the rule. An older director doesn't receive the rule, and masks with its built-in rules instead.

Rule Details

A rule's page has three tabs:

• Rule overview - The general settings, with Manage general settings, and Rule information: Key, Origin, Created and Last updated.
• Detection settings - The settings and the test results, with Manage detection settings.
• Activity logs - A searchable record of actions performed on the rule.

The header shows the status and origin tags, and an Actions menu: Clone rule, Reset to platform (after a change), and Hide rule or Delete rule.

A built-in rule's page says This rule comes with the platform. You can change it, disable it or hide it. Reset to platform undoes your changes.

A hidden rule's page is read-only and says A hidden rule masks nothing. Restore it from the Actions menu to use it again.

Changing, Disabling, Hiding and Restoring Rules

• A built-in rule that you change shows Modified. Reset to platform replaces your changes, its on/off switch included, with the platform's current version.
• Disable rule keeps the rule in the list, but it masks nothing until you enable it again.
• Hide rule takes a built-in rule out of the list, and it masks nothing. The Hidden filter lists it. Restore rule brings it back in the platform's current version, without your earlier changes.
• Delete rule removes a rule of your own for good.
• Reset and restore fail when a rule of your own already has the same name. The message says A custom rule already uses this name. Rename that rule, then try again.
• When the platform updates a built-in rule, an untouched copy follows the update. A copy you changed keeps your changes, and a hidden rule stays hidden.

caution

If you switch off or hide every rule, masked captures hide nothing.

How a Rule Finds a Value

The pattern finds a match. The value group picks the part to mask, and the validator checks it. When the rule has keywords, one of them must come before the value. An exclusion is never masked.

When two rules find overlapping values, the value is masked once. The rule with the longer whole match wins. When both matches are the same length, the rule that comes first wins: credentials, then payment, national IDs, contact details and network, and by key within a category.

How a masked value reads:

• A card number becomes 411111******1111.
• A rule that keeps the word before the value gives password=[masked:generic-assignment].
• A rule that keeps nothing, or a value too short to keep that many characters, gives [masked:<rule key>].

Delivery and Limits

• Every change goes to the tenant's directors at once. The platform's own capture masking uses a change within 30 seconds.
• Directors from version 2.5.1 receive the active rules in one file, library/sensitive_data/rules.json. Hidden rules are left out of it.
• Host agents don't receive the rules. Only directors do.
• A standalone director reads the same file from config/library/sensitive_data/rules.json:

{"version": 1, "rules": [
  {"key": "contoso-api-key", "name": "Contoso API key", "category": "credentials",
   "pattern": "\\bctso_[A-Za-z0-9]{32}\\b", "ignoreCase": false, "valueGroup": 0,
   "validator": "", "keywords": [], "keywordDistance": 30,
   "exclusions": ["ctso_00000000000000000000000000000000"],
   "keepFirst": 5, "keepLast": 0, "enabled": true}
]}

Limit	Value
Rules of your own per tenant	500
All of a tenant's rules together	768 KiB

Templates, the REST API and PowerShell

Sensitive data rules can also be managed as code:

• In a deployment template, as the VirtualMetric/library/sensitiveDataRules resource type.
• Through the REST API, as the Library kind sensitivedatarules.
• With PowerShell, through Get-VMetricSensitiveDataRule, New-VMetricSensitiveDataRule, Set-VMetricSensitiveDataRule and Remove-VMetricSensitiveDataRule.

GitOps content sync doesn't carry these rules yet.