# Writing native tests

Write a Maester check as a native test, the default format from Maester 3.0.

Source: https://maester.dev/docs/writing-tests

# Writing native tests

:::tip[New to writing tests?]
Start with [Your first custom test](./first-custom-test.md), a short walk-through for checking one rule of your
own. This page is the full reference.
:::

An AI coding assistant can write the test for you: [Write a test with AI](./with-ai.md) has the prompt and how to
check the result.

From Maester 3.0 a test is two files: `Test.<ID>.ps1`, which holds one PowerShell function with a
`[MaesterTest(...)]` attribute, and `Test.<ID>.md`, which holds the description and remediation shown in the
report. Every test that ships with Maester is written this way, and your own tests use the same format.

The engine reads the attribute without running your code, checks that the services the test needs are
connected, runs the function, catches any error, and writes the result row. Your function only has to read the
tenant, decide, and return `$true` or `$false`.

Tests written for Maester 2.x with Pester (`*.Tests.ps1`) still run. See
[Pester-format tests](./pester-format-tests.md), which also shows how to convert them with `Convert-MtTest`.

## Quick start

Prefer to click through the choices? The [test builder](/test-builder) explains every attribute and
writes the code for you.

```powershell
# 1. Create custom/Test.CONTOSO.1001.ps1 and custom/Test.CONTOSO.1001.md
New-MtTest -Id CONTOSO.1001 -Title 'Guest invitations are restricted' -Service Graph -Severity High `
    -Product entra -Feature external-identities -Area external-access

# 2. Edit the two files, then check them. No tenant connection is needed for this step.
Get-MtTest -Path ./custom

# 3. Run the test on its own while you write it
Connect-Maester
Invoke-MtTest -Path ./custom/Test.CONTOSO.1001.ps1

# 4. Run it with everything else
Invoke-Maester -Path .
```

`New-MtTest` writes to `./custom` unless you pass `-Path`. It supports `-WhatIf` and refuses to overwrite
existing files unless you pass `-Force`. Only `-Id` and `-Title` are required; `-Product`, `-Feature`, `-Area`,
`-ResourceType`, `-UserImpact`, `-ImplementationCost` and `-Mitre` fill in the attributes of the same name.
`-Setting EmergencyAccessAccounts, ParkedDomains` adds a parameter for each shared setting the test needs (see
[Values set once for many tests](#values-set-once-for-many-tests)).

## Files and layout

Put your tests in a folder of your own, usually `custom/` inside the folder you run Maester from:

```text
maester/
  maester-config.json        (optional: your run configuration)
  custom/
    Test.CONTOSO.1001.ps1
    Test.CONTOSO.1001.md
    suite.json               (optional: name, tags and source for every test in the folder)
    maester-config.json      (optional: an overlay on the root config)
```

`Invoke-Maester -Path ./maester` finds every `Test.*.ps1` file under the path, in any subfolder up to three
levels deep (`./maester/a/b/c/Test.CONTOSO.1001.ps1`). If you leave out `-Path`, Maester looks for custom
tests only in the `custom` folder of the current folder, and reads the config files of the current folder. Without
a `custom` folder only the built-in tests run.

Pester-format tests (`*.Tests.ps1`) must be in a `custom` folder: `-Path` itself, or a `custom` folder under it.
See [Pester-format tests](./pester-format-tests.md).

Maester matches the `custom` folder name without regard to case, so a `Custom` folder created by Maester 2.x keeps
working, and `New-MtTest` writes into it when it exists.

Rules for a test file:

- The file name is `Test.<ID>.ps1`, where `<ID>` is the `Id` in the attribute. The `.md` file has the same
  name with the `.md` extension and sits in the same folder. A custom test without its `.md` file is reported
  as `Error` with reason `InvalidMetadata`.
- The file contains exactly one function with a `[MaesterTest(...)]` attribute. It may contain other functions
  (helpers, or a family's instance source).
- At the top level the file contains only function definitions: no statements, no script-level `param()`
  block. Discovery never executes the file, and this rule keeps it that way.
- The test function has `[CmdletBinding()]`.

### Test IDs

An ID is letters and digits in segments separated by `.` or `-` (and `_` after the first segment). It starts
with a letter, contains at least one digit or separator, and is at most 64 characters. `CONTOSO.1001`,
`CONTOSO-ID-07` and `CT0001` are all valid. IDs are compared case-insensitively.

A test that implements a benchmark requirement is named after it: the suite's prefix, then the requirement's
ID as the benchmark writes it, without a version suffix. That is also its `Id` in the framework file. The prefix ends in a period;
the rest keeps the benchmark's own punctuation, so a reader can search for the ID the benchmark uses.

| Test ID | Prefix | The benchmark's ID |
| --- | --- | --- |
| `CIS.M365.5.2.2.2` | `CIS.M365.` | Recommendation `5.2.2.2` of the CIS Microsoft 365 Foundations Benchmark |
| `CISA.MS.AAD.3.1` | `CISA.` | SCuBA policy `MS.AAD.3.1v1`, without its version suffix |
| `ASD.ISM-1504` | `ASD.` | ISM control `ISM-1504`, written with a hyphen by ASD |
| `DISA.V-270233` | `DISA.` | Rule `V-270233` of a DISA STIG (vulnerability IDs are unique across STIGs) |

Use a prefix of your own. These prefixes belong to the tests that ship with Maester: `MT.`, `CISA.`, `CIS.`,
`EIDSCA.`, `ORCA.`, `AD-`, `AZDO.`, `ASD.`, `DISA.` and `MT1060.`. `MP.` is reserved for Maester Packs, third-party test packs
planned for a later release. A custom test with one of these prefixes still runs, with a
warning. A custom test that uses the exact ID of a built-in test is not run at all, and the built-in runs
instead. To change how a built-in test behaves, set its parameters or severity in the
[run configuration](../configuration/run-configuration.md), or copy it under your own ID and disable the
built-in.

Two native tests with the same ID (or the same function name) both get an `Error` row with reason
`DuplicateId`, and neither runs. When a native test and a Pester-format test share an ID, the native test
runs and the Pester test does not; it is listed under `Selection.Superseded` in the result with `MatchedBy`
`NativeTest`.

## A complete example

`custom/Test.CONTOSO.1002.ps1`:

```powershell
function Test-ContosoDisabledCaPolicyReason {
    [MaesterTest(
        Id           = 'CONTOSO.1002',
        Title        = 'Disabled Conditional Access policies state why they are disabled.',
        Product      = 'entra',
        Feature      = 'conditional-access',
        Area         = 'access-control',
        ResourceType = 'ConditionalAccessPolicy',
        Severity     = 'Medium',
        Service      = 'Graph',
        License      = 'AAD_PREMIUM',
        Tag          = 'Contoso'
    )]
    [CmdletBinding()]
    [OutputType([bool])]
    param(
        # Text that a disabled policy's name must contain.
        [string] $ReasonMarker = 'Disabled:'
    )

    $policies = Get-MtConditionalAccessPolicy
    $withoutReason = @($policies | Where-Object { $_.state -eq 'disabled' -and $_.displayName -notlike "*$ReasonMarker*" })

    if ($withoutReason.Count -eq 0) {
        Add-MtTestResultDetail -Result 'Well done. Every disabled policy states why it is disabled.'
        return $true
    }

    $result = "These disabled policies do not state why they are disabled:`n`n%TestResult%"
    Add-MtTestResultDetail -Result $result -GraphObjects $withoutReason -GraphObjectType ConditionalAccess
    return $false
}
```

`custom/Test.CONTOSO.1002.md`:

```md
Disabled Conditional Access policies must say why they are disabled, in the form `Disabled: <reason>`.

#### Remediation action

1. In the Microsoft Entra admin center, open **Protection** > **Conditional Access** > **Policies**.
2. For each policy listed below, add the reason to its name, or delete the policy.

#### Related links

- [Conditional Access policies](https://entra.microsoft.com/#view/Microsoft_AAD_ConditionalAccess/ConditionalAccessBlade/~/Policies)

<!--- Results --->
%TestResult%
```

There is no connection check, no licence check and no `try`/`catch` in the function. The engine does all three.

## The `[MaesterTest]` attribute

The attribute is read from the file's syntax tree, so its arguments must be constants:

- Named arguments only: `Id = 'CONTOSO.1001'`, never a positional value.
- Values are quoted strings, `$true`, `$false`, or a list in parentheses such as `('T1078.004', 'T1110')`.
- A single string is accepted for a list property: `Service = 'Graph'`.
- A `bool` property can be written as a bare flag: `LongRunning` means `LongRunning = $true`.
- No variables, expressions, `@()` arrays or hashtables.

A value that breaks these rules, an unknown property, or a value outside the allowed list makes the test an
`Error` row with reason `InvalidMetadata`, naming the file and line. The function is never called. If the
message says a property or value "may come from a newer version of Maester", the test was written for a later
release than the one you run.

The attribute answers six questions about a test. Only `Id` and `Title` are required in a custom test; a test
that ships with Maester must also answer the others marked "Built-in tests". The
[attribute rubric](./attribute-rubric.md) says how to choose each value.

| Question | Properties |
| --- | --- |
| Which test is this? | `Id`, `Title` |
| What is it about? | `Product`, `RelatedProduct`, `Feature`, `Area`, `ResourceType` |
| What do I fix first? | `Severity`, `UserImpact`, `ImplementationCost` |
| What does it map to? | `Mitre`, `Control` |
| Can it run here? | `Service`, `License`, `Permission`, `TenantType`, `Cloud`, `Platform` |
| Is it current? | `Lifecycle`, `ReplacedBy` |

| Property | Type | Allowed values | Required | When omitted | What it does |
| --- | --- | --- | --- | --- | --- |
| `Id` | string | ID rules above | Always | | Stable identifier. The file is `Test.<Id>.ps1`. Select the test with `-TestId`. |
| `Title` | string | One line | Always | | The title shown in the report. The result row's `Name` is `<Id>: <Title>`. |
| `Product` | string | A product key from the taxonomy (below), or an alias of one. A custom test may use any one line of up to 40 characters | Built-in tests | Empty | The product where the fix is made. Stored and written to the result as the key. The console progress and the run summary group tests by it; tests without one are shown under Other. |
| `RelatedProduct` | string[] | Product keys or aliases; a custom test may use other names | No | None | Other products the test also checks or depends on. Must not repeat `Product`. `-Product` matches them too. |
| `Feature` | string | A feature key of the test's product. A custom test may use any one line of up to 60 characters | Built-in tests | Empty | The part of the product the test is about, for example `conditional-access`. |
| `Area` | string | An area key. A custom test may use any one line of up to 60 characters | Built-in tests | Empty | The security theme, whatever the product, for example `authentication`. |
| `ResourceType` | string | A resource type key such as `Tenant` or `User`. A custom test may use another word of letters and digits | Built-in tests | Empty | What a failing result points at. |
| `Severity` | string | `Critical`, `High`, `Medium`, `Low`, `Info` | Built-in tests | Empty | Default severity. A `TestSettings` row in the config overrides it. |
| `UserImpact` | string | `Low`, `Medium`, `High` | Built-in tests, unless `Severity` is `Info` | Empty | How much the recommended fix disrupts end users. |
| `ImplementationCost` | string | `Low`, `Medium`, `High` | Built-in tests, unless `Severity` is `Info` | Empty | How much administrator work the recommended fix takes. |
| `Mitre` | string[] | MITRE ATT&CK technique IDs (`T1110`, `T1110.003`), or the single value `None` | Built-in tests | None | The techniques the checked control prevents or detects. `None` cannot be combined with an ID. |
| `Control` | string[] | `<framework-key>:<requirement-id>`, for example `cis-m365:5.1.5.1` or `nist-800-53-r5:IA-2(1)` | No | None | The requirements of frameworks the test implements or supports. The first one that a framework file knows becomes the `Benchmark` of the result. See [Mapping a test to a framework](#mapping-a-test-to-a-framework). |
| `Service` | string[] | A name from the service registry (below), or `None` | Built-in tests | No requirement | Services that must **all** be connected. |
| `License` | string[] | Service plan names, not product names; `&` joins plans | No | No requirement | The tenant needs **any one** element. See [License](#license). |
| `Permission` | string[] | `<Service>:<Kind>:<Name>`; `\|` joins alternatives | No | None | The permissions the test needs. The identity needs **every** element, and **any one** alternative inside an element. Recorded on the result row; not enforced in 3.0. See [Permission](#permission). |
| `TenantType` | string[] | `Workforce`, `External` | No | `Workforce` | Tenant types the test applies to. Recorded; enforced only when you turn it on. |
| `Cloud` | string[] | `Commercial`, `GCC`, `GCCHigh`, `DoD`, `China`, `Bleu`, `Delos`, `GovSG` | No | All clouds | Clouds the test applies to. Recorded; enforced only when you turn it on. |
| `Platform` | string[] | `Windows`, `Linux`, `MacOS` | No | All platforms | Operating systems the test can run on. |
| `Lifecycle` | string | `Preview`, `Stable`, `Deprecated` | No | `Stable` | A `Preview` test is not run unless `-IncludePreview`, any `-Tag`, or its exact ID in `-TestId`. A `Deprecated` test still runs; the result row says so. |
| `ReplacedBy` | string | A test ID | No | Empty | The test that took the place of a `Deprecated` test. |
| `LongRunning` | bool | | No | `$false` | Not run unless `-IncludeLongRunning`, `-Tag LongRunning` or `-Tag CAWhatIf`, or its exact ID in `-TestId`. |
| `Exclusive` | bool | | No | `$false` | The test must not run at the same time as another test. Recorded for a future parallel scheduler; no effect in 3.0. |
| `InstanceSource` | string | Name of a function in the same file | No | One result | Makes the test a family with one result per instance. |
| `Tag` | string[] | Non-empty, no commas (spaces allowed) | No | None | Labels of your own for `-Tag` and `-ExcludeTag`. See [Tags](#tags). |
| `Author` | string[] | GitHub handles | Built-in tests | None | Initial authors, credited on maester.dev. |
| `Contributor` | string[] | GitHub handles | No | None | Further contributors, in order of first contribution. |
| `HelpUrl` | string | An `https://` URL | No | The suite's `HelpUrlTemplate` with `{Id}` replaced; none for a custom test without one | The "learn more" link on the result row. |

`Category` and `Preview` were removed in Maester 3.0. A test that still has one is an `Error` row with reason
`InvalidMetadata` and a message that names the replacement: `Product` and `Feature` instead of `Category`, and
`Lifecycle = 'Preview'` instead of `Preview`. The names `OptionalService` and `TimeoutSeconds` are reserved for
later versions and are rejected today.

### Product, feature, area and resource type

The keys come from one registry,
[`powershell/assets/MaesterTaxonomy.psd1`](https://github.com/maester365/maester/blob/main/powershell/assets/MaesterTaxonomy.psd1):
the products with their features, the areas and the resource types. Product, feature and area keys are lowercase
kebab-case (`entra`, `conditional-access`, `privileged-access`); resource types are PascalCase (`Tenant`,
`ConditionalAccessPolicy`). The display names live in the registry only.

- A key is matched without regard to case and stored in the registry's spelling. A product can also be named by
  an alias: `Product = 'Entra ID'` loads and is stored as `entra`.
- A test that ships with Maester must use keys from the registry, and a `Feature` that belongs to its
  `Product`. Anything else is `InvalidMetadata`.
- A custom test may use any value. A value that matches the registry is stored as the key; any other value is
  kept as you wrote it.

The result row's `Block`, which 2.x readers and the XML test-case names use, is derived: the display name of
`Product` when the registry knows it, else `Product` as written, else `Feature`, else the `Name` of the folder's
`suite.json`, else `Custom`.

### Tags

`Tag` holds labels that you find useful for your own tests. `-Tag` and `-ExcludeTag` match them, together with
the `Tags` of the folder's `suite.json`. The result row's `Tag` lists the `Tag` values of the test and nothing
else.

Tags are not how Maester classifies a test. Product, feature, area, suite and severity are attributes, and
`Invoke-Maester` selects by them with `-Product`, `-Feature`, `-Area`, `-Suite` and `-Severity`. The tests that
ship with Maester carry no tags for those things. The tags they had in Maester 2.x, such as `CA` or `CIS`, still
select the same tests; see [Selection](../configuration/run-configuration.md#selection).

## The Markdown file

`Test.<ID>.md` has no front matter. It holds:

1. The description: what the test checks and why it matters.
2. A `#### Remediation action` section with the steps to fix a failing result, ideally with a direct link to
   the admin portal page.
3. Optionally `#### Related links`.
4. Exactly one `<!--- Results --->` line followed by `%TestResult%`.

Everything above `<!--- Results --->` becomes the description in the report. The text after it is the result
template: `%TestResult%` is replaced with the `-Result` you pass to `Add-MtTestResultDetail`. The `.md` is also
shown for rows the engine produced without running the test (not selected, skipped because a service is not
connected), so the reader always sees what the test is about.

## What the function returns

| The function does | Result | Reason code |
| --- | --- | --- |
| returns `$true` | `Passed` | |
| returns `$false` | `Failed` | |
| calls `Add-MtTestResultDetail -Investigate` | `Investigate`, whatever it returns | |
| calls `Add-MtTestResultDetail -SkippedBecause <code>` | `Skipped`. The call ends the test. | `TestSkipped`, or `NotApplicable` for `-SkippedBecause NotApplicable` |
| calls `Add-MtTestResultDetail -SkippedBecause Error`, or throws any error | `Error` | `TestError` |
| returns `$null` without skipping or `-Investigate` | `Skipped` | `NoResult` |
| returns something that is not a boolean, or more than one value | `Error` | `InvalidReturn` |
| runs longer than the configured timeout | `Error` | `Timeout` |

When several apply, they are evaluated in this order: `-SkippedBecause Error`, `-Investigate`, any other
skip, an uncaught error, the returned value.

Watch for stray output. A command whose output you do not capture (`$list.Add($x)` on an `ArrayList`, for
example) adds a second value to the function's output and turns the result into `InvalidReturn`. Assign such
calls to `$null` or pipe them to `Out-Null`.

### Add-MtTestResultDetail

[`Add-MtTestResultDetail`](../commands/Add-MtTestResultDetail.mdx) works as in 2.x. The parameters you use in a
native test:

- `-Result`: the Markdown result. It replaces `%TestResult%` in the `.md` file.
- `-GraphObjects` and `-GraphObjectType`: a list of objects rendered as a table with links to the admin
  portal, inserted where `%TestResult%` appears in `-Result`. See
  [Formatting test results](./formatting-test-results.md).
- `-Investigate`: the result needs a person to look at it.
- `-SkippedBecause`: skip the test for a reason only the test can see, such as `NotApplicable`,
  `NotSupportedAppPermission` or `Custom` with `-SkippedCustomReason`. Do not use the `NotConnected*` and
  `NotLicensed*` codes for conditions the attribute can declare; use `Service` and `License`.
- `-Description`: replaces the description from the `.md` file. Leave it out to use the `.md`.

`-TestTitle` and `-Severity` are ignored in a single-result native test: the title and severity come from the
attribute and the config. In a family, `-TestTitle` sets the instance's title when the instance object has no
`Title`. `-Severity` is used only when neither the config nor the attribute sets one.

### Embedding another tool's report

A test or a family's instance source that runs another tool can embed that tool's own single-file HTML report in
the Maester report with `Add-MtReportAttachment`:

```powershell
Add-MtReportAttachment -Name 'Zero Trust Assessment' -Path "$outputPath/ZeroTrustAssessmentReport.html"
```

The report lists it near the bottom of the left menu (a pack's test's report goes in that pack's menu instead), and shows a button with that name on its overview page and
on the test's result (on every instance of a family). Each opens the embedded report in the main panel, beside the
Maester navigation, with a link to open it in a new browser tab. The attached report runs in a sandboxed frame
without access to the Maester report or its data. It works offline, so the Maester report stays one
file to share. The file is read and compressed at once, so you can delete the tool's output folder afterwards. The JSON
results file lists the attachment's name and size, not its content, and the HTML report leaves HTML attachments out
when `-RedactUserIdentity` is used.

With `-ContentType application/json`, an attachment is data for a view built into the report, which finds it by
its `Name` (the Zero Trust Assessment pack attaches its Zero Trust plan as `zero-trust-workshop`). It is not shown
as a button, and under `-RedactUserIdentity` it is redacted like the results instead of being left out.

### Errors and skips

Do not wrap the body in `try`/`catch` only to report `-SkippedBecause Error`. The engine runs every test inside
its own `try`, so any terminating error ends the test and becomes an `Error` row with the exception message as
`ReasonDetail` and the error record in `ErrorRecord`. Statement-terminating errors (a failed command, a method
call on `$null`) also end the test, so it cannot carry on with empty data and return `$true` by accident.

You may still use `try`/`catch` for a condition you handle, for example a 403 from Graph that you report as
`-SkippedBecause NotAuthorized`. A `-SkippedBecause` call made inside your own `try` block still skips the
test; the engine recognises the skip when your `catch` passes it to `Add-MtTestResultDetail -SkippedBecause
Error -SkippedError $_`.

Warnings and non-terminating errors written by the test are shown on the console and stored on the result row
in `Diagnostics`.

## Declare what the test needs

Do not check connections or licences in the test. Declare them, and the engine decides before the test runs.
A test whose requirement is not met gets a `Skipped` row with a reason code and its `.md` description, so it
still appears in the report. See [Applicability](../configuration/applicability.md) for the order of the
checks and every reason code.

### Service

`Service` lists the services that must all be connected. Use `None` for a test that needs no connection.
The names come from the service registry:

| Service | Also accepted | Connected by |
| --- | --- | --- |
| `Graph` | | `Connect-Maester` |
| `Azure` | | `Connect-Maester -Service Azure` |
| `ExchangeOnline` | `Exchange` | `Connect-Maester -Service ExchangeOnline` |
| `SecurityCompliance` | `EOP` | `Connect-Maester -Service SecurityCompliance` |
| `SharePointOnline` | `SharePoint` | `Connect-Maester -Service SharePointOnline` |
| `Teams` | | `Connect-Maester -Service Teams` |
| `Dataverse` | | `Connect-Maester -Service Dataverse`; counts as connected when Graph is |
| `AzureDevOps` | | `Connect-Maester -Service AzureDevOps` (or `Connect-ADOPS`) |
| `GitHub` | | `Connect-Maester -Service GitHub` |
| `ActiveDirectory` | `AD` | `Connect-Maester -Service ActiveDirectory` |

A test that is not connected to one of its services is `Skipped` with reason `ServiceNotConnected`.
`ActiveDirectory` is opt-in: its tests are `NotRun` with reason `OptInServiceNotConnected` until you connect it.
A custom test that names a service the registry does not know is `Skipped` with reason `ServiceNotRegistered`.

### License

`License` lists Microsoft **service plan** names: the `servicePlanName` values of the tenant's `subscribedSkus`.
The tenant needs any one element; inside an element, `&` means all of the joined plans.

Use the service plan, not the product. Microsoft lists both in
[Product names and service plan identifiers for licensing](https://learn.microsoft.com/entra/identity/users/licensing-service-plan-reference):
take the name from the **Service plans included** column, such as `AAD_PREMIUM_P2`. The **Product name** and
**String ID** columns, such as Microsoft 365 E5 and `SPE_E5`, are products. A product is a bundle that Microsoft
re-packages over time, and the same feature ships in many of them, so a test that named a product would be
skipped in tenants that have the feature through another one. The [test builder](/test-builder) has a
licence explorer that finds the service plan from a product name, a friendly name or a GUID.

```powershell
License = 'AAD_PREMIUM_P2'                        # Entra ID P2
License = ('AAD_PREMIUM', 'INTUNE_A')             # Entra ID P1 (or better) or Intune
License = 'AAD_PREMIUM_P2&INTUNE_A'               # Entra ID P2 and Intune
```

Maester maps these tokens to the service plan and SKU IDs it has always used to detect each product, so
government, education and suite variants match too:

| Token | Tenant has |
| --- | --- |
| `AAD_PREMIUM` | Entra ID P1 or better (P1, P2 or ID Governance) |
| `AAD_PREMIUM_P2` | Entra ID P2 |
| `Entra_Identity_Governance` | Microsoft Entra ID Governance |
| `AAD_WRKLDID_P1`, `AAD_WRKLDID_P2` | Workload ID P1 or better, Workload ID P2 |
| `ATP_ENTERPRISE` | Defender for Office 365 Plan 1 or Plan 2 |
| `THREAT_INTELLIGENCE` | Defender for Office 365 Plan 2 |
| `EOP_ENTERPRISE` | Exchange Online Protection |
| `EXCHANGE_S_STANDARD` | Any Exchange Online mailbox licence |
| `EXCHANGE_DLP` | Exchange Online DLP (Exchange Online Plan 2, the DLP plan, or Microsoft 365 Business Premium) |
| `M365_ADVANCED_AUDITING` | Microsoft 365 Advanced Auditing |
| `DEFENDER_XDR` | Any Defender XDR workload (Endpoint P2, Office 365 P2, Identity or Cloud Apps) |
| `LOCKBOX_ENTERPRISE` | Customer Lockbox |
| `INTUNE_A` | Microsoft Intune |
| `AGENT_365` | Microsoft Agent 365 (Agent 365 or Microsoft 365 E7) |

`EXCHANGE_DLP` and `DEFENDER_XDR` are Maester's own tokens for a group of plans; every other token in the table
is the service plan's own name.

Any other token is compared literally with the tenant's service plan names, so a custom test can use any plan
name.

For the tests shipped with Maester this is checked. The unit test "Names a service plan in License, not a
product" in `powershell/tests/general/NativeTestAttributes.Tests.ps1` fails when a `License` value is neither a
name from Microsoft's Service plans included column nor a token of `powershell/assets/MaesterLicenseTable.psd1`.
The list of Microsoft's names is `website/src/data/license-service-plans.json`, which a scheduled workflow
refreshes from the Microsoft page every day. Only enabled subscriptions count. When the licences cannot be read (no Graph connection, or no
permission), the licence check is not applied and the test runs.

A test the tenant is not licensed for is `Skipped` with reason `LicenseNotFound`. The message names each licence
by Microsoft's friendly name for the service plan and says what the tenant lacks, for example "This test requires
Microsoft Entra ID P2 and Agent 365. The tenant does not have Agent 365." The names come from the same Microsoft
page, in `powershell/assets/MaesterServicePlanNames.json`, so a custom test's plan is named too. Maester's grouping
tokens (`AAD_PREMIUM`, `ATP_ENTERPRISE`, `EXCHANGE_DLP` and the like) describe every plan they accept instead, and a
plan Microsoft does not list keeps its own name. When `License` has several elements, the message names what is
missing from the element the tenant is closest to.

### Permission

`Permission` says what the identity Maester signs in with must be granted for the test to read its data. It
is recorded on the result row and in the catalog, so the docs and the report can say what to grant; Maester 3.0
does not skip a test on it.

Each permission is written `<Service>:<Kind>:<Name>`:

- **Service** is a name from the service registry (above) that the test also lists in `Service`.
- **Kind** is the kind of grant, from the list below.
- **Name** is the permission as the product spells it, for example `Policy.Read.All` or `View-Only Configuration`.
  It is everything after the second colon, so it may itself contain colons.

The test needs **every** element of the list. Inside an element, `|` joins **alternatives**: any one of them is
enough. Put the least privileged alternative first.

```powershell
Permission = 'Graph:Scope:Policy.Read.All'                                       # one permission
Permission = ('Graph:Scope:User.Read.All', 'Graph:Scope:AuditLog.Read.All')      # both
Permission = 'Graph:Scope:Policy.Read.All | Graph:Scope:Directory.Read.All'      # either
Permission = ('Graph:Scope:Policy.Read.All | Graph:Scope:Directory.Read.All',    # (either of these)
              'Graph:Scope:AuditLog.Read.All')                                   # and this
Permission = 'Graph:Scope:RoleManagement.Read.Directory | Graph:Role:Global Reader'  # a scope or a role
Permission = ('Graph:Scope:Policy.Read.All',                                     # two services: the test
              'ExchangeOnline:Role:View-Only Configuration')                     # lists both in Service
```

This is the other way round from `License`, where the list means "any one" and `&` means "all": a test usually
needs several permissions at once and has an alternative for only some of them. To say "(A and B) or C", write
"(A or C) and (B or C)": `('X:Scope:A | X:Scope:C', 'X:Scope:B | X:Scope:C')`.

| Service | Kind | Example name | What it is |
| --- | --- | --- | --- |
| `Graph` | `Scope` | `Policy.Read.All` | A Microsoft Graph permission that works as a delegated or an application permission |
| | `DelegatedScope` | `User.Read` | A Graph permission the test can use only delegated, with a user signed in |
| | `ApplicationScope` | `User.Read.All` | A Graph permission the test can use only as an application permission |
| | `Role` | `Global Reader` | A Microsoft Entra role, by its display name |
| `Azure` | `Role` | `Reader` | An Azure role; the scope it is assigned at is chosen at setup |
| | `Action` | `Microsoft.Authorization/policyAssignments/read` | An Azure resource provider operation |
| `ExchangeOnline` | `Role` | `View-Only Configuration` | An Exchange management role or role group, or an Entra role that grants one |
| | `Cmdlet` | `Get-OrganizationConfig` | A cmdlet the test runs |
| `SecurityCompliance` | `Role` | `View-Only DLP Compliance Management` | A Purview role or role group, or an Entra role that grants one |
| | `Cmdlet` | `Get-DlpCompliancePolicy` | A cmdlet the test runs |
| `SharePointOnline` | `Scope` | `Sites.FullControl.All` | A SharePoint API permission (not the Graph one of the same name) |
| | `Role` | `SharePoint Administrator` | A Microsoft Entra role |
| `Teams` | `Role` | `Teams Administrator` | A Microsoft Entra role |
| | `Cmdlet` | `Get-CsTeamsMeetingPolicy` | A cmdlet the test runs |
| `Dataverse` | `Role` | `System Administrator` | A Dataverse security role |
| `AzureDevOps` | `Scope` | `vso.project` | An OAuth or personal access token scope |
| | `Role` | `Project Collection Administrators` | A security group or access level |
| `GitHub` | `Permission` | `administration:read` | A GitHub App or fine-grained token permission |
| | `Scope` | `read:org` | A classic personal access token scope |
| | `Role` | `Organization owner` | An organization or enterprise role |
| `ActiveDirectory` | `Ldap` | `user:ReadProperty`, `*.ntSecurityDescriptor:ReadControl`, `Configuration/*:ReadProperty` | An LDAP right on an object class, `[<partition>/]<objectClass>[.<attribute>]:<right>`; the partition is `Configuration` or `Schema`, and the domain when omitted |
| | `Cim` | `root\MicrosoftDNS:MicrosoftDNS_Zone:Read` | Remote CIM (WMI) access, `<namespace>:<class>:<right>` |
| | `File` | `SYSVOL:Read` | A share on the domain controllers, `<share>:<right>` |
| | `Remoting` | `Microsoft.PowerShell` | A PowerShell remoting endpoint |
| | `Group` | `Remote Management Users` | Membership of a group |

The kinds are listed in `powershell/assets/MaesterServiceRegistry.psd1`. Maester writes a registered service
and kind in the registry's spelling and puts one space around each `|`. For a test shipped with Maester the
reader also checks that the service is one the test lists in `Service` and that the kind belongs to it; a custom
test may name a service or kind the registry does not know, such as a pack connector's.

Name the permission the test's code actually uses, the least privileged one that works. Do not list a
permission that a broader one in the same element already includes.

### TenantType and Cloud

`TenantType` (`Workforce` or `External`) and `Cloud` describe where a test is meaningful. Maester detects both
and records them in the result's `TenantContext`, but by default does not skip tests on them. A run turns
enforcement on with `Environment.Enforce` in the [run configuration](../configuration/run-configuration.md).
Declare them now if you know them; a test without `TenantType` is treated as `Workforce`.

### Platform

Declare `Platform = 'Windows'` when the test calls a Windows-only command or module. On other operating
systems it is `Skipped` with reason `PlatformMismatch`. Most tests run everywhere and declare nothing.

## Parameters

A test's `param()` block lists the values a user can tune per tenant: thresholds, switches, objects to exclude.
Values come from the run configuration or from `Invoke-MtTest -Parameter`; a test always has to run with none
set, using its defaults.

```powershell
param(
    # Longest validity period, in days, that a certificate may be issued for.
    [ValidateRange(1, 3650)]
    [int] $MaximumValidityDays = 365,

    # Members of these groups are exempt from this check.
    [MaesterParameter(Kind = 'Entra.Group')]
    [string[]] $ExcludedGroups,

    # How strictly to evaluate.
    [ValidateSet('Strict', 'Lenient')]
    [string] $Mode = 'Strict'
)
```

Rules:

- Types are `int`, `bool`, `switch`, `string` or `string[]`, and `object[]` for a parameter bound to a setting
  that holds a list of objects.
- Names follow the [parameter naming rules](./attribute-rubric.md#parameter-names): `MaximumInactiveDays`,
  `ExcludedGroups`, `Strict`.
- Defaults are constants. No parameter is mandatory.
- Every parameter has a description: a `#` comment on the line directly above it, or `.PARAMETER` help. Only
  the comment line nearest the parameter is read, so keep the comment to one line or use `.PARAMETER`.
  `Get-MtTest` shows the description.
- Give thresholds a `ValidateRange`. A `ValidateSet` makes the parameter a list of choices.
- `Instance` and every name that starts with `Mt` belong to the engine and cannot be set.

Set values in the config:

```json
{
  "TestSettings": [
    { "Id": "CONTOSO.1002", "Parameters": { "ReasonMarker": "Off:" } },
    { "Id": "MT.1198", "Parameters": { "MaximumValidityDays": 180 } }
  ]
}
```

The engine checks each value against the `param()` block before the test runs. A name the test does not have,
a fraction for an `int`, a number for a `string`, a string for a `bool` or `switch`, or a value outside the
`ValidateRange` or `ValidateSet` gives that test an `Error` row with reason `InvalidConfiguration`; the run
carries on. A `string[]` parameter accepts one string or a list. The effective values, and where each came from
(`Default`, `Config`, `Setting` or `Parameter`), are written to the result row's `Parameters`.

### Values set once for many tests

Some values are the same for every test that needs them: the domains that never send email, the emergency access
accounts. An administrator sets these once in `GlobalSettings`, and a test receives one as a parameter bound to
the setting:

```powershell
param(
    # Domains that never send email.
    [MaesterParameter(Setting = 'ParkedDomains')]
    [string[]] $ParkedDomains = @()
)
```

```json
{
  "GlobalSettings": { "ParkedDomains": ["contoso-old.com"] },
  "TestSettings": [ { "Id": "CIS.M365.2.1.9", "Parameters": { "ParkedDomains": [] } } ]
}
```

The most specific value wins: `Invoke-MtTest -Parameter` or `Invoke-Maester -Parameter`, then the test's
`TestSettings` row, then `GlobalSettings`, then the parameter's default. A built-in setting must be registered in
`powershell/assets/MaesterSettings.psd1`, and the parameter's type must match it; the parameter takes the
setting's kind when it names none. A custom test or pack binds its own settings with a `Namespace.Key` name, such
as `Contoso.Regions`.

Maester checks `GlobalSettings` against the registry once, before the run: a value of the wrong type or one the
setting does not allow stops the run with one message, and a key it does not know is reported as a likely typo.
So give the parameter the same type, default and `ValidateSet` as the setting, and pass the value on to the
helpers that need it; a built-in test never calls `Get-MtSetting`. If another test calls yours as a
[shim](#tests-that-another-suite-already-has), it declares the same bound parameter and passes it on.

A test or helper that does call `Get-MtSetting`, as custom tests and packs may, has the name of each setting it
read recorded on its result (`SettingsRead`).

The [test builder](/test-builder) has a step for parameters: it lists the shared settings, writes the bound
parameter, checks each name against the [naming rules](./attribute-rubric.md#parameter-names) as you type, and
shows the `maester-config.json` an administrator would write. `New-MtTest -Setting <name>` writes the same
bound parameter.

Bind a parameter only to a fact about the tenant or a choice the organisation makes once. A value a benchmark
sets stays a parameter of that test. Read [Parameters bound to a setting](./attribute-rubric.md#parameters-bound-to-a-setting)
before adding one.

### Parameter kinds

`[MaesterParameter(Kind = '...')]` says what a string parameter holds, so a user interface can offer a picker
and the engine can check the value's shape:

| Kind | Value |
| --- | --- |
| `Entra.User` | User object ID |
| `Entra.Group` | Group object ID |
| `Entra.ServicePrincipal` | Service principal object ID |
| `Entra.Application` | Application object ID |
| `Entra.ConditionalAccessPolicy` | Conditional Access policy ID |
| `Entra.NamedLocation` | Named location ID |
| `Entra.Domain` | Domain name |
| `Entra.DirectoryRole` | Directory role definition ID |
| `Azure.Subscription` | Subscription ID |
| `Azure.ResourceGroup` | Resource group ID (`/subscriptions/<id>/resourceGroups/<name>`) |
| `Azure.Resource` | Azure resource ID |

A value that does not match the kind's pattern (for example a malformed GUID) is `InvalidConfiguration`. In the
config a value can be the ID itself or an object `{ "Id": "...", "DisplayName": "..." }`; the test always
receives the ID.

## Families: one test, several results

Some checks produce one result per item: per drift folder, per recommendation, per user. Declare
`InstanceSource` with the name of a function in the same file. The engine calls that function when the test
runs (never during discovery) and runs the test once per object it returns, passing the object as `$Instance`.

```powershell
function Get-ContosoAdminUnit {
    [CmdletBinding()]
    param()
    foreach ($unit in Invoke-MtGraphRequest -RelativeUri 'directory/administrativeUnits') {
        [pscustomobject]@{
            Id    = $unit.id                     # required: the instance suffix
            Title = "Administrative unit '$($unit.displayName)' is restricted"   # optional
            Data  = $unit                        # optional: anything the test needs
        }
    }
}

function Test-ContosoAdminUnitRestricted {
    [MaesterTest(
        Id = 'CONTOSO.2001',
        Title = 'Administrative units are restricted.',
        Severity = 'Medium',
        Service = 'Graph',
        InstanceSource = 'Get-ContosoAdminUnit'
    )]
    [CmdletBinding()]
    [OutputType([bool])]
    param(
        # The administrative unit to check, supplied by the engine.
        $Instance
    )

    $restricted = [bool]$Instance.Data.isMemberManagementRestricted
    Add-MtTestResultDetail -Result "Member management restricted: $restricted"
    return $restricted
}
```

The instance object has:

| Property | Required | Meaning |
| --- | --- | --- |
| `Id` | Yes | The suffix. The row's ID is `<parent ID>.<Id>`, for example `CONTOSO.2001.4f6c...`. It must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$` and be unique within the family. |
| `Title` | No | The instance's title. Defaults to `-TestTitle` from the helper, then the attribute's title. |
| `Severity` | No | The instance's severity, used when the config sets none. |
| `Tag` | No | Extra tags written to the instance's row. |
| `Fields` | No | Facts about the instance that a report can group or filter by, as a hashtable of names to strings or arrays of strings, for example `@{ Pillar = 'Identity'; Category = 'Access control' }`. Written to the row's `Fields`; other values are left out. |
| `Product`, `Feature`, `Area`, `RelatedProduct`, `UserImpact`, `ImplementationCost`, `Mitre` | No | What this instance is about, when that differs from its family, with the same keys as the attribute. A family that reads results from another tool, such as the Zero Trust Assessment pack, places each result under its own product this way. A value the taxonomy does not know is left out, and the row keeps the family's. |
| `Control` | No | The requirements this instance implements, as one `<framework-key>:<requirement-id>` value or a list, for example `'cisa-scuba:MS.AAD.3.1'`. They replace the family's `Control`, and the row gets its own `Benchmark` and the requirements' `Related` references, as a test does (see [What the result gets](#what-the-result-gets)). A family that reads another tool's results per policy, such as the ScubaGear pack, maps each result to its requirement this way. A value that is not written `<framework-key>:<id>`, or that names a requirement the framework file does not have, is left out; when none is left, the row keeps the family's `Control` and `Benchmark`. |
| `Data` | No | Anything the test needs. It is not written to the result. |

Each instance row carries `ParentId` (the family's ID) and `InstanceId`. When the family does not produce
instance rows, the result has one row on the parent ID instead:

| What happened | Row on the parent ID |
| --- | --- |
| The family was not selected, was disabled, or a gate failed | The usual `NotRun` or `Skipped` reason |
| The instance source returned nothing | `Skipped` with `NoInstances` |
| The instance source threw, did not finish within the family's timeout, or ended with `exit` | `Error` with `InstanceSourceFailed` |
| An instance `Id` is missing, invalid or repeated | `Error` with `InvalidInstanceId` |

The instance source runs under the same timeout as the test (the family's `TimeoutSeconds`, or the run's
`Execution` timeout). The source gets that long to return, and then each instance gets that long to run.

Selection and configuration understand families:

- `-TestId CONTOSO.2001` or `-TestId 'CONTOSO.2001.*'` selects the whole family.
- `-TestId CONTOSO.2001.abc` runs only that instance; the others get `NotRun` rows with reason `NotSelected`.
- `-ExcludeTestId` and `"Enabled": false` on an instance ID drop that instance after the source has run.
- A `TestSettings` row on an instance ID wins over a row on the parent ID. Its `Parameters` and
  `TimeoutSeconds` apply to that instance only; a parameter it does not name keeps the value of the family's row.
- IDs under a family's parent are never reported as unknown, even if that instance does not exist in this run.

## Settings shared by many tests

A custom test receives a tenant-wide setting from `GlobalSettings` the way a built-in test does: as a parameter
bound to the setting (see [Values set once for many tests](#values-set-once-for-many-tests)). The value then
shows in `Get-MtTest` and in the results, and a `TestSettings` row can override it for that test:

```powershell
param(
    # Emergency access (break glass) accounts and groups.
    [MaesterParameter(Setting = 'EmergencyAccessAccounts')]
    [object[]] $EmergencyAccessAccounts = @(),

    # SharePoint site the check reads.
    [MaesterParameter(Setting = 'Contoso.SiteUrl')]
    [string] $SiteUrl = 'https://contoso.sharepoint.com'
)
```

Name your own settings `Namespace.Key`. A key without a dot is one of Maester's settings, listed in the
[run configuration reference](../configuration/run-configuration.md#globalsettings); any other key without a dot
gives a warning when the run starts. Use a plain parameter, with no setting, when the value belongs to one test.

`Get-MtSetting -Name 'Contoso.SiteUrl' -Default '...'` still reads a setting directly, and is the only way in a
Pester-format test or in a helper that runs before any test. A native test that calls it has the setting's name
recorded on its result (`SettingsRead`), but the value is not shown and cannot be overridden per test.

## What a custom test can call

Each custom test file is loaded into its own private module. It can call every command Maester exports,
including `Invoke-MtGraphRequest`, `Add-MtTestResultDetail`, `Get-MtSetting`, `Get-MtTenantContext`,
`Test-MtConnection`, `Get-MtLicenseInformation`, `Get-MtUser`, `Get-MtConditionalAccessPolicy` and
`Invoke-MtTest`. It cannot call Maester's private functions, and it cannot replace a Maester command.

Functions defined in the same file are visible to the test. Functions in another test file are not. To share
code between files, put it in a helper `.ps1` (one that is not named `Test.*.ps1`) and dot-source it from
inside the test function: `. "$PSScriptRoot/Contoso.Helpers.ps1"`. `$PSScriptRoot` is the test file's folder.

To reuse a built-in check, run it through the engine instead of calling its function:

```powershell
$row = Invoke-MtTest -Id MT.1005
$row.Result   # Passed, Failed, Skipped, ...
```

The built-in check functions are no longer exported in 3.0, and they no longer contain their own connection
checks, so calling one directly is not supported.

## A suite manifest for your folder

A `suite.json` in a test folder sets what is true for every test in it (and its subfolders, unless a nearer
`suite.json` exists):

```json
{
  "Id": "Contoso",
  "Name": "Contoso security baseline",
  "Source": "Custom",
  "Tags": ["Contoso"],
  "HelpUrlTemplate": "https://wiki.contoso.com/maester/{Id}",
  "RequiresMaester": "3.0"
}
```

| Key | Effect |
| --- | --- |
| `Id` | Written to each result row as `Suite`, which `-Suite` selects by. Defaults to `Custom`. |
| `Name` | The `Block` of the result rows of tests that have neither `Product` nor `Feature`. |
| `Source` | Written to each result row as `Source`. Defaults to `Custom`. |
| `Tags` | Matched by `-Tag` and `-ExcludeTag` for every test in the folder. Not written to the result rows. |
| `HelpUrlTemplate` | The `HelpUrl` of tests that do not set one; `{Id}` is replaced. |
| `RequiresMaester` | Tests in the folder need at least this Maester version. On an older version each one is an `Error` row with reason `RequiresNewerMaester`. |
| `NoResultReason` | The skip text shown when a test returns `$null` without calling `Add-MtTestResultDetail`. Defaults to "The test returned no result." |
| `Frameworks` | The frameworks whose catalogue is in the folder. See [Benchmarks and framework files](#benchmarks-and-framework-files). |

A single file can state the same as `RequiresMaester` with
`#Requires -Modules @{ ModuleName = 'Maester'; ModuleVersion = '3.1' }`.

### Benchmarks and framework files

A test says which requirement of a framework it implements in its own attribute, with `Control`:

```powershell
[MaesterTest(
    Id      = 'CONTOSO.BL.1.1',
    Title   = 'Admins sign in with phishing-resistant MFA',
    Control = 'contoso-baseline:1.1'
)]
```

A `Control` value is `<framework-key>:<requirement-id>`. This is the only way a test is mapped to a framework. It
is the same for the CIS, CISA and ASD tests that ship with Maester (`Control = 'cis-m365:5.1.5.1'`,
`Control = 'cisa-scuba:MS.AAD.3.1'`, `Control = 'essential-eight:ISM-1504'`) and for a test of your own.

The frameworks that ship with Maester, with the key to use in `Control`:

| Key | Framework | Framework file | Declared in |
| --- | --- | --- | --- |
| `cis-m365` | CIS Microsoft 365 Foundations Benchmark | [tests/cis/frameworks/cis-m365.json](https://github.com/maester365/maester/blob/main/tests/cis/frameworks/cis-m365.json) | [tests/cis/suite.json](https://github.com/maester365/maester/blob/main/tests/cis/suite.json) |
| `cis-github` | CIS GitHub Benchmark | [tests/cis/frameworks/cis-github.json](https://github.com/maester365/maester/blob/main/tests/cis/frameworks/cis-github.json) | [tests/cis/suite.json](https://github.com/maester365/maester/blob/main/tests/cis/suite.json) |
| `cisa-scuba` | CISA SCuBA Secure Configuration Baselines for Microsoft 365 | [tests/cisa/frameworks/cisa-scuba.json](https://github.com/maester365/maester/blob/main/tests/cisa/frameworks/cisa-scuba.json) | [tests/cisa/suite.json](https://github.com/maester365/maester/blob/main/tests/cisa/suite.json) |
| `essential-eight` | ASD Essential Eight maturity model (each requirement is an ISM control; `Level` ML1 to ML3, `Strategy`) | [tests/asd/frameworks/essential-eight.json](https://github.com/maester365/maester/blob/main/tests/asd/frameworks/essential-eight.json) | [tests/asd/suite.json](https://github.com/maester365/maester/blob/main/tests/asd/suite.json) |
| `ism` | ASD Information security manual (ISM) (`Guideline`, `Section`, `Applicability`) | [tests/asd/frameworks/ism.json](https://github.com/maester365/maester/blob/main/tests/asd/frameworks/ism.json) | [tests/asd/suite.json](https://github.com/maester365/maester/blob/main/tests/asd/suite.json) |
| `disa-stig-entra-id` | DISA Microsoft Entra ID STIG (`Category` CAT I to CAT III, `RuleId`, `StigId`) | [tests/disa/frameworks/disa-stig-entra-id.json](https://github.com/maester365/maester/blob/main/tests/disa/frameworks/disa-stig-entra-id.json) | [tests/disa/suite.json](https://github.com/maester365/maester/blob/main/tests/disa/suite.json) |
| `disa-stig-intune-mdm` | DISA Microsoft Intune MDM Service Desktop and Mobile STIG | [tests/disa/frameworks/disa-stig-intune-mdm.json](https://github.com/maester365/maester/blob/main/tests/disa/frameworks/disa-stig-intune-mdm.json) | [tests/disa/suite.json](https://github.com/maester365/maester/blob/main/tests/disa/suite.json) |
| `disa-stig-intune-desktop` | DISA Microsoft Intune MDM Service Desktop STIG | [tests/disa/frameworks/disa-stig-intune-desktop.json](https://github.com/maester365/maester/blob/main/tests/disa/frameworks/disa-stig-intune-desktop.json) | [tests/disa/suite.json](https://github.com/maester365/maester/blob/main/tests/disa/suite.json) |

Open a file to see the requirement IDs you can reference and the version Maester implements. NIST SP 800-53 has
no framework file; its control IDs are in
[powershell/assets/MaesterNistControls.json](https://github.com/maester365/maester/blob/main/powershell/assets/MaesterNistControls.json)
and are written `nist-800-53-r5:<control>`. The [test builder](/test-builder) looks all of them up.

A framework file is the catalogue of one framework: its name, its version and every requirement, with the facts
about each. A suite that holds a framework file declares it in its `suite.json`:

```json
{
  "Id": "Contoso",
  "Name": "Contoso security baseline",
  "Frameworks": [
    { "Key": "contoso-baseline", "Version": "2.1" }
  ]
}
```

| Key | Content |
| --- | --- |
| `Key` | The key of the framework: lowercase letters, digits, dots and hyphens. It names the framework file, it is the first part of a `Control` value, and it is written to the result. |
| `Version` | The version of the framework. When omitted, the `Version` of the framework file is used. |

The file is `frameworks/<Key>.json`, beside the `suite.json`:

```json
{
  "Framework": "contoso-baseline",
  "Name": "Contoso Security Baseline",
  "Version": "2.1",
  "Url": "https://wiki.contoso.com/baseline",
  "Requirements": [
    { "Id": "1.1", "Level": "L1", "Related": ["nist-800-53-r5:IA-2(1)"] },
    { "Id": "1.2", "Level": "L2" }
  ]
}
```

- `Framework` is the `Key` from the `suite.json`. `Name` is the name of the framework and `Url` is where it is
  published. A `Version` here must be the one the `suite.json` declares.
- A requirement has an `Id`, written as a string. It is the second part of a `Control` value, and it is matched
  exactly, including upper and lower case.
- `Related` lists controls of other frameworks that the requirement relates to, written
  `<framework-key>:<control-id>` with a lowercase key, like a `Control` value.
- Any other key is a fact about the requirement, such as `Level`, `Profile` or `Criticality`. Keep to identifiers
  and facts. Do not copy the text of a benchmark into the file: it is often licensed, and the test's own title
  already says what is checked.
- The file does not list tests. A `Tests` key on a requirement and an `IdPrefix` key in the `suite.json` are
  ignored.

#### What the result gets

Maester looks up each `Control` value of a test, in the order they are written. A value resolves when its key is
a framework that has a file and its ID is a requirement in that file.

- `Benchmark` is built from the first value that resolves: the `Framework`, its `Version`, the `Requirement` and
  the requirement's own facts. When no value resolves, `Benchmark` is `null`.
- `Control` holds the values the test wrote, followed by the `Related` references of every requirement that
  resolved, each once.

`Get-MtTest` shows both. See [the result schema](../result-schema.md#benchmark).

An instance of a [family](#families-one-test-several-results) can set its own `Control`, resolved the same way against the frameworks its
family can name. A report lists every row that names a requirement under it, so when a pack's instance and a
Maester test name the same requirement, both are shown.

A framework is found by its key, whatever suite declares it. A custom test can name a framework that ships with
Maester (`cis-m365`, `cis-github`, `cisa-scuba`, `essential-eight`, `ism`) or one that a suite in your own test folder declares. Two suites
must not declare the same key: Maester warns, names both `suite.json` files and uses the first, and a framework
that ships with Maester comes before one of yours.

A `Control` value whose key no framework file has, such as `nist-800-53-r5:IA-2(1)` or `iso-27001-2022:A.5.17`,
is written to the result as it is. It needs no file.

A problem with a framework never stops a run. Maester writes a warning that names the file and what is wrong:

- The file is missing or is not valid JSON, its `Framework` or `Version` differs from the `suite.json`, a
  requirement has no `Id` or an `Id` that is listed twice, or a `Related` reference is not written
  `<framework-key>:<control-id>`. No test gets a `Benchmark` or `Related` controls from that framework.
- A `Control` names a framework that has a file and a requirement that is not in it, usually a typing error. That
  value gives no `Benchmark`; the other tests keep theirs.

#### Mapping a test to a framework

| The test | What to write | What the result gets |
| --- | --- | --- |
| Implements a requirement of a benchmark that has a framework file (`CIS.M365.5.1.5.1`, `CONTOSO.BL.1.1`) | `Control = 'cis-m365:5.1.5.1'`. If the requirement is new, add it to the framework file. | `Benchmark`, and the reference in `Control` |
| Exists for its own reasons and also supports such a requirement (a Maester test that checks what a CISA policy asks for) | `Control = 'cisa-scuba:MS.AAD.3.1'` | The same |
| Supports a control of a framework that has no file | `Control = 'nist-800-53-r5:IA-2(1)'` | The reference in `Control` |

A test can list several values. Put the requirement the test is written for first: the first value that resolves
becomes the `Benchmark`. Use the keys and requirement IDs of the framework files that ship with Maester so your
results line up with the built-in ones. For a framework that has no file, choose a lowercase key such as
`nist-800-53-r5` or `iso-27001-2022` and keep to it.

#### Why the framework file still exists

A `key:id` string names a requirement. It cannot hold what is true of the requirement whatever test implements
it. The framework file holds that:

- the name and the version of the framework, in one place, so a new version of a benchmark is one edit;
- the facts about each requirement: the CIS level and profiles, the CISA criticality (SHALL or SHOULD), and where
  the policy stands upstream (current, moved or removed);
- the `Related` controls, such as the NIST SP 800-53 controls CISA maps each policy to, so that every test of the
  requirement carries them without repeating them;
- the full list of requirements. A requirement that no test names is a gap you can see, and a `Control` that
  names a requirement the list does not have is a typing error that Maester reports.

For the tests that ship with Maester the module build fails on such an error, and the unit tests check that every
CIS and CISA test names a requirement of its own suite's framework.

A `Benchmark` or a `Control` says that a test supports a requirement. It does not say that a tenant complies with
the framework.

## Tests that another suite already has

Benchmarks overlap. CIS, CISA SCuBA and ASD all ask for multifactor authentication for every user, and Maester
has its own test for it as well. Before you write a test for a benchmark requirement, look for a test that
already reads the same setting, then decide from the benchmark's own sources whether it asks for anything that
test does not check.

1. **Find the tests that read the setting.** `Get-MtTest -Product entra -Feature conditional-access | Format-Table
   Id, Title` lists the candidates; searching `tests/` for the Graph endpoint or the setting name finds the rest.
   Look in every suite, not only the one you are writing for.
2. **Read what the benchmark asks, in its own sources.** The one-line text of a requirement is usually generic.
   The detail that decides between two implementations is in the benchmark's audit or assessment guidance and in
   any reference configuration it publishes: the audit and remediation sections of a CIS benchmark, the SCuBA
   baseline and ScubaGear for CISA, and for ASD the ISM, the Essential Eight assessment process guide and the ASD
   Blueprint for Secure Cloud. Note every concrete value they name: a number of days, a threshold, a list of
   roles, an authentication strength, a rule that must be set to Block.
3. **Choose what to write:**

| What the benchmark asks | What you write |
| --- | --- |
| What an existing test checks | A shim: a test with its own ID, title, `Control` and guide whose body returns the result of the existing test |
| The same check with a different value, which the existing test takes as a parameter | A shim that passes the value, with the benchmark's value as the default of its own parameter |
| Something no test checks, or a rule no parameter covers | A test of its own. Read the data with the helpers in `powershell/internal/services/<service>/`, and move a helper there when another suite has the code you need; don't copy another test's logic |

A shim for ASD's lockout requirement, which asks for what CIS 5.2.3.8 checks with a lower maximum:

```powershell
function Test-MtAsdSmartLockoutThreshold {
    <# ...help that says what ASD asks and which test this runs... #>
    [MaesterTest(
        Id = 'ASD.ISM-1403',
        Title = 'User accounts are protected by lockout mechanisms aligned to a maximum of five failed logon attempts.',
        Product = 'entra',
        Feature = 'password-protection',
        Area = 'authentication',
        ResourceType = 'Tenant',
        Severity = 'Medium',
        UserImpact = 'Low',
        ImplementationCost = 'Low',
        Mitre = ('T1110.001', 'T1110.003'),
        Control = ('ism:ISM-1403', 'nist-800-53-r5:AC-7'),
        Service = 'Graph',
        Author = 'jusso-dev'
    )]
    [CmdletBinding()]
    [OutputType([bool])]
    param(
        # The most failed sign-ins allowed before an account is locked. ASD sets five; CIS allows ten.
        [ValidateRange(1, 50)]
        [int] $MaximumLockoutThreshold = 5
    )

    return Test-MtCisSmartLockoutThreshold -MaximumLockoutThreshold $MaximumLockoutThreshold
}
```

A shim:

- declares the same values as the test it calls for `Product`, `RelatedProduct`, `Feature`, `Area`,
  `ResourceType`, `Severity`, `UserImpact`, `ImplementationCost`, `Mitre`, `Service`, `License`, `Permission`,
  `TenantType`, `Cloud`, `Platform`, `Lifecycle`, `LongRunning` and `Exclusive`. Its own are `Id`, `Title`,
  `Control`, `Author`, `Contributor` and its parameters.
- names the requirement it implements first in `Control`, and that requirement is not added to the `Control` of
  the test it calls, so a run reports it once.
- calls a test that is not a shim itself, and passes only parameters that test declares.
- says in its guide what the benchmark asks and links the test it runs by its ID.

`powershell/tests/general/TestShims.Tests.ps1` finds every built-in test whose body is a single `return` of
another built-in test's function and fails when one of these is not true. The result text comes from the test
that is called, so write that test's messages without naming its benchmark.

When the two benchmarks drift apart, change the shim. If the benchmark of the shim asks for something new,
replace the `return` with a test of its own. If the test it calls changes for its own benchmark, check the shims
that call its function and decide for each whether the change applies to its benchmark too.

A Maester test that also checks what a benchmark asks for needs no shim; it names the requirement in its
`Control`, as [Mapping a test to a framework](#mapping-a-test-to-a-framework) describes. A custom test cannot be a
shim, because the functions of built-in tests are not exported; it runs a built-in test with `Invoke-MtTest -Id`
instead (see [What a custom test can call](#what-a-custom-test-can-call)).

## What a benchmark test evaluates

A CIS test does what the benchmark's **PowerShell audit procedure** does: it reads the same objects and passes on
the same condition. The PowerShell steps are the part of an Automated recommendation that can be automated, and
usually its most precise wording. When the UI steps and the PowerShell steps differ, follow the PowerShell steps.

- **Scope.** When the audit reads every object, the test reads every object, including ones that apply to no
  user. CIS 6.2.1 runs `Get-HostedOutboundSpamFilterPolicy` and checks `AutoForwardingMode` on each policy, so
  the test checks custom policies that have no rule. When the audit reads one object, the test reads only that
  one: CIS 2.1.15 runs `Get-HostedOutboundSpamFilterPolicy -Identity Default`, so the test does not evaluate
  custom policies. Don't narrow a test to the policies users receive, or widen it beyond what the audit reads.
  CIS leaves exceptions to the organization, so a test does not decide them for it.
- **Values.** Use the thresholds the audit names or its script sets, such as the 90% of the extension list in
  CIS 2.1.11. When organizations may need a different value, make it a parameter with the benchmark's value as
  its default.
- **Missing data.** When the object the audit reads is not returned, the test fails or reports an error. It never
  passes because there was nothing to check.

A view of the policies users actually receive belongs in a Maester (MT) test, not in a benchmark test. The same
rule applies to CISA tests, with the SCuBA baseline and ScubaGear's policy in place of the CIS audit procedure.

## Running and inspecting tests

### Get-MtTest

`Get-MtTest` lists tests and their metadata without connecting to a tenant and without running any test code.

```powershell
Get-MtTest -Product entra -Feature conditional-access |
    Format-Table Id, Title, Severity                             # built-in tests
Get-MtTest MT.1198 | Select-Object -ExpandProperty Parameters    # a test's parameters
Get-MtTest -Path ./custom | Where-Object { -not $_.IsValid } |
    Select-Object Id, File, Errors                               # validate your own tests
```

Without `-Path` it returns the tests that ship with Maester. With `-Path` (a file or a folder) it reads and
validates those files; `IsValid` is `$false` and `Errors` lists every problem with its line number. `-Id`
accepts exact IDs and `*` wildcards. `-Product`, `-Feature`, `-Area`, `-Severity` and `-Suite` select by
attribute, as they do on `Invoke-Maester`, and `-Tag` matches what a run with that tag would select.

### Invoke-MtTest

`Invoke-MtTest` runs tests through the engine, with the same gates, parameter checks and result rows as
`Invoke-Maester`, and returns the rows. It writes no report.

```powershell
Invoke-MtTest -Path ./custom/Test.CONTOSO.1002.ps1
Invoke-MtTest -Path ./custom/Test.CONTOSO.1002.ps1 -Parameter @{ ReasonMarker = 'Off:' }
Invoke-MtTest -Path ./custom                       # every native test in the folder
Invoke-MtTest -Id MT.1005, 'CISA.MS.AAD.3.*'       # built-in tests
Invoke-MtTest -Path ./custom -Id CONTOSO.2001.abc  # one instance of a family
```

- `-Id` takes exact IDs or `*` wildcards. Without `-Path` it selects built-in tests; with `-Path` it selects
  among the tests in that file or folder. A family instance ID runs only that instance.
- `-Parameter` is allowed only when exactly one test is selected. Its values win over the config.
- `-Config` takes a path, an object or a list of them, as for `Invoke-Maester`. Without it, a test that calls
  `Invoke-MtTest` during an `Invoke-Maester` run uses the run's configuration; otherwise Maester reads the config
  files found from `-Path` (or the current folder).
- Tests you name run even if they are preview or long-running tests, and `Selection.DefaultAction` is
  ignored. A test disabled in the config is still `NotRun`.

Dot-sourcing a test file at the prompt defines the function but gives it no engine: no service checks, no
error handling and no `.md`. Use `Invoke-MtTest` while you write a test.

### Console output

`Invoke-Maester -Verbosity Normal` prints one line per finished native test with its result, ID, title and
duration, plus the first line of the reason for `Failed` and `Error`. `-Verbosity Detailed` also prints
`Running <ID>` when a test starts. The default, `None`, prints only the summary.

## Next steps

- [Formatting test results](./formatting-test-results.md)
- [Advanced guide](./advanced-concepts.md): `Invoke-MtGraphRequest`, caching and other helpers
- [Applicability and reason codes](../configuration/applicability.md)
- [Run configuration](../configuration/run-configuration.md)
- [Pester-format tests](./pester-format-tests.md)

For inspiration, see the community's [custom Maester tests](https://devclate.github.io/Custom-Maester-Tests/)
by Clayton Tyger (written for 2.x in Pester format; they still run, and `Convert-MtTest` can convert them).