Skip to main content

Maester adds ORCA tests

ยท 7 min read
Mike Soule
Maester maintainer + National Director, Sentinel Technologies
Thomas S. Schmidt
Cybersecurity Architect, TDC Erhverv

The Maester module can now dynamically build the necessary files for testing and reporting on all the Office 365 Recommended Configuration Analyzer (ORCA) controls. Providing users with a single report covering many controls that existed before Maester and which are still valuable. ๐Ÿš€

Here's a sneak peek at the some of the checks from ORCA that will be included in the next Maester release.

Subset of Orca results in Maester

What is ORCA?โ€‹

Cam Murray created the Office 365 Recommended Configuration Analyzer (ORCA) PowerShell module to help align tenant configuration with Microsoft's recommended configurations. Many of these settings are available in the configuration analyzer today, but ORCA provided these insights earlier and often in a more concise approach. Building these configuration items as tests in Maester provided an awesome way to build on the core ORCA module value and bring even more context into the configuration state of a tenant.

How to build?โ€‹

The ORCA module utilizes user defined types with enumerations and classes heavily. This is the best approach for an individual module and helps produce well-typed and structured code bases. It also can be challenging when trying to merge code bases dynamically as those types within the module need to exist outside the module.

Git Submodules have their own issues too so definitely avoid those unless the challenges are worth it.

To handle this, PowerShell has this awesome feature called the abstract syntax tree (AST). This feature allows you to parse PowerShell files and interface with them as structured objects. Using the AST the Maester team was able to build a script to parse the ORCA code base and dynamically build the necessary functions, tests, and report details for incorporating with the Maester module.

Similarly and even more elegantly the EIDSCA team was able to incorporate their module as well.

How to contribute?โ€‹

Update-OrcaTests.ps1โ€‹

Located in /build/orca/Update-OrcaTests.ps1 is the build mechanism for each test derived from ORCA.

  1. The latest ORCA module is pulled from orca.git into the subfolder /orca.
  2. Prerequisits are initialized as decribed above and the following files are created in /powershell/internal/orca:
    • Add-IsPresetValue.ps1
    • Get-ORCACollection.ps1
    • Get-PolicyStateInt.ps1
    • Get-PolicyStates.ps1
    • Get-AnyPolicyState.ps1
  3. Each ORCA check located in /build/orca/orca/Checks/ are processed and compiled into Maester tests like so:
    1. A $content variable is prepared and contains the following properties:
      PropertyPurpose
      fileName of the check script
      contentContent of the check script
      nameName of the check
      passText requirement of passing
      failText for remediation
      funcName of the test function
      controlCheck control number
      areaCheck category/area
      descriptionCheck description
      linksCheck related links
    2. Each property listed above are populated using regular expressions on the raw value of the ORCA check content.
    3. $testId is derived from $content.func to fit the Maester test id format.

      If it is not possible to derive it using the expected ORCA test function name format (ORCA.n or ORCA.n.n), where the last number is optional and can be any number of digits, it will skip processing this test. A manual fixed id must be added to $mapping = @{} at line ~187

    4. A pester test instruction file is generated and added to /tests/orca/ for each processed test, that instructs Maester to run the below mentioned function and process the result.
    5. A powershell function file is generated and added to /powershell/public/orca/ that contains the logic of initiating and processing the ORCA check result.
    6. A markdown file is generated and added to /powershell/public/orca/ that is associated with the above mentioned PowerShell file in which the description, remediation action and related links are placed.
  4. ScriptsToProcess is made available for the Maester module manifest.
  5. FunctionsToExport is made available for the Maester module manifest.

When modifying the /build/orca/Update-OrcaTests.ps1 it is important to run it as well in order to build all ORCA test files as described. See How to build?

Test functionโ€‹

Each test function generated for each ORCA check contains various logic:

  1. Skip if not connected to Exchange Online.
  2. Set $SCC to true if connected to Security & Compliance, otherwise set to false.
  3. If Get-ORCACollection has not been run before and stored in $__MtSession.OrcaCache then execute Get-ORCACollection -SCC:$SCC and store it in the cache.
  4. Run the ORCA class New-Object -TypeName <Check> associated with the test upon the ORCA result $obj.Run($Collection) stored in the cache.
  5. First, if the ORCA check should be skipped if it SkipInReport is specified in the check output.
  6. Second, determine if the ORCA check should be skipped by evaluating if $obj.CheckFailed is true, if $obj.Completed is false or if the check output specifies that this is a Security & Compliance check and we are not connected to Security & Compliance.
  7. Determine if the check passed or failed based on $obj.ResultStandard (See ORCA config level for more information).
  8. Determine if we need to expand the results of the check $obj.ExpandResults otherwise add details to Maester and return.
  9. Expand each nested result in $obj.ExpandResults into a markdown table showing the result of each individual result.
  10. Add details to Maester and return.

Markdown fileโ€‹

Each test function has it's associated markdown file that provides Maester with test descriptions. It is comprised of description, remediation action and related links from the ORCA check.

Caveats to be aware ofโ€‹

ORCA result is only processed onceโ€‹

Each test will call Get-ORCACollection only if it has not been run and stored in the cache $__MtSession.OrcaCache before.

SkipInReportโ€‹

Some ORCA checks might have the property SkipInReport set. This must be handled in each test within a try-catch-finally block due to ORCA returning a Continue statement.

Test name should fit Maester formatโ€‹

Every ORCA test should fit into the Maester test name format.

Regular expressionsโ€‹

Regular expressions used to capture the raw content of each ORCA check is carefully crafted and allows us to adapt and overcome differences, such as:

  • Always add punctuation to fail recommendation text if not present to stay consistent between tests.
  • Dynamically populate name of the test if there are more than one ORCA check (The file name contains _) determined by property uniqueness.
  • Replace _ with .
  • Capture entire text of ORCA check importance, even if it varies between apostrophe and qoutes.
  • Remove HTML tags from ORCA check description.

Security & Complianceโ€‹

The command Get-ORCACollection requires the parameter of -SCC in order to include Security & Compliance tests in each test. However, we only set this to true if we are connected to Security & Compliance.

โš ๏ธ WARNING: We require each ORCA check to include the property SCC in it's object output in order for us to skip it in Maester if we are not connected to Security & Compliance. A pull request has been submitted by Thomas S. Schmidt to add this in the upstream module and is awaiting approval. For now it has been manually implemented in /build/orca/orca/ORCA.psm1 @ line 388 and /build/orca/orca/Checks/check-ORCA242.ps1 @ line 29.

ORCA config levelโ€‹

Maester base the result on ORCA config level Standard. The property ResultStandard of the ORCA check output will be either Pass or Informational if the test passed.

Resultsโ€‹

With this build script, the Maester module can now dynamically build the necessary files for testing and reporting on all the ORCA controls. Providing users with a single report covering many controls that existed before Maester and which are still valuable.

ORCA

Acknowledgementsโ€‹

Huge shoutout to the Maester team for all of their awesome contributions in this substantial addition, including:

Contributorsโ€‹