Set up Maester in Azure DevOps

This guide will walk you through setting up Maester in Azure DevOps and automate the running of tests using Azure DevOps Pipelines.

Why Azure DevOps?

Azure DevOps is a great way to automate the daily running of Maester tests to monitor your tenant. You can use Azure DevOps to run Maester tests on a schedule, such as daily, and view the results in the Azure DevOps interface.

Azure DevOps comes with a free tier that includes 1,800 minutes of Maester test runs per month (unlimited hours if you use a self-hosted agent).

Azure DevOps has native integration with Microsoft Entra including single sign on, user and group management as well as support for conditional access policies.

Set up the Maester repository in Azure DevOps

Pre-requisites

• If this is your first time using Azure DevOps, you will first need to create an organization.
  • Azure DevOps - Create an organization
    tip

    To enable the free tier, to use a Microsoft-hosted agent, for Azure Pipelines you will need to submit this form https://aka.ms/azpipelines-parallelism-request (it can take a few days before you can use the pipeline.) In the interim you can use a self-hosted agent to get started.

• Create a new project to host your Maester tests and Azure Pipeline.
  • Azure DevOps - Create a project

Import the Maester Tests repository

• Select Repos from the left-hand menu
• Select the Import button in the Import a repository section
• Enter the URL of the Maester repository https://github.com/maester365/maester-tests
• Select Import to import the repository into your Azure DevOps project.

Set up the Azure Pipeline

There are many ways to authenticate with Microsoft Entra in Azure DevOps. We recommend using workload identity federation as it is more secure, requires less maintenance and is the easiest to set up.

If you’re unable to use more advanced options like certificates stored in Azure Key Vault, which need an Azure subscription, there’s also guidance available for using client secrets.

• Workload identity federation (recommended) uses OpenID Connect (OIDC) to authenticate with Microsoft Entra protected resources without using secrets.
• Client secret uses a secret to authenticate with Microsoft Entra protected resources.

Workload identity federation (recommended)

Pre-requisites

• An Azure subscription is required for this method.
  • If you don't have an Azure subscription, you can create one by following Create a Microsoft Customer Agreement subscription or ask your Azure administrator to create one.

Create an empty Azure Resource Group

This empty resource group is required to set up workload identity federation authentication. No Azure resources will be created in this resource group and there are no costs associated with it.

• Open the Azure portal
• Select Create a resource > Resource group
• Enter a name for the resource group (e.g. Maester-Resource-Group)
• Select any region
• Select Review + create > Create

Create a new workload identity federation service connection

• In the Azure DevOps project, go to Project settings > Service connections.
• Select New service connection, and then select Azure Resource Manager.
• Select Workload identity federation (automatic).
• Specify the following parameters:
  • Subscription: Select an existing Azure subscription.
  • Resource Group: Select the resource group created in the previous step. (e.g. Maester Resource Group) Leaving this field empty will grant Contribute access to all resources in the subscription.
  • Service connection name: A name for this connection (e.g. Maester Service Connection)
• Select Save to create the connection.

Grant permissions to Microsoft Graph

• Select the service connection you created in the previous step (e.g. Maester Service Connection)
  • Service connections are listed under Project settings > Service connections.
• Select Manage Service Principal to open the Service Principal in the Entra portal.
• Select API permissions > Add a permission
• Select Microsoft Graph > Application permissions
• Search for each of the permissions and check the box next to each permission:
  • Directory.Read.All
  • Policy.Read.All
  • Reports.Read.All
  • DirectoryRecommendations.Read.All
  • PrivilegedAccess.Read.AzureAD
  • IdentityRiskEvent.Read.All
  • RoleEligibilitySchedule.Read.Directory
• Select Add permissions
• Select Grant admin consent for [your organization]
• Select Yes to confirm

Create Azure Pipeline

• Open your Azure DevOps project
• Select Pipelines > New pipeline
• Select Azure Repos Git as the location of your code
• Select the repository where you imported the Maester tests
• Select Starter pipeline
• Replace the content of the azure-pipelines.yml file with the code below
• Verify the azureSubscription value is set to the service connection you created in the previous step (e.g. Maester Service Connection)
• Select Validate and save > Save
• Select Run to run the pipeline
• Select Job to view the test results

# Maester Daily Tests

trigger:
  - main

schedules:
  - cron: "0 0 * * *"
    displayName: Daily midnight build
    branches:
      include:
        - main

pool:
  vmImage: ubuntu-latest

steps:
  - task: AzurePowerShell@5
    displayName: "Run Maester"
    inputs:
      azureSubscription: "Maester Service Connection"
      pwsh: true
      azurePowerShellVersion: LatestVersion
      ScriptType: InlineScript
      Inline: |
        # Connect to Microsoft Graph
        $accessToken = (Get-AzAccessToken -ResourceTypeName MSGraph).Token | ConvertTo-SecureString -AsPlainText -Force
        Connect-MgGraph $accessToken

        # Install Maester
        Install-Module Maester -Force

        # Configure test results
        $PesterConfiguration = New-PesterConfiguration
        $PesterConfiguration.TestResult.Enabled = $true
        $PesterConfiguration.TestResult.OutputPath = '$(System.DefaultWorkingDirectory)/test-results/test-results.xml'

        # Run Maester tests
        Invoke-Maester -Path $(System.DefaultWorkingDirectory)/tests/Maester/ -PesterConfiguration $PesterConfiguration -OutputFolder '$(System.DefaultWorkingDirectory)/test-results'
  - publish: $(System.DefaultWorkingDirectory)/test-results
    displayName: Publish Maester Html Report
    artifact: TestResults
  - task: PublishTestResults@2
    displayName: Publish Pester Test Results
    inputs:
      testResultsFormat: "NUnit"
      testResultsFiles: "**/test-results.xml"
      failTaskOnFailedTests: true

Client secret

Create an Entra Application

• Open Entra admin center > Identity > Applications > App registrations
  • Tip: enappreg.cmd.ms is a shortcut to the App registrations page.
• Select New registration
• Enter a name for the application (e.g. Maester DevOps Account)
• Select Register

Grant permissions to Microsoft Graph

• Open the application you created in the previous step
• Select API permissions > Add a permission
• Select Microsoft Graph > Application permissions
• Search for each of the permissions and check the box next to each permission:
  • Directory.Read.All
  • Policy.Read.All
  • Reports.Read.All
  • DirectoryRecommendations.Read.All
  • PrivilegedAccess.Read.AzureAD
  • IdentityRiskEvent.Read.All
  • RoleEligibilitySchedule.Read.Directory
• Select Add permissions
• Select Grant admin consent for [your organization]
• Select Yes to confirm

Create a client secret

• Select Certificates & secrets > Client secrets > New client secret
• Enter a description for the secret (e.g. Maester DevOps Secret)
• Select Add
• Copy the value of the secret, we will use this value in the Azure Pipeline

Create Azure Pipeline

• Open your Azure DevOps project
• Select Pipelines > New pipeline
• Select Azure Repos Git as the location of your code
• Select the repository where you imported the Maester tests
• Select Starter pipeline
• Select Variable to open the variables editor and add the following variables.
• In the Entra portal, open the application you created earlier and copy the following values from the Overview page:
  • Name: TENANTID, Value: The Directory (tenant) ID of the Entra tenant
  • Name: CLIENTID, Value: The Application (client) ID of the Entra application you created
  • Name: CLIENTSECRET, Value: The client secret you copied in the previous step
    • Important: Tick the Keep this value secret checkbox
• Replace the content of the azure-pipelines.yml file with the code below
• Select Validate and save > Save
• Select Run to run the pipeline
• Select Job to view the test results

# Maester Daily Tests

trigger:
  - main

schedules:
  - cron: "0 0 * * *"
    displayName: Daily midnight build
    branches:
      include:
        - main

pool:
  vmImage: ubuntu-latest

steps:
  - pwsh: |
      # Connect to Microsoft Graph
      $clientSecret = ConvertTo-SecureString -AsPlainText $env:PS_ClientSecret -Force
      [pscredential]$clientSecretCredential = New-Object System.Management.Automation.PSCredential($env:CLIENTID, $clientSecret)
      Connect-MgGraph -TenantId $env:TENANTID -ClientSecretCredential $clientSecretCredential

      # Install Maester
      Install-Module Maester -Force

      # Configure test results
      $PesterConfiguration = New-PesterConfiguration
      $PesterConfiguration.TestResult.Enabled = $true
      $PesterConfiguration.TestResult.OutputPath = '$(System.DefaultWorkingDirectory)/test-results/test-results.xml'

      # Run Maester tests
      Invoke-Maester -Path $(System.DefaultWorkingDirectory)/tests/Maester/ -PesterConfiguration $PesterConfiguration -OutputFolder '$(System.DefaultWorkingDirectory)/test-results'
    env:
      PS_ClientSecret: $(CLIENTSECRET)
    continueOnError: true
    displayName: Run Maester Tests
  - publish: $(System.DefaultWorkingDirectory)/test-results
    artifact: TestResults
    displayName: Publish Maester Html Report
  - task: PublishTestResults@2
    inputs:
      testResultsFormat: "NUnit"
      testResultsFiles: "**/test-results.xml"
    displayName: Publish Pester Test Results

Viewing test results

• Select Pipelines > Runs to view the status of the pipeline
• Select on a run to view the test results

Summary view

The summary view shows the status of the pipeline run, the duration, and the number of tests that passed, failed, and were skipped.

Maester report

The Maester report can be downloaded and viewed by selecting the Published artifact.

Tests view

The Tests tab shows a detailed view of each test, including the test name, duration, and status.

Logs view

In the Summary tab select on any of the errors to view the raw logs from Maester.

Keeping your Maester tests up to date

The Maester team will add new tests over time. To get the latest updates, use the commands below to update your Azure repository with the latest tests.

• Clone your fork of the maester-tests from Azure DevOps to your local machine. See Clone an existing Git repo.
• Update the Maester PowerShell module to the latest version and load it.
• Change to the maester-tests\tests directory.
• Run Update-MaesterTests.

cd maester-tests\tests

Update-Module Maester -Force
Import-Module Maester
Update-MaesterTests