Skip to main content

Set up Maester in GitLab

This guide will walk you through setting up Maester in GitLab and automate the running of tests using GitLab Pipelines (jobs).

Set up your Maester tests project (group) in GitLab​

Pre-requisites​

Create a blank new project to always use the latest available public Maester Tests​

  • On the left sidebar, at the top, select 'Create new ()' and 'New project/repository'.
  • Select Create a blank project
    • Repository name: E.g. maester-tests
    • Private: Select this option to keep your tests private

There are many ways to authenticate with Microsoft Entra from GitHub Actions. We recommend using workload identity federation as it is more secure, requires less maintenance and is the easiest to set up.

If you’re unable to use more advanced options like certificates stored in Azure Key Vault, which need an Azure subscription, there’s also guidance available for using client secrets.

  • Workload identity federation (recommended) uses OpenID Connect (OIDC) to authenticate with Microsoft Entra protected resources without using secrets.
  • Client secret uses a secret to authenticate with Microsoft Entra protected resources.

Create an Entra Application​

  • Open Entra admin center > Identity > Applications > App registrations
  • Select New registration
  • Enter a name for the application (e.g. Maester DevOps Account)
  • Select Register

Grant permissions to Microsoft Graph​

  • Open the application you created in the previous step
  • Select API permissions > Add a permission
  • Select Microsoft Graph > Application permissions
  • Search for each of the permissions and check the box next to each permission:
    • DeviceManagementConfiguration.Read.All
    • DeviceManagementManagedDevices.Read.All
    • Directory.Read.All
    • DirectoryRecommendations.Read.All
    • IdentityRiskEvent.Read.All
    • Policy.Read.All
    • Policy.Read.ConditionalAccess
    • PrivilegedAccess.Read.AzureAD
    • Reports.Read.All
    • RoleEligibilitySchedule.Read.Directory
    • RoleManagement.Read.All
    • SecurityIdentitiesSensors.Read.All
    • SecurityIdentitiesHealth.Read.All
    • SharePointTenantSettings.Read.All
    • UserAuthenticationMethod.Read.All
  • Optionally, search for each of the permissions if you want to allow privileged permissions:
    • RoleEligibilitySchedule.ReadWrite.Directory
      • Required for eligible role assignments (Reference)
  • Select Add permissions
  • Select Grant admin consent for [your organization]
  • Select Yes to confirm
(Optional) Grant permissions to Exchange Online

(Optional) Grant permissions to Exchange Online​

The Exchange Online Role Based Access Control (RBAC) implementation utilizes service specific roles that apply to an application and the below configuration allows the authorization chain to the App Registration you created in the previous steps.

The Exchange Online permissions are necessary to support tests that validate Exchange Online configurations, such as the CISA tests.

  • Open the application you created in the previous step
  • Select API permissions > Add a permission
  • Select APIs that my organization uses > search for Office 365 Exchange Online > Application permissions
  • Search for Exchange.ManageAsApp
  • Select Add permissions
  • Select Grant admin consent for [your organization]
  • Select Yes to confirm
  • Connect to the Exchange Online Management tools and use the following to set the appropriate permissions:
New-ServicePrincipal -AppId <Application ID> -ObjectId <Object ID> -DisplayName <Name>
New-ManagementRoleAssignment -Role "View-Only Configuration" -App <DisplayName from previous command>
(Optional) Grant permissions to Teams

(Optional) Grant permissions to Teams​

The Teams Role Based Access Control (RBAC) implementation utilizes service specific roles that apply to an application and the below configuration allows the authorization chain to the App Registration you created in the previous steps.

The Teams permissions are necessary to support tests that validate Teams configurations.

  • Open Roles and administrators
  • Search and select Teams Reader
  • Select Add assigment
  • Select No member selected
  • Search for the name of previously created application
  • Select previously created application and select Select to confirm
  • Select Next to confirm
  • Ensure that Active and Permanently assigned are ticked
  • Enter Justification
  • Select Assign to confirm
(Optional) Grant permissions to Azure

(Optional) Grant permissions to Azure​

The Azure Role Based Access Control (RBAC) implementation utilizes Uniform Resource Names (URN) with a "/" separator for heirarchical scoping. There exists resources within the root (e.g., "/") scope that Microsoft retains strict control over by limiting supported interactions. As a Global Administrator you can elevate access to become authorized for these limited interactions.

The Azure RBAC permissions are necessary to support tests that validate Azure configurations, such as the CISA tests.

The following PowerShell script will enable you, with a Global Administrator role assignment, to:

  • Identify the Service Principal Object ID that will be authorized as a Reader (Enterprise app Object ID)
  • Install the necessary Az module and prompt for connection
  • Elevate your account access to the root scope
  • Create a role assignment for Reader access over the Root Scope
  • Create a role assignment for Reader access over the Entra ID (i.e., aadiam provider)
  • Identify the role assignment authorizing your account access to the root scope
  • Delete the root scope role assignment for your account
$servicePrincipal = "<Object ID of the Entra App>"
$subscription = "<Subscription ID>"
Install-Module Az.Accounts -Force
Install-Module Az.Resources -Force
Connect-AzAccount
#Elevate to root scope access
$elevateAccess = Invoke-AzRestMethod -Path "/providers/Microsoft.Authorization/elevateAccess?api-version=2015-07-01" -Method POST

#Assign permissions to Enterprise App
New-AzRoleAssignment -ObjectId $servicePrincipal -Scope "/" -RoleDefinitionName "Reader" -ObjectType "ServicePrincipal"
New-AzRoleAssignment -ObjectId $servicePrincipal -Scope "/providers/Microsoft.aadiam" -RoleDefinitionName "Reader" -ObjectType "ServicePrincipal"

#Remove root scope access
$assignment = Get-AzRoleAssignment -RoleDefinitionId 18d7d88d-d35e-4fb5-a5c3-7773c20a72d9|?{$_.Scope -eq "/" -and $_.SignInName -eq (Get-AzContext).Account.Id}
$deleteAssignment = Invoke-AzRestMethod -Path "$($assignment.RoleAssignmentId)?api-version=2018-07-01" -Method DELETE

Add federated credentials​

  • Select Certificates & secrets
  • Select Federated credentials, select Add credential
  • For Federated credential scenario, select Other issuer
  • Fill in the following fields
    • Issuer: Your GitLab organization url or standard gitlab url https://gitlab.com
    • Type: Select Explicit subject identifier
    • Value: project_path:<mygroup>/<myproject>:ref_type:branch:ref:<branch> E.g. project_path:maester/maester-tests:ref_type:branch:ref:main
    • Name: Credential name E.g. gitlab-federated-identity
    • Description: Credential name E.g. GitLab service account federated identity
    • Audience: Your GitLab organization url or standard gitlab url https://gitlab.com
  • Select Add

📖 For detailed Azure integration guidance: GitLab provides comprehensive documentation on integrating with Azure services. See the GitLab Azure integration guide for advanced authentication patterns, best practices, and troubleshooting tips.

Create GitLab variables​

  • Open your maester-tests GitLab project and go to Settings
  • Select CI/CD > Variables > CI/CD Variables
  • Add the three secrets listed below by selecting Add variable
  • To look up these values you will need to use the Entra portal, open the application you created earlier and copy the following values from the Overview page:
    • Visibility: Visible, Key: AZURE_TENANT_NAME, Value: The primary domain name of the Entra tenant
    • Visibility: Visible, Key: AZURE_TENANT_ID, Value: The Directory (tenant) ID of the Entra tenant
    • Visibility: Visible, Key: AZURE_CLIENT_ID, Value: The Application (client) ID of the Entra application you created
  • Define which services should be connected using the other variables in order to run the corresponding tests.
    • Visibility: Visible, Key: CONNECTION_EXCHANGE, Value: "true" if you want to connect to the service and execute tests for this service too, "false" if not
    • Visibility: Visible, Key: CONNECTION_IPP, Value: "true" if you want to connect to the service and execute tests for this service too, "false" if not
    • Visibility: Visible, Key: CONNECTION_TEAMS, Value: "true" if you want to connect to the service and execute tests for this service too, "false" if not
  • Save each secret by selecting Add variable at the bottom.

Create .gitlab-ci.yml file (or use pipeline editor)​

stages:
  - test

run_maester_tests_inline:
  stage: test
  image: mcr.microsoft.com/microsoftgraph/powershell:latest
  id_tokens:
    AZURE_FEDERATED_TOKEN:
      aud: https://gitlab.com
  variables:
    TENANTID: $AZURE_TENANT_ID
    CLIENTID: $AZURE_CLIENT_ID
    CONNECTION_EXCHANGE: $CONNECTION_EXCHANGE
    CONNECTION_IPP: $CONNECTION_IPP
    CONNECTION_TEAMS: $CONNECTION_TEAMS

  before_script:
    - mkdir test-results
    - mkdir public-tests
    - pwsh -c 'Write-host "Running in project $env:CI_PROJECT_NAME with results at $env:CI_JOB_URL ($env:CI_JOB_URL)."'
  script:
    - |
      pwsh -Command '
        #region prepare execution
        # Install Maester
        #Install-Module Maester -AllowPrerelease -Force
        Install-Module Maester -Force
        Install-Module Az.Accounts -Force

        # Latest public tests
        Set-Location public-tests
        Install-MaesterTests
        Set-Location ..


        # Declare functions
        function Get-AccessToken {
            param([string]$Scope)

            $TokenBody = @{
                grant_type            = "client_credentials"
                client_id             = $env:AZURE_CLIENT_ID
                client_assertion_type = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"
                client_assertion      = $env:AZURE_FEDERATED_TOKEN.Trim()
                scope                 = $Scope
            }

            $Response = Invoke-RestMethod -Uri "https://login.microsoftonline.com/$env:AZURE_TENANT_ID/oauth2/v2.0/token" -Method POST -Body $TokenBody -ContentType "application/x-www-form-urlencoded"
            return $Response.access_token
        }

        # Configure test results
        $PesterConfiguration = New-PesterConfiguration
        $PesterConfiguration.Output.Verbosity = "None"
        Write-Host "Pester verbosity level set to: $($PesterConfiguration.Output.Verbosity.Value)"

        $MaesterParameters = @{
            Path                 = "public-tests"
            PesterConfiguration  = $PesterConfiguration
            OutputFolder         = "test-results"
            OutputFolderFileName = "test-results"
            PassThru             = $true
        }

        $MaesterParameters.Add("DisableTelemetry", $false )
        Write-Host "Disable pester telemetry set to: $($MaesterParameters.DisableTelemetry)"

        $AdditionalConnections = @{
            Exchange      = [System.Convert]::ToBoolean($env:CONNECTION_EXCHANGE)
            IPP           = [System.Convert]::ToBoolean($env:CONNECTION_IPP)
            Teams         = [System.Convert]::ToBoolean($env:CONNECTION_TEAMS)
        }

        Write-Host "Additional connections configuration:"
        $AdditionalConnections.GetEnumerator() | ForEach-Object {
            Write-Host "  $($_.Key): $($_.Value)"
        }

        #endregion prepare execution
        #region connect to services

        # Connect as service principal
        Write-Host "Connect service principal"
        Connect-AzAccount -ServicePrincipal -Tenant $env:AZURE_TENANT_ID -ApplicationId $env:AZURE_CLIENT_ID -FederatedToken $env:AZURE_FEDERATED_TOKEN | Out-Null

        # Get Graph token and connect to Microsoft Graph
        Write-Host "Connect Graph"
        $graphToken = (Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com").Token
        Connect-MgGraph -AccessToken $graphToken -NoWelcome

        # Connect to Exchange Online and IPP
        if ($AdditionalConnections.Exchange -eq $true -or $AdditionalConnections.IPP -eq $true) {
            # Can be reduced after release from version 3.8.2
            if ($AdditionalConnections.IPP -eq $false) {
                Install-Module -Name ExchangeOnlineManagement -Force
            } else {
                Install-Module -Name ExchangeOnlineManagement -Force -AllowPrerelease #-AllowPrereleas because accesstoken Auth to IPP is only allowed in 3.8.1-Preview1 and newer
            }

            # Get Exchange Online token using Az authentication
            $exchangeToken = Get-AccessToken -Scope "https://outlook.office365.com/.default"

            if ($AdditionalConnections.Exchange -eq $true) {
                Write-Host "Connect Exchange"
                Connect-ExchangeOnline -AccessToken $exchangeToken -Organization $env:AZURE_TENANT_NAME
            }

            if ($AdditionalConnections.IPP -eq $true) {
                Write-Host "Connect IPP"
                Connect-IPPSSession -AccessToken $exchangeToken -Organization $env:AZURE_TENANT_NAME
            }
        }

        # Connect to Microsoft Teams
        if ($AdditionalConnections.Teams -eq $true) {
            Install-Module -Name MicrosoftTeams -Force

            Write-Host "Connect Teams"

            # Get Graph token using federated credentials
            $graphToken = Get-AccessToken -Scope "https://graph.microsoft.com/.default"

            # Get Teams token using federated credentials
            $teamsToken = Get-AccessToken -Scope "48ac35b8-9aa8-4d74-927d-1f4a14a0b239/.default"  # Microsoft Teams Application ID

            # Connect to Microsoft Teams with both tokens
            Connect-MicrosoftTeams -AccessTokens @("$graphToken", "$teamsToken")
        }

        #endregion connect to services

        #region run tests

        # Run Maester tests
        #$results = Invoke-Maester -Path public-tests -PesterConfiguration $PesterConfiguration -OutputFolder test-results -OutputFolderFileName "test-results" -PassThru
        $results = Invoke-Maester @MaesterParameters

        #endregion run tests
        #region end script

        # View summary report
        $results | Format-List Result, FailedCount, PassedCount, SkippedCount, TotalCount, TenantId, TenantName, CurrentVersion, LatestVersion

        # Flag status to GitLab
        if ($results.Result -ne "Passed") {
            Write-Warning "Status = $($results.Result): see Maester Test Report for details."
        }
        #endregion end script
      '
  after_script:
    - pwsh -c 'Write-host "Report can be opened at ($env:CI_JOB_URL/artifacts/external_file/test-results/test-results.html)."'
  artifacts:
    when: on_success
    paths:
      - test-results/
    expire_in: 1 week

  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule" || $CI_PIPELINE_SOURCE == "web"'

Manually running the Maester tests​

To manually run the Maester tests workflow

  • Open your maester-tests GitLab project and go to Build
  • Select Piplines from the left pane
  • Select New pipline on the right top button
  • And again New pipline to run new pipeline

Viewing the test results​

  • Open your maester-tests GitLab project and go to Build
  • Select Artifacts from the left pane
  • Search a artifact to view the results e.g. run_maester_tests_*
  • Select browse on the right to open the folder test-results

Summary view A summary view is not available with GitLab in comparison to GitHub.