Declutter Your Azure by automating Post-PR Resource Cleanup

Maintaining a clutter-free Azure environment after Pull Request (PR) merges is crucial for developers and infrastructure personnel using Azure. This step-by-step guide introduces automated resource cleanup using the NuGet tool azure-resources-cleaner to optimize costs and enhance system efficiency efficiently. Automated post-PR has benefits extending beyond tidiness to cost optimization and improved operational efficiency.

Why does this tool exist?

When working with Azure Pipelines, you will occasionally use the reviewApp in your deployment jobs. For Kubernetes environments, this creates a resource backed by a new namespace with the given name. This was introduced in 2019. Unfortunately, Azure does not clean up these resources, and they can delay your pipelines from starting if you have many resources in your environment. I saw a delay of more than 5 minutes with just 40 PRs (1 or 2 per day for a month). These need to be cleaned up manually.

While this works for Kubernetes clusters, other resources such as Azure Static WebApps do not have this feature. See Azure/static-web-apps#497.

Sometimes, you have a whole resource group you need just for a PR and you wish to delete it once the PR is merged or closed/abandoned.

With GitHub, you can automate Azure Static WebApps environments using the official action, but other resource types require manual effort.

The azure-resources-cleaner exists to automate the process.

How does this tool work?

There is extensive documentation here. For this post, I will only pick what is necessary.

You need access to your Azure Subscription. The tool uses managed identity and will work on your machine when you have signed in using the Azure CLI, Visual Studio Code, Visual Studio, etc.

For it to find the resources to clean, they must be named using a specific pattern. Just the resources for the Pull Request. The supported naming formats are listed here.

Say you have a Pull Request #48, Azure Static WebApp environments named review-app-48, ra-48, or ra48 will be deleted. Azure Container Apps with names ending in review-app-48, ra-48, or ra48 will be removed. More on the supported resource types here.

Installation and setup

To kickstart your automated resource cleanup journey, you first need to install the azure-resources-cleaner tool from NuGet. Follow the installation instructions provided.

dotnet install --global azure-resources-cleaner

Once it is installed, you can access the tool using azrc. For example azrc -h or azrc —version.

Execution

For the PR above, the tool will search through the subscriptions that it has access to when you run it.

azrc --pr 48
 
# if the tool is not installed globally
# dotnet azrc --pr 48

A sample output would be:

14:13:19 dbug:  Finding azure subscriptions ...
14:13:21 dbug:  Searching in subscription 'Pay-As-You-Go (1)' ...
14:13:21 dbug:  Searching in subscription 'Pay-As-You-Go (2)' ...
14:13:21 dbug:  Searching in subscription 'Pay-As-You-Go (3)' ...
14:13:21 dbug:  Searching in subscription 'MY-COMPANY' ...
14:13:24 info:  Deleting app 'app-website-ra48' in Environment '/subscriptions/***/resourceGroups/XYZ/providers/Microsoft.App/managedEnvironments/apps-dev'
14:13:43 info:  Deleting app 'app-dashboard-next-ra48' in Environment '/subscriptions/***/resourceGroups/XYZ/providers/Microsoft.App/managedEnvironments/apps-dev'

It is good practice to limit the subscription within which it should operate if you have access to more than one.

azrc --pr 48 --subscription '00000000-0000-0000-0000-000000000000'
# you can use the subscription name instead
# azrc --pr 48 --subscription 'MY-COMPANY'
 
# If the tool is not installed globally
# dotnet azrc --pr 48 --subscription '00000000-0000-0000-0000-000000000000'
# You can use the subscription name instead
# dotnet azrc --pr 48 --subscription 'MY-COMPANY'

Automating with GitHub Workflows

I found this to be the simplest way to get this done because of the awesome GitHub triggers.

name: Remove Review Resources
 
on:
  pull_request:
    types: [closed]
    branches: [main]
  workflow_dispatch:
    inputs:
      pr:
        description: 'Pull request number'
        required: true
        type: number
 
env:
  AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
 
jobs:
  remove:
    if: ${{ github.actor != 'dependabot[bot]' }}
    runs-on: ubuntu-latest
    name: 🗑️ Remove
 
    steps:
      - name: Azure Login
        uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
 
      - name: Remove review resources
        run: |
          dotnet tool install --global azure-resources-cleaner && \
          azrc \
          --pr ${{ inputs.pr || github.event.pull_request.number }} \
          --subscription ${{ env.AZURE_SUBSCRIPTION_ID }}

With this workflow, the Azure CLI task provides the credentials that will then be used by the azure-resources-cleaner task. The workflow_dispatch trigger allows you to manually trigger the cleanup should there be an issue such as expired credentials.

Automating for Azure DevOps

This one gets a bit tricky, to be honest, because there is no way to trigger pipelines when a Pull Request status changes. Instead, for years I have had a ServiceHook that listens to git.pullrequest.updated events then handles it. Thankfully, this is not too hard to set up, and we can use an Azure ContainerApp so that we only pay for what we use.

The setup for the listener app is documented here and that for service hooks and subscriptions is here.

How to deploy a review app, you asked?

For Azure Static WebApps in Azure Repos, here's an example:

variables:
  swaEnvironmentForPR: 'ra$(System.PullRequest.PullRequestId)' # hyphens are not accepted
 
steps:
  - task: AzureStaticWebApp@0
    displayName: Deploy to SWA
    inputs:
      app_location: '/build'
      api_location: ''
      skip_app_build: true
      skip_api_build: true
      azure_static_web_apps_api_token: $SWA_TOKEN
      ${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}:
        deployment_environment: $(swaEnvironmentForPR)
      workingDirectory: '$(Pipeline.Workspace)'
      verbose: true

That one was trivial let us try one with Bicep which is where I get to do lots more. We need a parameter (reviewAppNameSuffix) that we assign for pull requests only.

Your pipeline:

variables:
  reviewAppNameSuffix: '-ra$(System.PullRequest.PullRequestId)'
 
- stage: Deploy
  dependsOn: Build
 
  jobs:
  - deployment: Deploy
    environment: my-app
    pool:
      vmImage: 'ubuntu-latest'
 
    # exclude PR branches for dependabot
    condition: and(succeeded(), not(contains(variables['System.PullRequest.SourceBranch'], 'dependabot/')))
 
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureCLI@2
            displayName: "Deploy Azure Container Apps"
            inputs:
              azureSubscription: $(azureConnection)
              scriptType: bash
              scriptLocation: inlineScript
              ${{ if not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')) }}:
                inlineScript: >
                  az deployment group create
                  --resource-group 'MY-APPS'
                  --template-file $(Pipeline.Workspace)/deploy/apps.bicep
                  --name '$(Build.BuildNumber)'
              ${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}:
                inlineScript: >
                  az deployment group create
                  --resource-group 'MY-APPS'
                  --template-file $(Pipeline.Workspace)/deploy/apps.bicep
                  --name 'pr$(System.PullRequest.PullRequestId)'
                  --parameters "reviewAppNameSuffix=$(reviewAppNameSuffix)"

Unfortunately, since we moved from managing Kubernetes clusters to using Azure ContainerApps, I forgot how to use reviewApp and create dynamic namespaces. If you need help you can reach out to me and I can help you out.

Now the bicep file, we deploy two apps

@description('Location for all resources.')
param location string = resourceGroup().location
 
@description('The suffix used to name the app as a reviewApp')
param reviewAppNameSuffix string = ''
 
param containerImageTag string = 'fb14de2'
 
var isReviewApp = reviewAppNameSuffix != null && !empty(reviewAppNameSuffix)
var acrServerName = 'contoso${environment().suffixes.acrLoginServer}'
var appEnvironmentName = isReviewApp ? 'apps-dev' : 'apps'
 
var definitions = [
  {
    name: 'dashboard'
    minReplicas: 1 // always available
  }
  {
    name: 'website'
    minReplicas: 1 // always available
  }
]
 
// Existing resources
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' existing = { name: 'apps', scope: resourceGroup('MY-COMPANY') }
resource appEnvironment 'Microsoft.App/managedEnvironments@2023-05-01' existing = { name: appEnvironmentName, scope: resourceGroup('MY-COMPANY') }
 
// Main app
resource containerApps 'Microsoft.App/containerApps@2023-05-01' = [for (def, index) in definitions: {
  name: 'apps-${def.name}${reviewAppNameSuffix}'
  location: location
  properties: {
    managedEnvironmentId: appEnvironment.id
    configuration: {
      ingress: {
        external: true
        targetPort: 3000
        traffic: [ { latestRevision: true, weight: 100 } ]
      }
      registries: [ { identity: managedIdentity.id, server: acrServerName } ]
    }
    template: {
      containers: [
        {
          image: '${acrServerName}/apps/web/${def.name}:${containerImageTag}'
          name: def.name
          resources: { cpu: json('0.5'), memory: '1.0Gi' }
        }
      ]
      scale: {
        minReplicas: isReviewApp ? 0 : contains(def, 'minReplicas') ? def.minReplicas : 0
        maxReplicas: isReviewApp ? 1 : 10
      }
    }
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${managedIdentity.id}': {/*ttk bug*/ }
    }
  }
}]
 
output fqdns array = [for (def, index) in definitions: containerApps[index].properties.configuration.ingress.fqdn]

The same bicep can be used but this time with GitHub Workflows:

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 
env:
  reviewAppNameSuffix: ${{ (github.event_name == 'pull_request' && format('-ra{0}', github.event.number)) || '' }}
 
jobs:
  Deploy:
    runs-on: ubuntu-latest
    needs: Build
 
    concurrency:
      group: ${{ github.workflow }}
 
    steps:
      - name: Download Artifact (deploy)
        uses: actions/download-artifact@v4
        with:
          name: deploy
          path: ${{ github.workspace }}/deploy
 
      - name: Azure Login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
 
      - name: Deploy
        uses: azure/arm-deploy@v2
        with:
          subscriptionId: ${{ env.AZURE_SUBSCRIPTION_ID }}
          resourceGroupName: 'MY-COMPANY'
          template: '${{ github.workspace }}/deploy/apps.bicep'
          parameters: 'reviewAppNameSuffix=${{ env.reviewAppNameSuffix }}'
          deploymentName: '${{ env.version }}'
          scope: 'resourcegroup'

Conclusion

With this setup, I have ensured that for all my clients, I can validate pull requests before they are merged. I have an additional PR comment step that allows me to get a preview URL that I can share with my team or client for feedback. Fewer bugs make it to production or result in the "works on my machine" kind of scenario once deployed to production.

I do hope this helps you create and maintain preview environments like Vercel and other providers while saving on costs.

If you like my work, share, and feel free to sponsor me.