Server maintenance with fan-out pattern in Azure DevOps YAML

Today, I am going to show you an implementation of well-known fan-out pattern in Azure DevOps YAML (AZDO) CI/CD pipeline using Azure DevOps REST API. I will present a simple way to get an one or a few pipelines be executed from one central place called management plain in fan-out mode.

What is fan-out and how it might be useful in CI/CD workflow?

Fanout pattern in serverless architectures means spreading of a message to one or multiple destinations possibly in parallel, and not halting the process that executes the messaging to wait for any response to that message. We will use the idea to delegate execution of specific CI/CD or configuration management to separate processes in AZDO and (or) doing it in parallel.

Below is a schematic view of fan-out preset runner to maintain blue/green deployment process as a management element of release flow process.

So, the solution consist of the next few parts:

• The core element is server maintenance login. The IaC run book, which contains management steps. The runbook is responsible for managing IIS pool using appcmd.exe configuration tool. It should be noticed, the solution might be successfully applied to any kind of server OS using PowerShell, Python or whatever configuration language you prefer.
• Management plain is AZDO pipeline which manage what preset should be executed and on what environment, blue or green production server’s pool.
• Preset runner pipelines. Go-live, Go-offline and Recycle in my case. The set of pipelines which are responsible to maintenance logic execution with predefined set of configuration. It allows us: to avoid manual configuration each time we provide a maintenance process; to save a time to delivery; reduce a risk of misconfiguration and human factor.
• Blue/green server pools are active and passive production legs simultaneously.
• Load balancer is responsible for swaping

Now, let’s go deeper and see how the logic works.

Maintenance logic

As I noticed before, the solution may base on any automation tool you like, any programing language and might be adopted to manage Windows or Linux servers. I am using PowerShell runbooks and appcmd.exe tool to do so. Production environment provisioned and hosted on premises, using NetScaler load balancing solution.

The management plain looks like:

name: "IIS Maintenance-$(Date:yyMMdd).$(Rev:rr)"

trigger: none

pool:
  name: company-dotNET

parameters:
- name: Environment
  values:
  - BLUE
  - GREEN
  default: BLUE

- name: StartApp
  displayName: StartApi (AppPoolStart + OnDemand + Preload=True + WebsiteStart)
  type: boolean
  values:
  - true
  - false
  default: false
- name: StopApp
  displayName: Stop API (AppPoolStop + OnDemand + Preload=True + WebsiteStart)
  type: boolean
  values:
  - true
  - false
  default: false
- name: RecycleApp
  displayName: Recycle API (AppPoolRecycle + OnDemand + Preload=True + WebsiteStart)
  type: boolean
  values:
  - true
  - false
  default: false

variables:
- group: company-project Project Configuration

stages:
- stage: StartApp
  condition: ${{ parameters.StartApp }}
  jobs:
  - job: StartApp
    steps:
    - task: PowerShell@2
      displayName: Start App (AppPoolStart + AlwaysRunning + Preload=True + WebsiteStart)
      inputs:
        targetType: filePath
        filePath: ./Tools/AzdoRestApi/pipelines/run-pipeline.ps1
        arguments: > 
          -AzureDevOpsPAT "$(B64-TOKEN)" 
          -OrganizationName "$(organizationName)" 
          -ProjectName "$(teamProjectName)" 
          -PipelineId 101 #ID of preset runner pipeline in AZDO
          -BranchName "main"
          -TemplateParameters '{ "Environment": "${{ parameters.Environment }}", "AppPoolAction": "start", "AppPoolStartMode": "AlwaysRunning", "Preload": "True", "WebSiteAction": "start"  }'
        showWarnings: true

- stage: StopApp
  condition: ${{ parameters.StopApp }}
  jobs:
  - job: StopApp
    steps:
    - task: PowerShell@2
      displayName: Stop App (AppPoolStop + OnDemand  + Preload=False + WebsiteStop)
      inputs:
        targetType: filePath
        filePath: ./Tools/AzdoRestApi/pipelines/run-pipeline.ps1
        arguments: > 
          -AzureDevOpsPAT "$(B64-TOKEN)" 
          -OrganizationName "$(organizationName)" 
          -ProjectName "$(teamProjectName)" 
          -PipelineId 101 #ID of preset runner pipeline in AZDO
          -BranchName "main"
          -TemplateParameters '{ "Environment": "${{ parameters.Environment }}", "AppPoolAction": "stop", "AppPoolStartMode": "OnDemand", "Preload": "False", "WebSiteAction": "stop" }'
        showWarnings: true

- stage: RecycleApi
  condition: ${{ parameters.RecycleApp }}
  jobs:
  - job: RecycleApp
    steps:
    - task: PowerShell@2
      displayName: Recycle App (AppPoolRecycle , WebsiteRecycle, StartModeNoOps)
      inputs:
        targetType: filePath
        filePath: ./Tools/AzdoRestApi/pipelines/run-pipeline.ps1
        arguments: > 
          -AzureDevOpsPAT "$(B64-TOKEN)" 
          -OrganizationName "$(organizationName)" 
          -ProjectName "$(teamProjectName)" 
          -PipelineId 101 #ID of preset runner pipeline in AZDO 
          -BranchName "main"
          -TemplateParameters '{ "Environment": "${{ parameters.Environment }}", "AppPoolAction": "recycle", "Preload": "True", "WebSiteAction": "recycle" }'
        showWarnings: true

Management plain contains three configuration presents:

• one to bring active slot online setting
• one to stop application and go passive offline
• one to recycle application pool and web sites

Preset runner pipeline, in our case this is separate AZDO pipeline with ID=101, containing run book written in PowerShell. The logic is responsible for reconfiguration application pools and web sites on demand:

parameters:
- name: AppPoolName
  type: string
- name: AppPoolAction
  values:
  - start
  - stop
  - recycle
  default: start
- name: AppPoolStartMode
  values:
  - OnDemand
  - AlwaysRunning
  default: AlwaysRunning
- name: Preload
  type: boolean
  values:
  - true
  - false
  default: true
- name: WebSiteAction
  displayName: WebSite Action
  values:
  - start
  - stop
  - recycle
  default: start

steps:
- powershell: |
    $SystemDirectory = [Environment]::SystemDirectory
    cd $SystemDirectory/inetsrv

    Write-Host "##[debug] Managing Site ${{ parameters.AppPoolName }}" 
    Write-Host "WebSiteAction: ${{ parameters.WebSiteAction }}"
    Write-Host "Preload: ${{ parameters.Preload }}"


    // Web Site management
    If ('${{ parameters.WebSiteAction }}' -eq 'recycle') {
      .\appcmd.exe stop site /site.name:${{ parameters.AppPoolName }}
      .\appcmd.exe start site /site.name:${{ parameters.AppPoolName }}
    }
    Else {
      .\appcmd.exe ${{ parameters.WebSiteAction }} site /site.name:${{ parameters.AppPoolName }}
    }


    // Preload management
    .\appcmd.exe set app "${{ parameters.AppPoolName }}/" /preloadEnabled:${{ parameters.Preload }}
    Write-Host "`n"

    Write-Host "##[debug] Managing AppPool ${{ parameters.AppPoolName }}"
    Write-Host "AppPoolAction: ${{ parameters.AppPoolAction }}"
    Write-Host "AppPoolStartMode: ${{ parameters.AppPoolStartMode }}"
    .\appcmd ${{ parameters.AppPoolAction }} apppool /apppool.name:${{ parameters.AppPoolName }}


    // Start Mode management
    If ("${{parameters.AppPoolStartMode}}" -Eq "AlwaysRunning") {
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} -startMode:AlwaysRunning
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} -autoStart:true
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} /processModel.idleTimeout:00:00:00
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} /processModel.startupTimeLimit:00:10:00
    }
    Else {
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} -startMode:OnDemand
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} -autoStart:false
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} /processModel.idleTimeout:00:02:00
      .\appcmd.exe set apppool /apppool.name:${{ parameters.AppPoolName }} /processModel.startupTimeLimit:00:20:00
    }

    // WebAdministration log
    Import-Module WebAdministration
    Get-ItemProperty IIS:AppPools/${{ parameters.AppPoolName }} | Select *
    Get-ItemProperty IIS:AppPools/${{ parameters.AppPoolName }} | Select-Object autoStart, startMode -ExpandProperty processModel | select autoStart, startMode, startupTimeLimit, idleTimeout
    Get-ItemProperty IIS:Sites/${{ parameters.AppPoolName }} | Select-Object -ExpandProperty applicationDefaults
  displayName: 'Manage AppPool and Site'
  errorActionPreference: 'silentlyContinue'
 

Fan-out execution with Azure DevOps REST API

To execute a pipeline in fan-out, I leverage Azure DevOps REST API:

First, create a JSON post request which contains all of parameters defined above, stages to run, proxy settings and even number of build artifact (if need it).

Second, set up authentication header with appropriate access token generated in AZDO.

And, Invoke-RestMethod:

Invoke-RestMethod -Method POST -ContentType 'application/json' -Headers $header -Body $body -Uri $url;

Below is presented main execution logic:

[CmdletBinding()]

### Usage example
# [string] $AzureDevOpsPAT = <PAT>,
# [string] $OrganizationName = 'company-prod',
# [string] $ProjectName = 'project',
# [string] $PipelineId = 101ex. company.project (https://dev.azure.com/company-prod/projecting/_build?definitionId=101),
# [string] $BranchName ~~ refs/heads/<branch-name>, default refs/heads/main
# [string[]] $StagesToSkip #stagesToSkip ~~ [ "Test", "Dev" ], default []

param(
    [Parameter(Mandatory=$true)]
    [string] $AzureDevOpsPAT,
    [Parameter(Mandatory=$true)]
    [string] $OrganizationName,
    [Parameter(Mandatory=$true)]
    [string] $ProjectName,
    [Parameter(Mandatory=$true)]
    [string] $PipelineId,
    [Parameter(Mandatory=$false)]
    [string] $BranchName,
    [Parameter(Mandatory=$false)]
    [string] $BuildId,
    [Parameter(Mandatory=$false)]
    [string] $Proxy,
    [Parameter(Mandatory=$false)]
    [string[]] $StagesToSkip,
    [Parameter(Mandatory=$false)]
    [string] $TemplateParameters
)    
    
$url = 'https://dev.azure.com/'+ $OrganizationName + '/' + $ProjectName + '/_apis/pipelines/' + $PipelineId + '/runs?api-version=6.0-preview.1';
$header = @{Authorization=("Basic {0}" -f $AzureDevOpsPAT)};

if([string]::IsNullOrWhiteSpace($BranchName))
{
   $BranchName = "refs/heads/main"
}

if([string]::IsNullOrWhiteSpace($StagesToSkip))
{
    $StagesToSkip = '[]'
}

if([string]::IsNullOrWhiteSpace($TemplateParameters))
{
    $TemplateParameters = '{}'
}

$buildAdd =""
if($BuildId -ne "latest")
{    $buildAdd = ',
        "pipelines": {
                "build": {
                    "version": "' + ${BuildId} + '"
                }
         }
    '
}

$body = '{
    "resources": {
        "repositories": {
            "self": {
                "refName": "' + ${BranchName} + '"
            }
        }' + ${buildAdd} + 
    '},
    "stagesToSkip":' + $StagesToSkip + ',
    "templateParameters":' + $templateParameters + '
}';

Write-Output $body

if ([string]::IsNullOrWhiteSpace($Proxy))
{
    $response = Invoke-RestMethod -Method POST -ContentType 'application/json' -Headers $header -Body $body -Uri $url;    
} 
else 
{
    $response = Invoke-RestMethod -Method POST -ContentType 'application/json' -Headers $header -Body $body -Uri $url -Proxy $Proxy; 
}

Write-Output $response.resources.pipelines
$webui = $response._links.web.href
Write-Host "`n`n"
Write-Host "##[section]pipeline web ui link: $webui`n`n"

In the post, I presented a way to create a maintenance process in Azure DevOps using pipelines and REST API. The maintenance process executed leveraging PowerShell scripts and YAML configuration.

I hope, you like the solution, if yes, please, click the link below or follow me on twitter.

Be an ethical, save your privacy!