Nowadays a hybrid approach in building infrastructure is become a standard which commonly used. We are placing more and more elements of infrastructure, to a cloud. Like automation, CI/CD environment and web servers. No need to host and maintain on-premise integration servers any more. But having in-place servers are still critical for many companies. Saving critical applications and data in own data center is important for compliance and security reasons. This is why it becomes vital having simple and reproducible approach to delivery code to end customer. And very often, it is more important to expose new feature for internal testing first as soon as it was developed. I would like to present an architecture approach for automated feature test environment based on hybrid Azure DevOps Ci/CD servers infrastructure and on-premise Windows-based application servers.
First, starting from definition of the problem. A simple diagram below:
The essentials parts here are:
- continuous integration and continuous delivery systems are hosted on Azure DevOps. It contains Azure Repos with source code, build agent pool hosted in Azure and Azure Artifact feed to safely store application package and share them across different feature test environments;
- and on premise application Windows server with IIS application pool installed. The server will host a number of application slots to be used by testers checking specific feature at any time they wish to.
The idea under the picture is following. A developer working on feature branch commits a pull request with latest changes. It triggers simple CI pipeline which basically should contains only restoration process, building source code and unit testing. These are everything we need and the process does not take a lot of time. Why is time so important here? It is because of the number of parallel features developed may vary depends on project scale and might be even a few commits per a few minutes. So, we do not want to stuck process of code delivery in push hours.
Build process
So, here is a build pipeline in YAML:
name: "$(Build.BuildId)$(rev:.r)"
parameters:
- name: fastBuild
displayName: Skip Unit Tests
type: boolean
values:
- true
- false
default: false
- name: buildConfiguration
displayName: Build Configuration
type: string
values:
- 'Release'
- 'Debug'
default: 'Release'
- name: appName
displayName: Application Name
type: string
default: $(app-name)
- name: solutionName
displayName: Solution Name
type: string
default: $(solution-name)
- name: testRunnerConfiguration
displayName: Test Runner Configuration
type: string
values:
- 'DotNetCore'
- 'PowerShell'
- 'VSTest'
default: 'PowerShell'
- name: runInParallel
displayName: Run Tests Projects In Parallel
type: boolean
values:
- true
- false
default: true
variables:
- group: Pipeline Secrets
trigger: none
pool:
name: Hosted Windows 2019 with VS2019
jobs:
- job: CI_ON_PullRequest
displayName: CI pipeline on Pull Request
steps:
##########################################
### Preparing and Build solution
##########################################
- task: UseDotNet@2
displayName: 'Use .NET Core sdk '
inputs:
useGlobalJson: true
performMultiLevelLookup: true
- task: NuGetAuthenticate@0
displayName: 'NuGet Authenticate'
- task: DotNetCoreCLI@2
displayName: Build
inputs:
command: 'build'
projects: |
**/${{ parameters.solutionName }}.sln
arguments: '-c ${{ parameters.buildConfiguration }} /p:DebugType=embedded /p:InformationalVersion=$(Build.BuildNumber) /p:DefineConstants=${{ variables.buildConstants }} /p:NuGetRestorePackages=false'
##########################################
### Tests
##########################################
- ${{ if eq(parameters.testRunnerConfiguration, 'DotNetCore') }}:
- template: ${{ variables.templateFolder }}/dotnet-core-test-runner.yml
parameters:
fastBuild: ${{ parameters.fastBuild}}
buildConfiguration: ${{ parameters.buildConfiguration }}
sourceBranch: ${{ variables['Build.SourceBranchName'] }}
- ${{ if eq(parameters.testRunnerConfiguration, 'PowerShell') }}:
- template: ${{ variables.templateFolder }}/powershell-test-runner.yml
parameters:
fastBuild: ${{ parameters.fastBuild}}
buildConfiguration: ${{ parameters.buildConfiguration }}
sourceBranch: ${{ variables['Build.SourceBranchName'] }}
solutionName: ${{ parameters.solutionName }}
runInParallel: ${{ parameters.runInParallel }}
- ${{ if eq(parameters.testRunnerConfiguration, 'VSTest') }}:
- template: ${{ variables.templateFolder }}/vs-test-runner.yml
parameters:
fastBuild: ${{ parameters.fastBuild}}
buildConfiguration: ${{ parameters.buildConfiguration }}
sourceBranch: ${{ variables['Build.SourceBranchName'] }}
runInParallel: ${{ parameters.runInParallel }}
##########################################
### Creating Artifacts
##########################################
- template: ${{ variables.templateFolder }}/dotnet-core-publish.yml
parameters:
buildConfiguration: ${{ parameters.buildConfiguration }}
artifactName: '${{ parameters.appName }}'
additionalParameters: '/p:CopyOutputSymbolsToPublishDirectory=false'
listOfProjectToPublish:
- ${{ parameters.appName }}It contains a configuration which allows to enable/disable fast build by switching on and off unit tests. As you can noticed, I have added a few options for test runners. I did it because of execution time may significantly varies depends on execution methods.
Using VSTest, for example, it is able to run tests in parallel in very straightforward way:
- task: VSTest@2
displayName: Unit Tests
inputs:
testSelector: testAssemblies
testAssemblyVer2: |
**\*.Tests.Unit.dll
!\obj*
codeCoverageEnabled: False
runOnlyImpactedTests: True
runInParallel: ${{ parameters.runInParallel }}Deployment to available slot
After the artifact was created, a tester or developer deploys a code at one of the free slot available on IIS server. Below is a main delivery pipeline YAML:
"Release-$(rev:r)"
trigger: none
resources:
pipelines:
- pipeline: build
source: CI_ON_PullRequest
repositories:
- repository: REPOSITORY
type: git
name: REPOSITORY
appendCommitMessageToRunName: false
parameters:
### Reservation
- name: CreateReservation
displayName: Create Reservation
type: boolean
values:
- true
- false
default: false
- name: ReservationFrom
displayName: Reservation Day From
type: string
default: yyyy-MM-dd
- name: ReservationTo
displayName: Reservation Day To
type: string
default: yyyy-MM-dd
- name: DeployLeg1
displayName: Deploy Leg 1
type: boolean
values:
- true
- false
default: false
...
- name: DeployLegN
displayName: Deploy Leg N
type: boolean
values:
- true
- false
default: false
variables:
- group: PIPELINESECRETS
- group: Project Configuration
- group: Reservation Configuration
stages:
- template: ${{ variables.TemplateFolder }}/deploy-slot.yml
parameters:
DeploymentLeg: 1
Condition: ${{ and(eq(variables['Build.Reason'], 'Manual'), eq(parameters.DeployLeg1, true)) }}
HealthCheckEnabled: ${{ parameters.HealthCheckEnabled }}
CreateReservation: ${{ parameters.CreateReservation }}
ReservationFrom: ${{ parameters.ReservationFrom }}
ReservationTo: ${{ parameters.ReservationTo }}
...
- template: ${{ variables.TemplateFolder }}/deploy-slot.yml
parameters:
DeploymentLeg: N
Condition: ${{ and(eq(variables['Build.Reason'], 'Manual'), eq(parameters.DeployLeg19, true)) }}
HealthCheckEnabled: ${{ parameters.HealthCheckEnabled }}
CreateReservation: ${{ parameters.CreateReservation }}
ReservationFrom: ${{ parameters.ReservationFrom }}
ReservationTo: ${{ parameters.ReservationTo }}The main idea is having a 1 of N slots in the IIS pool ready to host feature branch. Deploying the pipeline, a testes specify a slot and optionally reserve it for further usage. The workflow might be automated using pipeline chain trigger:
trigger: none
resources:
pipelines:
- pipeline: build
source: CI_ON_PullRequest
trigger: true # enable the triggerReservation system
To make things even more convenient I created a reservation system. Any developer may books a specific slot for a some amount time to prevent the slot to be used by another person and loose changes.
I leverage Azure CosmoDB to store reservation information and Azure API for retrieving data. This is an awesome way to lock specific feature slot preventing another deployment to overwrite the test environment (deploy-slot.yml);
parameters:
- name: DeploymentLeg
type: number
- name: CreateReservation
type: boolean
- name: ReservationFrom
type: string
- name: ReservationTo
type: string
- name: Condition
type: string
default: ''
- name: DependsOn
type: object
default: []
stages:
- stage: FeatureTestDeploy${{ parameters.DeploymentLeg }}
displayName: Deploy to FeatureTest ${{ parameters.DeploymentLeg }}
${{ if eq(length(parameters.DependsOn), 0) }}:
dependsOn: []
${{ else }}:
dependsOn: ${{ parameters.DependsOn }}
${{ if ne(parameters.Condition, '') }}:
condition: ${{ parameters.Condition }}
...
jobs:
#############################
#### Create Reservation #####
#############################
- job: CreateReservation${{ parameters.DeploymentLeg }}
displayName: Create Reservation
pool: Hosted Agent Pool
steps:
- checkout: REPOSITORY
- powershell: |
IF ('${{ parameters.CreateReservation }}' -eq 'true') {
$reservationFrom = ([DateTime]'${{ parameters.ReservationFrom }}').toString('yyyy-MM-ddT00:00:00')
$reservationTo = ([DateTime]'${{ parameters.ReservationTo }}').toString('yyyy-MM-ddT23:59:59')
}
ELSE {
$reservationFrom = $(Get-Date -Format 'yyyy-MM-ddT00:00:00')
$reservationTo = $(Get-Date -Format 'yyyy-MM-ddT23:59:59')
}
Write-Host ReservationFrom: $reservationFrom
Write-Host ReservationTo: $reservationTo
Write-Host StorageUri: https://$(StorageAccount).table.core.windows.net/$(TableName)
IF ([DateTime]$reservationFrom -gt [DateTime]$reservationTo) {
Write-Host "##[error] reservation date From > reservation date To"
exit 1
}
IF ([DateTime]$reservationFrom -lt [DateTime]((Get-Date).ToString('yyyy-MM-ddT00:00:00'))) {
Write-Host "##[error] reservation date From < Today"
exit 1
}
IF ([DateTime]$reservationFrom -gt [DateTime]((Get-Date).AddDays($(MaxForDays)).ToString('yyyy-MM-ddT00:00:00'))) {
Write-Host "##[error] reservation date From > Today + $(MaxForDays) days"
exit 1
}
IF ((([DateTime]$reservationFrom).AddDays($(MaxForDays))) -lt [DateTime]$reservationTo) {
Write-Host "##[error] reservation for more than $(MaxForDays) days not allowed"
exit 1
}
. .\Tools\AzdoRestApi\storage\azure-tables-crud.ps1 -StorageAccount $(StorageAccount) -AccessKey $(Accesskey)
$now = (Get-Date).ToString('yyyy-MM-ddTHH:mm:ss')
$currectReservations = GetTableEntity -TableName $(TableName) -Filter "PartitionKey%20eq%20'${{ parameters.DeploymentLeg }}'%20and%20From%20lt%20datetime'$now'%20and%20To%20gt%20datetime'$now'"
$reservations = GetTableEntity -TableName $(TableName) -Filter "PartitionKey%20eq%20'${{ parameters.DeploymentLeg }}'%20and%20From%20le%20datetime'$reservationTo'%20and%20To%20ge%20datetime'$reservationFrom'"
IF (($currectReservations -eq $null) -and ($reservations -eq $null)) {
Write-Host "##[section] No reservation from $reservationFrom to $reservationTo exists on FT ${{ parameters.DeploymentLeg }}"
IF ('${{ parameters.CreateReservation }}' -eq 'true') {
$body = @{
RowKey = "$(Build.BuildId)"
PartitionKey = "${{ parameters.DeploymentLeg }}"
"From@odata.type" = "Edm.DateTime"
From = $reservationFrom
"To@odata.type" = "Edm.DateTime"
To = $reservationTo
Author = "$(Build.RequestedFor)"
}
PutTableEntity -TableName $(TableName) -Entity $body
Write-Host "Reservation created successfully"
$body
}
}
ELSE {
IF ($currectReservations[0].Author -eq "$(Build.RequestedFor)") {
Write-Host "##[section] Reservation for $($currectReservations[0].Author) already exists from $($currectReservations[0].From) to $($currectReservations[0].To) on FT ${{ parameters.DeploymentLeg }}"
$currectReservations
}
ELSE {
Write-Host "##[error] Reservation for $($currectReservations[0].Author) already exists from $($currectReservations[0].From) to $($currectReservations[0].To) on FT ${{ parameters.DeploymentLeg }}"
$currectReservations
exit 1
}
}
displayName: Check/Create ReservationIt must be mentioned, the Azure VMs pools might be replaced with on premise virtual machines to decrease latency.
All of this stuff configured to be monitored with Azure AppInsights installing SDK agents to provide health check functionality and alerting. Data collected inside Azure log workspace for monitoring and alerting in real time.
So, it was a quick presentation of building automated feature test environment on hybrid infrastructure with Azure, Azure DevOps server and on premise infrastructure. If you like the post, please subscribe to my newsletter and like the page below and go to the link for more interesting content.
Be an ethical, save your privacy!