Secret management in Azure DevOps Pipelines

Secret management is an essential part of release management process. As a DevOps you have to be sure, that automated process have appropriate access to all resources it have to. From another hand, we need a control over the way how exactly sensitive information stores and transfers to executive process. It might be even more complicated if we are talking about automated process where credential rotation policies must be applied. The most of these problem might be resolver using central save credential store.

Today, I am going to show how safely use Azure DevOps (AZDO) YAML pipelines with built in credential store and the way to integrate AZDO with Azure KeyVault, to list and get encrypted secrets in run time only. Using Azure KeyVault, we may easy implement secret rotation as well. And the most interesting part will be a secure sharing of sensitive variables loaded between jobs running on even separate physical agents.

Using pipeline variables

First, and the most easiest way to store sensitive information is pipeline variables. This is a very simple way with no additional configuration needed. Right from pipeline settings:

Keep this value secret to encrypt the information in AZDO. You may also check ‘Let users override this value when running this pipeline‘ and user will able to modify the value right before running of the pipeline.

This is very straightforward but it might cause additional operational overhead if the same credentials should be shared across a number of pipelines.

Using variable groups

Variable groups functionality is part of Azure DevOps library. This is a central place to store and logically group sensitive information in AZDO.

Go to the Library and create a new variable group. Just do not link to Azure key vault for now.

Now, we able to specify a variable group as a source in yaml template. It makes possible to use the same set of credentials across different pipelines. What really nice about yaml pipelines is ability to use shell operations like ##VSO to share secrets across different steps.

name: "Release-$(rev:r)"

trigger: none

appendCommitMessageToRunName: false

parameters:
- name: MyParam
  displayName: My parameter
  type: boolean
  values:
  - true
  - false
  default: true
  
pool: managed-agent
    
variables:
- group: AWESOME-VARIABLE-GROUP

stages:
- stage: Deployment
  displayName: Deployment
  jobs:
  - job: ListVariableGroup
    steps:
    - checkout: none
    - powershell: |
        Write-Host "$(AWESOME-PASSWORD)" # to write to output directly
        Write-Host "##vso[task.setvariable variable=AwesomeSecretCode;isOutput=true;]$(AWESOME-SECRET-CODE)" # to save in environment variable
    - template: deploy-template-a.yml
        parameters:
          AwesomeSecretCode: $(AwesomeSecretCode)
    - template: deploy-template-b.yml
        parameters:
          AwesomeSecretCode: $(AwesomeSecretCode)

As you could noticed, the pipeline runs one single managed agent with only one job.

Sharing variables across jobs

What if we need to share variable across different jobs or deployments or even different physical/virtual agents?

To do so, use yaml jobs dependency. First, we have to create a separate job just for retrieving sensitive information and store it in pipeline environment variables:

jobs:
  - job: DownloadVariableGroupSecrets
    displayName: Download Variable Group Secrets
    pool: managed-agent
    variables:
    - group: AWESOME-VARIABLE-GROUP
    steps:
    - checkout: none
    - powershell: |
        Write-Host "##vso[task.setvariable variable=AwesomePassword;isOutput=true;]$((AWESOME-PASSWORD)"
        Write-Host "##vso[task.setvariable variable=AwesomeSecretCode;isOutput=true;]$(AWESOME-SECRET-CODE)"
      displayName: Assign variable group secrets to variables
      name: VariableGroupSecrets

Now, let’s create two separate deployment jobs, one for test and another for production environments:

jobs:
  - job: DownloadVariableGroupSecrets
    displayName: Download Variable Group Secrets
    pool: managed-agent
    variables:
    - group: AWESOME-VARIABLE-GROUP
    steps:
    - checkout: none
    - powershell: |
        Write-Host "##vso[task.setvariable variable=AwesomePassword;isOutput=true;]$((AWESOME-PASSWORD)"
        Write-Host "##vso[task.setvariable variable=AwesomeSecretCode;isOutput=true;]$(AWESOME-SECRET-CODE)"
      displayName: Assign variable group secrets to variables
      name: VariableGroupSecrets
      
      
- deployment: DeployTest
  displayName: Deploy to test VM
  dependsOn: ['DownloadVariableGroupSecrets']
  timeoutInMinutes: 0
  cancelTimeoutInMinutes: 1
  environment:
    name: test
    resourceType: VirtualMachine
  variables: 
    DatabasePassword: $[ dependencies.DownloadVariableGroupSecrets.outputs['VariableGroupSecrets.AwesomePassword'] ]
  strategy:
    runOnce:
      deploy:
        steps:
        - template: deploy-template-test.yml
            parameters:
              DatabasePassword: $(DatabasePassword)
      
- deployment: Deploy
  displayName: Deploy to prod VM slot BLUE
  dependsOn: ['DownloadVariableGroupSecrets','DeployTest']
  timeoutInMinutes: 0
  cancelTimeoutInMinutes: 1
  environment:
    name: prod
    resourceType: VirtualMachine
    tags: blue
  variables: 
    DatabasePassword: $[ dependencies.DownloadVariableGroupSecrets.outputs['VariableGroupSecrets.DatabasePassword'] ]
    SlotAccessCode: $[ dependencies.DownloadVariableGroupSecrets.outputs['VariableGroupSecrets.AwesomeSecretCode'] ]
  strategy:
    runOnce:
      deploy:
        steps:
        - template: deploy-template-prod.yml
            parameters:
              DatabasePassword: $(DatabasePassword)
              SlotAccessCode: $(SlotAccessCode)

In this example I have been using environments configured before which targets to specific pools of VMs. It is highly recommended doing this way, because of more granular control over the deployment environment configuration. We will able to set approval policies or branch control policies.

Using Azure KeyVault

First, we need to connect to Azure KeyVault service with service principal. Open Project Settings and go to Service connections. Select Azure Resource Manager connection and create Service principal manually (recommended). Be sure you have key vault and service principal account created in Azure. You will need provide Subscription Id, Subscription Name, Service Principal Id, Service principal key and Tenant Id.

Verify and save.

Go and create another variable group. Now, link secret from Azure key vault as variables and select keys from the list.

So, we able to use credentials loaded right from Azure KeyVault, just adding the variable group to pipeline section as we done before. It will create additional step in pipeline execution process to get secrets from Azure and store in local environment variables.

- bash: |
      echo $(AWESOME-SECRET-CODE) | base64

Log output:
YXdlc29tZS1zZWNyZXQK

$ echo 'YXdlc29tZS1zZWNyZXQK' | base64 -d
awesome-secret

Be an ethical, save your privacy!