Use Virtual Machine Scale Set for your Azure DevOps Agents with Cloud-Init

Do you use Azure DevOps self-hosted agents in Azure inside a Virtual Machine, and you need to have a solution to scale the agents in peak demands. Not enough, your machine uses a customized image due a few dependencies needed by your application, and you also don’t know how to achieve that? Then, this article is for you. Get to know what Azure Virtual Machine Scale Set (VMSS) is, what is Cloud-Init and how to address this challenge in a nutshell.

First of all, let’s understand what an agent in the context of Azure DevOps is: To build your code or deploy your software using Azure Pipelines, you need at least one agent. As you add more code and people, you’ll eventually need more. When your pipeline runs, the system begins one or more jobs. An agent is a computing infrastructure with installed agent software that runs one job at a time. Click here to understand more about Azure Pipelines Agents.

Once you have an agent installed in a machine, let’s say you set up a VM to host your agent in which you will deploy your application. With only one VM, you will face issues to scale your jobs on-demand. Each agent executes only one job at a time. If you suddenly have a peak of ten new jobs, your pipeline workflow will be slow, as the agent will queue all the nine jobs.

Illustrative image with queue dominos.

To address this challenge, you need a scalable solution that is performative, and will provision new fresh DevOps agents for you to sustain the increase of demand and deprovision when demands decrease. Here is the point where the VMSS arises.

Virtual Machine Scale Set (VMSS) lets you create and manage a group of load balanced VMs in Azure that automatically increase or decrease in response to demand or a defined schedule. Click here to understand more about Azure Virtual Machine Scale Set.

In our scenario, we need our VMs to be provisioned with a few dependencies: Docker and Kubernetes. For this, let’s make use of Cloud-Init to install these upon the machine, right after it is provisioned.

Cloud-init is a service that sets up your VM instance with the wanted configuration and software ready to use. It uses metadata given by the cloud provider or the direct user. It does so by executing scripts, most commonly from the cloud-config file. Click here to understand more about Cloud-init.

In our example, we will create a VMSS in Azure via Azure Cloud Shell with the following commands:

Create a Resource Group called vmss-demo:

az group create -l eastus -n vmss-demo

To make use of Cloud-init, we need to create a cloud-config file which will contain the dependencies and commands we want to be executed on the VM. We can create this file with the following content:

#cloud-config.yml
package_update: true
packages:
  - apt-transport-https
  - curl
  - docker.io

runcmd:
  - curl -L -o /usr/bin/kubectl "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  - chmod +x /usr/bin/kubectl

As we are making all these steps in Azure Cloud Shell (ACS), you will need to upload this file to the ACS as you will need to reference it on the command to create the VMSS.

PS: it’s important to notice that if you create the VMSS from the Azure Portal, there will be an option called Orchestration Mode. In order to use it in Azure DevOps, this option needs to be set to Uniform orchestration, as Azure DevOps does not support Flexible orchestration yet. Click here to understand more.

Create a Virtual Machine Scale Set called vmss-demo-3u2h:

az vmss create \
  --resource-group vmss-demo \
  --name vmss-demo-3u2h \
  --image Canonical:0001-com-ubuntu-server-focal:20_04-LTS:latest \
  --upgrade-policy-mode automatic \
  --custom-data /home/wesley/cloud-config.yml \
  --admin-username wgomes022 \
  --generate-ssh-keys

Now that we have the VMSS already provisioned on Azure, let’s access one VM instance via SSH connection, and validate if our dependencies (Docker and Kubernetes) are installed indeed.

Access the VM via SSH:

ssh wgomes022@4.157.253.66 -p 50000

PS: you need to validate the public IP and port number of the chosen instance and update in the command, so the connection may be successful.

Now, let’s validate if Docker and Kubernetes are installed. If it is successful, the command will return the path of both dependencies:

which docker
which kubectl

Create a Deployment Group in Azure DevOps:

Now that we already have everything set up on Azure, it’s time to create a connection with Azure DevOps to use the agents in VMSS.

Go to your Azure DevOps organization -> project settings -> Agent pools -> Add pool -> New -> and fill the following labels:

• Pool type: Azure virtual machine scale set
• Azure subscription: <your-azure-subscription>
• Virtual machine scale set: vmss-demo-3u2h
• Name: <name-for-the-agent-pool>
• Description: <optional>

Screen that shows the creation process for an Agent Pool in Azure DevOps.

In Pool options:

• Check Automatically tear down virtual machines after every use: optional.
• Maximum number of virtual machines in the scale set: 4.
• Number of agents to keep on standby: 2.
• Delay in minutes before deleting excess idle agents: 5.

Now you can just create a release pipeline and targets this Agent Pool to test it out. You will notice that after a few minutes, 2 new agents will be provisioned, both in the VMSS.

Azure DevOps Agent Pool containing two new DevOps agents.

If you run multiple new release jobs, you will notice more VMSS will be raised in the VMSS (bound to the limit you defined as the maximum number of VMs in the creation of the Agent Pool) in order to sustain the increase demand on the new release jobs.

Hope this article helped you understand a bit more about a possible use case for VMSS in Azure DevOps. If you have any concerns or suggestions, don’t hesitate to reach out.