Create Docker image with Packer, PowerShell, PowerCLI and Azure DevOps self-hosted agent
Today we will create a Docker image that will contain Packer, Windows Update Provisioner for Packer, PowerShell, PowerCLI and Azure DevOps self-hosted agent.
To automate Packer image creation process we will utilize Azure DevOps, which is a service hosted in public cloud. What we want to do is to create Packer image on-premise in our datacenter, so there is a problem with communication between vCenter server and Azure DevOps.
Azure DevOps agent
Why Azure DevOps agent is needed?
Normally Azure DevOps does not have any connection with our datacenter. To establish connection between on-premise vCenter server and Azure DevOps, a dedicated agent placed inside our datacenter is needed. That agent needs to have network connection with our vCenter server and ESXi hosts. It can be placed in different VLAN, but that VLAN needs to be routable with vCenter and ESXi VLAN.
What Azure Devops agent does?
It actually triggers a tasks initiated from Azure DevOps portal locally in our infrastructure. All files reuired to create our builds needs to be first copied on Azure Devops agent and agent triggers the build job. The files copied on agent from Azure DevOps server are called artifacts. Because agent triggers a build jobs, it needs to be equipped with all necessary tools that are required to create Packer template. Those are:
- Packer
- Packer Windows Update provisioner
- PowerShell
- PowerCLI
Self-hosted Azure DevOps agent
Azure DevOps agent can be hosted in Azure or it can be hosted locally in our DC. Obviously we will need self-hosted agent.
First of all it allows direct communication with our datacenter. Second thing - it's free. For Azure DevOps agent hosted in Azure you normally needs to pay some money.
Of course everything depends on subscription level you own, but my assumption is that you don't have any available subscription and want to minimize operational costs.Remember, or goal is to build secure templates and minimize operational costs!
To establish communication between DevOps server and on-premise vCenter server a dedicated token will be used. This token will have to be generated on Azure Devops portal.
Support platforms for DevOps agents
Azure DevOps agent can be installed on Windows or on Linux. We will choose Linux as our destination platform and there is few reasons for that:
- When utilizing Linux you don't need to pay for Windows license. Since Azure DevOps agent will just trigger our build tasks, there is no need to pay Windows license for such service.
- When using Linux as agent platform we can utlize Docker and automate process of creating a docker image will all necessary tools.
You could probably use some bash script to automate process of installing and configuring all necessary tools on Linux, but using Docker will be much faster and efficient way to achieve expected result.
- Since DevOps agent keeps all artifacts locally in clear text (including passwords), security is a major concern here. It's very easy to erase Windows password using some small 3rd party tool and get access to all artifacts and all passwords on that agent. Using Linux and Docker container make a things much harder for potential attacker. We will use other methods to secure data on agent machine but Linux is much better option in terms of security.
Build Ubuntu VM
Create Ubuntu 21.04 Packer image
We could use Windows Subsystem for Linux (WSL) to speedup Docker image creation process but we want to build our final solution which will utilize Ubuntu VM. For this reason we will use Ubuntu 21.04. To create Ubuntu VM Template using Packer use this link.
Customize Ubuntu VM
To make life easier first enable root account and setup new root account password.
1sudo passwd root
Once done switch to root account typing command:
1su root
Username will change from your existing user to root. This means we have elevated privileges on the system.
Set static IP
1cd /etc/netplan/
2ls
Once done a configuration file will be displayed
Edit the file using nano editor:
1nano 00-installer-config.yaml
Edit IP, gateway and DNS settings to match your network configuration:
1# This is the network config written by 'subiquity'
2network:
3 ethernets:
4 ens192:
5 addresses:
6 - 192.168.1.222/24
7 gateway4: 192.168.1.1
8 nameservers:
9 addresses: [192.168.1.1, 8.8.8.8]
10 search: [lab.local]
11 version: 2
Type Ctrl+x and confirm saving changes typing y.
1netplan apply
This will apply network settings that has been modified. From now putty can be used to interact with VM through SSH console
Change hostname
Perform fallowing command to change hostname to ld9-vcsabackups01
1sudo hostnamectl set-hostname devopsagent01
Provide server description using fallowing command:
1sudo hostnamectl set-hostname "Azure DevOps Self-hosted Agent" --pretty
Once applied verify that changes have been applied
1hostnamectl
Upgrade system
Perform server upgrade
1sudo apt-get update
2sudo apt-get-upgrade
Install Docker Engine and Docker Compose
Install Docker Engine
Docker Engine allow us to run standalone containers.
First remove existing Docker Engine if exists:
1sudo apt-get remove docker docker-engine docker.io containerd runc
Update the apt package index and install packages to allow apt to use a repository over HTTPS:
1 sudo apt-get update
2 sudo apt-get install \
3 ca-certificates \
4 curl \
5 gnupg \
6 lsb-release
Add Docker’s official GPG key:
1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
Use the following command to set up the stable repository.
1echo \
2 "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
3 $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker Engine
1sudo apt-get update
2sudo apt-get install docker-ce docker-ce-cli containerd.io
Check Docker version:
1docker --version
2
3Docker version 20.10.10, build b485636
Install Docker Compose
Compose is a file format for describing distributed Docker apps, and it’s a tool for managing them. Docker Compose relies on Docker Engine for any meaningful work, so make sure you have Docker Engine installed either locally or remote.
On Linux, you can download the Docker Compose binary from the Compose repository release page on GitHub.
Run this command to download the current stable release of Docker Compose:
1sudo curl -L "https://github.com/docker/compose/releases/download/v2.2.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Apply executable permissions to the binary:
1sudo chmod +x /usr/local/bin/docker-compose
Note: If the command docker-compose fails after installation, check your path. You can also create a symbolic link to /usr/bin or any other directory in your path.
1sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
Test the installation:
1docker-compose --version
2
3Docker Compose version v2.2.2
Once Docker Engine and Docker Compose will be installed we need to add existing, non-root user to docker group to perform docker tasks without privileged mode:
1sudo usermod -aG docker your-user-name
Once added, restart your SSH session to apply changes.
Preparing for Docker container image creation
Inside your home directory create new folder named "Azure-DevOps-Agent":
1mkdir Azure-DevOps-Agent
2cd Azure-DevOps-Agent/
Once we are inside devopsagent folder create new file named Dockerfile:
1nano Dockefile
To build container image we need a file named Dockerfile which will include fallowing content:
1# Indicates the base image.
2FROM ubuntu:18.04
3
4# Define Args for the needed to add the package
5ARG PACKER_VERSION="1.7.8"
6ARG PACKER_WINDOWSUPDATE_VERSION="0.14.0"
7ARG PS_VERSION=7.1.3
8ARG PS_PACKAGE=powershell_${PS_VERSION}-1.ubuntu.18.04_amd64.deb
9ARG PS_PACKAGE_URL=https://github.com/PowerShell/PowerShell/releases/download/v${PS_VERSION}/${PS_PACKAGE}
10ARG TARGETARCH=amd64
11ARG AGENT_VERSION="2.195.2"
12
13# Define ENVs for Localization/Globalization
14ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false \
15 LC_ALL=en_US.UTF-8 \
16 LANG=en_US.UTF-8 \
17 # set a fixed location for the Module analysis cache
18 PSModuleAnalysisCachePath=/var/cache/microsoft/powershell/PSModuleAnalysisCache/ModuleAnalysisCache \
19 POWERSHELL_DISTRIBUTION_CHANNEL=PSDocker-Ubuntu-18.04
20ENV PACKER_VERSION=${PACKER_VERSION}
21ENV PACKER_WINDOWSUPDATE_VERSION=${PACKER_WINDOWSUPDATE_VERSION}
22ENV DEBIAN_FRONTEND=noninteractive
23RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
24
25# Install dependencies and clean up
26RUN apt-get update \
27 && apt-get install --no-install-recommends -y \
28 # curl is required to grab the Linux package
29 curl \
30 # less is required for help in powershell
31 less \
32 # requied to setup the locale
33 locales \
34 # required for SSL
35 ca-certificates \
36 gss-ntlmssp \
37 # PowerShell remoting over SSH dependencies
38 openssh-client \
39 # Install python
40 python3 \
41 python3-pip\
42 python3-boto\
43 # Install unzip
44 unzip \
45 # Install DevOps Agent packages
46 jq \
47 git \
48 iputils-ping \
49 libcurl4 \
50 libicu60 \
51 libunwind8 \
52 netcat \
53 libssl1.0 \
54 # Download the Linux package and save it
55 && echo ${PS_PACKAGE_URL} \
56 && curl -sSL ${PS_PACKAGE_URL} -o /tmp/powershell.deb \
57 && curl -LO https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip \
58 && curl -LO https://github.com/rgl/packer-plugin-windows-update/releases/download/v${PACKER_WINDOWSUPDATE_VERSION}/packer-plugin-windows-update_v${PACKER_WINDOWSUPDATE_VERSION}_x5.0_linux_amd64.zip \
59 && unzip '*.zip' -d /usr/bin \
60 && chmod +x /usr/bin/packer-plugin-windows-update_v${PACKER_WINDOWSUPDATE_VERSION}_x5.0_linux_amd64 \
61 && rm *.zip \
62 && mv /usr/bin/packer-plugin-windows-update_v${PACKER_WINDOWSUPDATE_VERSION}_x5.0_linux_amd64 /usr/bin/packer-plugin-windows-update \
63 && curl -LsS https://aka.ms/InstallAzureCLIDeb | bash \
64 && apt-get install --no-install-recommends -y /tmp/powershell.deb \
65 && apt-get dist-upgrade -y \
66 && apt-get clean \
67 && rm -rf /var/lib/apt/lists/* \
68 && locale-gen $LANG && update-locale \
69 # remove powershell package
70 && rm /tmp/powershell.deb \
71 # intialize powershell module cache
72 # and disable telemetry
73 && export POWERSHELL_TELEMETRY_OPTOUT=1 \
74 && pwsh \
75 -NoLogo \
76 -NoProfile \
77 -Command " \
78 \$ErrorActionPreference = 'Stop' ; \
79 \$ProgressPreference = 'SilentlyContinue' ; \
80 while(!(Test-Path -Path \$env:PSModuleAnalysisCachePath)) { \
81 Write-Host "'Waiting for $env:PSModuleAnalysisCachePath'" ; \
82 Start-Sleep -Seconds 6 ; \
83 }"
84
85RUN apt-get clean && \
86 rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
87
88WORKDIR /azp
89RUN if [ "$TARGETARCH" = "amd64" ]; then \
90 AZP_AGENTPACKAGE_URL=https://vstsagentpackage.azureedge.net/agent/${AGENT_VERSION}/vsts-agent-linux-x64-${AGENT_VERSION}.tar.gz; \
91 else \
92 AZP_AGENTPACKAGE_URL=https://vstsagentpackage.azureedge.net/agent/${AGENT_VERSION}/vsts-agent-linux-${TARGETARCH}-${AGENT_VERSION}.tar.gz; \
93 fi; \
94 curl -LsS "$AZP_AGENTPACKAGE_URL" | tar -xz
95
96COPY ./start.sh .
97RUN chmod +x start.sh
98
99
100# Install the VMware.PowerCLI Module
101SHELL [ "pwsh", "-command" ]
102RUN Install-Module VMware.PowerCLI, PowerNSX -Force -Confirm:0;
103RUN Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -confirm:$false
104
105## Install DevOps Agnet
106#CMD [ "pwsh" ]
107#CMD ["/bin/bash"]
108ENTRYPOINT [ "./start.sh" ]
In same directory as Dockerfile create new file named start.sh
1$ nano start.sh
and paste there fallowing content:
1#!/bin/bash
2set -e
3if [ -z "$AZP_URL" ]; then
4 echo 1>&2 "error: missing AZP_URL environment variable"
5 exit 1
6fi
7if [ -z "$AZP_TOKEN_FILE" ]; then
8 if [ -z "$AZP_TOKEN" ]; then
9 echo 1>&2 "error: missing AZP_TOKEN environment variable"
10 exit 1
11 fi
12 AZP_TOKEN_FILE=/azp/.token
13 echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
14fi
15unset AZP_TOKEN
16if [ -n "$AZP_WORK" ]; then
17 mkdir -p "$AZP_WORK"
18fi
19export AGENT_ALLOW_RUNASROOT="1"
20cleanup() {
21 if [ -e config.sh ]; then
22 print_header "Cleanup. Removing Azure Pipelines agent..."
23 # If the agent has some running jobs, the configuration removal process will fail.
24 # So, give it some time to finish the job.
25 while true; do
26 ./config.sh remove --unattended --auth PAT --token $(cat "$AZP_TOKEN_FILE") && break
27 echo "Retrying in 30 seconds..."
28 sleep 30
29 done
30 fi
31 }
32 print_header() {
33 lightcyan='\033[1;36m'
34 nocolor='\033[0m'
35 echo -e "${lightcyan}$1${nocolor}"
36 }
37 # Let the agent ignore the token env variables
38 export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE
39 source ./env.sh
40 print_header "1. Configuring Azure Pipelines agent..."
41 ./config.sh --unattended \
42 --agent "${AZP_AGENT_NAME:-$(hostname)}" \
43 --url "$AZP_URL" \
44 --auth PAT \
45 --token $(cat "$AZP_TOKEN_FILE") \
46 --pool "${AZP_POOL:-Default}" \
47 --work "${AZP_WORK:-_work}" \
48 --replace \
49 --acceptTeeEula & wait $!
50 print_header "2. Running Azure Pipelines agent..."
51 trap 'cleanup; exit 0' EXIT
52 trap 'cleanup; exit 130' INT
53 trap 'cleanup; exit 143' TERM
54 # To be aware of TERM and INT signals call run.sh
55 # Running it with the --once flag at the end will shut down the agent after the build is executed
56 ./run.sh "$@" --once &
57 wait $!
Alternatively once you are under your home directory run fallowing command:
1git clone https://github.com/danonh/Azure-DevOps-Agent
Once done you should see Azure-DevOps-Agent folder which will contain three files:
1├── Dockerfile
2├── Readme.md
3└── start.sh
You can find both files on my GitHub account
Changing product versions
Within Dockerfile I've hardcoded particular product versions:
- Packer version 1.7.8
- Windows Update Provisioner version 0.14.0
- PowerShell version 7.1.3
- Azure DevOps agent version 2.195.2
Obviously you may change product versions according your needs. This can be done modifying ARG values directly in the Dockerfile or provide alternative values during building process which will be passed to Dockerfile.
Build Docker image
Finally we are at stage where building process can be initiated.
If product versions has been manually changed directly inside Dockerfile then you can simply run fallowing command:
1cd Azure-DevOps-Agent/
2docker build -t packercontainer:local .
This will initiate building process that may take up to 15 minutes.
Alternatively you can pass particular product versions into docker build command:
1docker build -t packercontainer:local --build-arg PS_VERSION="7.1.4" --build-arg PACKER_VERSION="1.7.7" .
With those arguments we are telling Docker to include PowerShell version 7.1.4 instead 7.1.3 and Packer version 1.7.7 instead 1.7.8.
Don't forget to include . at the end of build command. This tells where Docker should look for Dockerfile.
Now sit down and relax.
If everything will go well you should see fallowing console output:
1Removing intermediate container 281635ad7c01
2 ---> fc1495d19d30
3Step 24/24 : ENTRYPOINT [ "./start.sh" ]
4 ---> Running in f2d498a10b01
5Removing intermediate container f2d498a10b01
6 ---> ae65ba27dd06
7Successfully built ae65ba27dd06
8Successfully tagged packercontainer:local
This would mean that our docker image has been created successfully. You can check if container image is visible running fallowing command:
1docker images
The output should look like this:
1docker images
2REPOSITORY TAG IMAGE ID CREATED SIZE
3packercontainer local ae65ba27dd06 15 minutes ago 2.25GB
To run the container we would have to pass few specific values from our Azure DevOps account, but that will be topic for some other blog entry.
Thanks for reading.