Using Packer to create Windows AWS AMIs for declarative build agents

This post was written by Chris O’Dell co-author of Team Guide to Software Releasability.

Key Takeaways

• Take advantage of the Cloud to scale your buildagents to meet development workloads (and save costs by turning them off when not in use
• Use Packer to build declarative golden images, including with Windows, to reduce handcrafted buildagents
• Developers can use Packer to amend the installed tooling and make these changes via Pull-Request process allowing for review from Ops
• Use Pester to test your created instance before baking to an image
• Run you image builds from your CI tool to increase visiblity and make it runnable from the press of a button

 Tools Used

• AWS EC2 & AMIs
• Packer
• PowerShell (v4.0 is installed in Windows Server 2012 R2)
• Chocolatey
• Pester

Take advantage of the Cloud’s elasticity to meet demand

In a development environment you will undoubtedly have a Continuous Integration tool of some sort for compiling the code, running tests and generating the final release package.  Generally CI tools use a server and agent architecture where the actual work of compilation and packaging takes place on the remote agents.  Example tools are TeamCity, Jenkins, Bamboo and more.

Unlike with production servers, there is a tendency to treat buildagents as pets, with each one being handcrafted and manually updated to include the latest tool needed by the developers.  The degradation happens gradually and with the best of intentions.  CI infrastructure tends to be managed centrally under an IT department banner which is separated from the development team.  When new functionality is needed, for example an upgrade of the .net framework, there can be a long turnaround as a request is made and the change implemented.  This separation of concerns and ownership, where communication takes place over work tickets, can result in short term solutions such as simply installing the requirement on each buildagent.  The result is unique buildagents with the increased likelihood for each to produce different results when the expectation is that they are interchangeable.

The goal of buildagents is to meet the development workload by having a scaleable farm of identical machines, each with the expected requirements.  For this, cloud based buildagents, backed by Packer built AMIs, is perfect.  The Packer scripts can be amended directly by the developers, reviewed by the central IT team as part of a pull-request model and finally used to build the required image for the buildagents to launch.  This gives the development team control of the buildagents capabilities with the central IT team still retaining oversight and confidence in the tools installed.

Almost all CI tools support cloud based buildagents.  By utilising the elasticity of EC2, constrained by custom rulesets to effectively manage costs, a CI tool can spin up enough buildagents to handle the development load.  The cloud buildagents are also terminated when load reduces and the entire fleet can be terminated over night or when there’s no active development.

Packer is a Hashicorp tool for creating golden images.  It consists of Builders, Provisioners and Post-Processors.  Builders create the instance on which an image is based; Provisioners perform installations and customisations; and Post-Provisioners perform tasks on the resulting image such as sharing to other AWS accounts.

Multiple builders are supported, of which AWS is just one.  Others include Azure, Google Cloud, Docker, Hyper-V and VMWare, making it a useful tool in on-prem situations as well as Cloud.  Provisioner support ranges from the basic shell script to the more advanced Chef or Puppet, among others.

Using Packer to build a Windows AMI

Choosing and configuring the AMI

A Packer build consists of a JSON build file and any supporting provisioning scripts.  This snippet from the build file of our example Bamboo Server shows an AWS Builder.  For AWS a base AMI must be used to build upon.  Here I’m using the source_ami_filter tag to select the latest Windows Server 2012 R2 image owned by Amazon.  This is to ensure I build upon a recently patched box from a trusted source.  I also specify an instance type which is used only when provisioning – you can choose different instance types when later launching the image as a buildagent.  AMIs are tied to regions and so we need to tell Packer which region to build this in.

"builders": [{
  "type": "amazon-ebs",
  "region": "eu-west-1",
  "instance_type": "t2.micro",
  "ami_name": "Bamboo Server 6.1 {{timestamp}}",
  "user_data_file": "./scripts/SetUpWinRM.ps1",
  "communicator": "winrm",
  "winrm_username": "Administrator",
  "winrm_use_ssl": true,
  "winrm_insecure": true,
  "source_ami_filter": {
    "filters": {
      "name": "Windows_Server-2012-R2_RTM-English-64Bit-Base-*"
    },
    "owners": ["801119661308"],
    "most_recent": true
  }
}]

As this is a Windows 2012 R2 instance we are using WinRM to remotely connect.  Unlike SSH on Linux, WinRM is not enabled by default and must be set up before anything, even Packer, can connect to it.  As such, we run a script as part of the Launch step in the standard AWS EC2 instance lifecycle.   Packer sends the script as User Data.  Setting up WinRM on Windows Server 2012 R2 is unnecessarily arcane and you can see the full commands I use in SetUpWinRM.ps1.

Once the instance is up and running Packer will connect and start the Provisioners.  The first two Powershell scripts I run, EC2Config.ps1 and BundleConfig.ps1, set the EC2Config values so that when the resulting image is launched it behaves like a brand new instance by running sysprep and generating a new unique Administrator password.

Installing the Agent tools

Now that that instance configuration is out of the way, we can customise the image to our specific needs.  The next two scripts SetUpDevTools.ps1 and InstallBambooServer.ps1 install the tools we want on out buildagents.  Note: the script InstallBambooServer.ps1 is for demo purposes and should be replaced with the equivalent CI tool Agent.

Chocolatey is used as the package manager for tooling.  It’s a fantastic resource filling a rather large gap in the Windows world.  Without Chocolatey we’d have to take our own copies of the installation programs and store them in an accessible location such as an S3 bucket.  This is something you could do if you’d rather not use a public repository.  There are also Pro licences for Chocolatey which allow you to self-host your own server and packages as another alternative.

The first line installs Chocolatey by running a remote script and the next line sets auto confirm so that installations don’t pause the Packer script awaiting user input.  Next I install the necessary tools followed by Pester – a PowerShell unit testing framework.

# Install Chocolatey
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
# Globally Auto confirm every action
choco feature enable -n allowGlobalConfirmation

choco install netfx-4.5.2-devpack
# Install build tools 2015
choco install microsoft-build-tools --version 14.0.25420.1
choco install nuget.commandline
choco install nunit-console-runner
choco install git

# Install Pester to run server tests
choco install Pester

Test the image to ensure confidence

As with all scripts, there’s a chance they can fail, particularly if relying on third party services such as Cloud hosting, and so tests should be written to ensure the process produced the expected outcome.  For this, I use Pester to run checks on the instance itself.

This section of the Packer build file uploads the test files to the instance, runs Pester, and downloads the results.

{
  "type": "file",
  "source": "./tests",
  "destination": "C:/Windows/Temp"
},
{
  "type": "powershell",
  "script": "./scripts/RunTests.ps1"
},
{
  "type": "file",
  "source": "TestResults.xml",
  "destination": "TestResults.xml",
  "direction": "download"
}

The tests query the state of the instance and confirm the presence of the development tools.  For example, the following tests in MSBuild14Installed.Tests.ps1 verify that the .net 4.5.2 framework is present and that MSBuild is in the expected location:

Describe 'Dev Tools Installer' {
  It 'installs Framework .Net 4.5.2 or newer' {
    Test-Path"HKLM:SOFTWAREMicrosoftNET Framework SetupNDPv4Full"| Should be $true
    $key=Get-ItemProperty"HKLM:SOFTWAREMicrosoftNET Framework SetupNDPv4Full"
    $key.Release| Should BeGreaterThan 378758#378758 is .net 4.5.1
  }

  It 'ensures MSBuild 14 is present at the expected location' {
    Test-Path"C:Program Files (x86)MSBuild14.0BinMSBuild.exe"| Should be $true
    }
}

When running Pester we tell it to produce NUnit equivalent XML output, which can then be easily parsed by most CI tools.  The command for this in RunTests.ps1 is

Invoke-Pester C:/Windows/Temp/* -OutputFile TestResults.xml -OutputFormat NUnitXml

Run your Packer builds from the CI tool

I suggest setting up the running of the Packer build as a job within your CI tool.  By running your Packer build as a CI job you gain a history of runs, a centralised place for anyone (potentially only authorised people if roles are used) to be able to execute the command, collect the outputs and display the results in a readable manner, plus a nice green or red indication of success or failure.

You also tackle the lack of visibility that occurs when a request changes hands between teams.  When the build is in CI then the requesting Developer can easily see when the new capabilities will be available on the buildagents.

Summary

The initial set up of this process is a bit of an investment – creating the initial Packer build, figuring out the WinRM connection, writing the powershell installation script, and so on.  But once it’s in place, all future buildagent updates will be much smoother, easier to implement, tested and the changes will be open and easily understood for all.  Gone will be the days where Developers are confused to see builds behave differently on one “identical” agent to another.

The full source for my packer build of an example Bamboo CI Server for building .net 4.5.2 web applications can be found on Github.  I’ve included a readme with instructions for running Packer.  It is licensed under the MIT Licence so feel free to make use of any bits you may find helpful.

The tooling we use to develop and build applications is continually changing and the use of source controlled build processes like this will enable your team to implement the new requirements with speed and grace.