1 of 33

Azure Infrastructure as Code �Framework

Presented by George Kopf

gkopf@Princeton.edu

2 of 33

Presentation Overview

2

What is and why use Infrastructure as Code?
What is and why use a coding framework?
A lesson in refactoring.
The current situation.
Next steps and future enhancements.

Newer tools for cloud environments:

Native (Azure, AWS, GCP, OCI)

Newer tools for cloud environments:

Native (Azure, AWS, GCP, OCI)

Newer tools for cloud environments:

Native (Azure, AWS, GCP, OCI)

3 of 33

Infrastructure as Code

- Isn’t that an oxymoron?

- - No. Those oxy-things have horns and don’t call me a moron.

3

4 of 33

Infrastructure as Code

4

Software that executes system commands!

Established tools for Linux and Windows:

Puppet, Chef, Ansible

Native tools for cloud environments:

Azure (CLI, PowerShell, ARM, Bicep)

AWS (Cloud Formation)

Terraform for cross-platform code.

5 of 33

Infrastructure as Code (IAC)

5

Maintains “state”. Prevents drift.
Repeatable across environments.
Easy to make incremental changes.
Debug the code not the environment.
The code is living documentation.

On prem, we have two levels of IAC.

When a VM is stood up, IAC code is executed to lay down the base configuration (Windows 10, Windows 11, Linux 7, Linux 8, etc.)

After the VM is delivered, we put a “role” on the VM and then IAC turns it into the kind of server we want. (e.g. Apache Proxy, Nginx Web Server, Docker Host, API Gateway, Development Server, etc…)

Infrastructure as Code is the configuration for target systems. To make changes to the system, you have to update your code. If somebody (or something) manually alters your system the code changes it back.

After decades of debugging systems by logging onto several VMs at once and comparing values and guessing what is going on, being able to read the code to determine the state is liberating.

Knowing that Dev, QA and Prod are identical is comforting.

Using diff to compare two different configurations, such a powerful tool.

And since we check everything into Git, we can compare current state with a previous commit. How often have you had to answer the question, “What’s changed recently?” With IAC you say “Give me a minute and I’ll let you know!”

6 of 33

The Framework

- I find it ironic that there are so many different ways to standardize.

6

7 of 33

Framework

7

Abstraction: Reference low-level common tasks
Reusability: Write once use everywhere
Extensibility: Additive code as needed
Modularity: Functional independence
Standardization: Easy to read and maintain

8 of 33

Framework

8

A standard coding structure and approach.

Many existing frameworks for software development.

Spring, Struts, Grail, Ruby, Django

A pseudo-framework for IAC

Roles

what is this resource supposed to do?

Profiles

Constants and environment specific parameter values

Components

Reusable code

9 of 33

Roles – Profiles - Components

9

Role – What will this VM be used for?
Profile – Server specific parameter values
Component – Reusable low level code.

10 of 33

10

Modules

Naming Conventions

Parameterization

Templates

Coding styles

Code Dependencies

Folder Structure

Segmentation

Encapsulation

Language

Tool Selection

11 of 33

The Journey

- It’s not the destination that’s important.

- - It’s all of the dead ends, rabbit holes and ditches you get to explore on the way.

11

12 of 33

Early steps on the wrong road.

Terraform: Lagging Azure releases. Overkill since we are focused one cloud provider.

Azure CLI: Low level code with minimal abstraction and reusability.

PowerShell: Difficult to extend. Coded ourselves into a corner.

ARM Templates: Giant JSON files with giant JSON parameter files.

We based our first iteration on the Puppet IAC Framework

Roles wasn’t broad enough to accommodate the set of resources needed to create the landing zone.

We settled on discreet “services” as the product for each pipeline.

Profiles transitioned into “configuration”.

This is where we capture environment and application specific configuration.

Components transitioned into “modules”.

Modules are reusable code just like a software library. Enabling us to write once to be used everywhere.

12

Roles wasn’t broad enough to accommodate the set of resources needed to create the landing zone. On-premise we only had to configure VMs to be a certain type of VM. In Azure we were trying to create Resource Groups, Virtual Networks, Load Balancers, Public IP Addresses, Container App Environments, etc. The conventional “role” became confusing and irrelevant. This eventually turned into “Pipelines” instantiating a “Service”.

Profiles transitioned into “Configuration”.

Common variables for the pipeline
Environment specific parameters
Resource specific parameters

Components are reusable code just like a library. Enabling us to write once and use everywhere. Azure calls them “Modules”.

Started with Terraform, moved to Azure CLI, then PowerShell, dabbled with ARM templates and finally ended up adopting Bicep.

We built entire services end-to-end using each one of these approaches until we ran into a dead end. Then we selected the next and did it all over again. This could have resulted in philosophical arguments with people campaigning for the one they knew the best, but we continued to push everybody to work together to find the best approach. I personally wrote the same pipeline 3 different ways.

13 of 33

Why Bicep?

Microsoft Domain Specific Language

Developed by Microsoft to address the shortcomings in other tools.
Compiles into ARM Templates.
Large public Microsoft repository of reusable modules (components).
Up to date with new functionality (even preview resources).
Microsoft is investing in Bicep and continues to improve it.
Both “code” and “configuration” are in Bicep syntax.

13

14 of 33

Each pipeline is organized around a discreet service.

Resource Group

14

Virtual Network and Subnets

Container App Environment

App Gateway

Container Registry

Applications

Log Analytics Workspace

API Manager

GraphQL Service

Each Service has:

One Pipeline

One or more Configuration files

One or more Modules

Key Vault

15 of 33

A picture is worth a thousand words.

15

main.bicep

DevOps Trigger

Pipeline YAML

env.bicepparam

main.bicep

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

Instantiated Azure Resources

common variables

YAML Templates

16 of 33

Adopted Microsoft Conventions

Our repositories follow the naming convention and folder structure of Microsoft.

GitHub repository of Microsoft modules: https://github.com/Azure/ResourceModules/tree/main/modules

…/network/application-gateway/main.bicep
…/key-vault/vault/main.bicep
… /resources/resource-group/main.bicep

16

17 of 33

Public vs. Private Bicep Registry

Utilizing Microsoft’s Bicep Registry whenever possible.

GitHub repository of https://github.com/Azure/bicep-registry-modules/tree/main/modules

…/network/virtual-network/main.bicep

17

But not every resource has as a public Bicep registry module.
Created a “private” Bicep registry and wrote our own modules.
This Bicep registry is backed by a Git repository.
CI/CD pipeline to promote from the git repo to the Bicep registry.

18 of 33

The Destination

“Would you tell me, please, which way I ought to go from here?” asked Alice

“That depends a good deal on where you want to get to,” said the Cat.

“I don’t much care where—” said Alice.

“Then it doesn’t matter which way you go,” said the Cat.

18

19 of 33

Pipeline

19

main.bicep

DevOps Trigger

Pipeline YAML

env.bicepparam

main.bicep

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

Instantiated Azure Resources

common variables

YAML Templates

20 of 33

YAML Pipelines

20

All pipelines run the same steps:

Set some known parameters

Environment (sbx, dev, qap, prd)
Specify the “template” repository

Generate READ.ME documentation
Run a what-if scenario

Require user verification to continue

Execute main.bicep to deploy infrastructure

21 of 33

Pipelines – Azure DevOps YAML file

21

az deployment sub create

--location "$(Location)"

--template-file "$(BicepTemplate)"

--parameters "$(BicepParam)"

parameters:

- name: env_param

displayName: Environment

type: string

values:

- "sbx"

- "dev"

- "qap"

- "prd"

default: "sbx“

variables:

- name: ModuleDirectoryName

value: 'AppGateway’

- name: location

value: ’eastus'

- name: BicepTemplate

value: '$(System.DefaultWorkingDirectory)/${{ variables.ModuleDirectoryName }}/Components/main.bicep'

- name: BicepParam

value: '$(System.DefaultWorkingDirectory)/${{ variables.ModuleDirectoryName }}/Configuration/${{parameters.env_param}}.bicepparam'

22 of 33

Pipelines – Azure DevOps YAML Templates

22

variables:

- name: ModuleDirectoryName

value: 'AppGateway’

. . .

- template: templates/commonVars.yml@Utilities

�stages:

- ${{ if eq(parameters.bicepLinter, true) }}:

- template: templates/bicepLinter.yml@Utilities

- ${{ if eq(parameters.generateReadMe, true) }}:

- template: templates/generateReadMeInline.yml@Utilities

- ${{ if eq(parameters.runWhatIf, true) }}:

- template: templates/validateAndWhatIf.yml@Utilities

parameters:

validationTimeout: 1

- ${{ if eq(parameters.runDeploy, true) }}:

- template: templates/deployBicep.yml@Utilities

23 of 33

Bicep Registries

23

env.bicepparam

main.bicep

DevOps Trigger

Pipeline YAML

env.bicepparam

main.bicep

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

Instantiated Azure Resources

common variables

YAML Templates

24 of 33

Reusable Components

The Princeton Bicep Registry contains custom code modules. To fill the gaps from what is available in the Microsoft Bicep Registry.

24

module common ‘br:puoitabr.azurecr.io/common/sis:v0.1' = { …

or

module appGatewayKeyVault 'br/public:avm/security/keyvault:1.0.2' = { …

This is where our reusable components are stored.

Princeton’s Bicep Registry

Microsoft’s Bicep Registry

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

25 of 33

These bicep modules call a Microsoft resource

25

// Create the application gateway

resource applicationGateway 'Microsoft.Network/applicationGateways@2023-02-01' = {

location: location

tags: tags

identity: {

type: 'UserAssigned'

userAssignedIdentities: {

'${appGatewayIdentityId}':{}

}

properties: {

sku: sku

enableHttp2: false

autoscaleConfiguration: . . .

26 of 33

26

properties: {

sku: sku

enableHttp2: false

autoscaleConfiguration: autoscaleConfiguration

gatewayIPConfigurations:[

{

properties: {

subnet: {

id: gatewayIPConfigurationsSnetId

}

]

frontendIPConfigurations: [

{

� properties: {

privateIPAllocationMethod: 'Dynamic'

publicIPAddress: {

id: publicIpAddressId

}

]

frontendPorts: frontendPorts

backendAddressPools: backendAddressPools

backendHttpSettingsCollection: beHttpSettingsCollection

httpListeners: httpsListeners

sslCertificates: appGatewaySslCerts

urlPathMaps: urlPathMaps

requestRoutingRules: requestRoutingRules

rewriteRuleSets: rewriteRuleSets

probes: probes

}

This architecture provides a layer of abstraction between our Bicep pipelines and the Microsoft resource procedures.

In this layer we can hardcode standards and prefill immutable parameters.

27 of 33

main.bicep

27

main.bicep

DevOps Trigger

Pipeline YAML

env.bicepparam

main.bicep

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

Instantiated Azure Resources

common variables

YAML Templates

28 of 33

Configuration aka: bicepparam file

28

using '../Components/main.bicep'

// Environment Variables

param subscriptionId = readEnvironmentVariable('SUBSCRIPTIONID','error')

// Global Parameters

param dataClassification = 'restricted'

// Resource Group Parameters

param resourceGroupName = '${env}-sas-shared-infrastructure-rg'

// App Gateway Parameters

param appGatewayIdPredicate = '/subscriptions/${subscriptionId}/resourceGroups/

${resourceGroupName}/providers/Microsoft.Network/

applicationGateways/${appGatewayName}'

param appGatewayFrontendPorts = [

{

id: '${appGatewayIdPredicate}/frontendPorts/port_80'

properties: {

port: 80

}

{

id: '${appGatewayIdPredicate}/frontendPorts/port_443'

properties: {

port: 443

} . . .

29 of 33

main.bicep

29

// Create App Gateway

module appGateway ‘br:puoitabr.azurecr.io/network/app-gateway:v0.1' = {

dependsOn: [ common, publicIp, appGatewaySnet, appGatewayManagedIdentity]

scope: resourceGroup(resourceGroupName)

params: {

appGatewayName: appGatewayName

appGatewayIdentityId: appGatewayManagedIdentity.outputs.id

location: location

tags: tags

sku: sku

autoscaleConfiguration: appGatewayAutoscaleConfiguration

publicIpAddressId: publicIp.outputs.id

gatewayIPConfigurationsName: appGatewayIPConfigurationsName

gatewayIPConfigurationsSnetId: appGatewaySnet.id

frontendIPConfigurationsName: appGatewayFrontendIPConfigurationsName

frontendPorts: appGatewayFrontendPorts

backendAddressPools: union(appGatewayBackendAddressPools, appGatewayBackendAddressPoolsTigerSpeed)

backendHttpSettingsCollection: union(appGatewayBackendHttpSettingsCollection,

appGatewayBackendHttpSettingsCollectionTigerSpeed)

httpsListeners: union(appGatewayHttpsListeners, appGatewayHttpsListenersTigerSpeed)

appGatewaySslCerts: union(appGatewaySslCerts, appGatewaySslCertsTigerSpeed)

urlPathMaps: union(appGatewayUrlPathMaps, appGatewayUrlPathMapsTigerSpeed)

requestRoutingRules: union(appGatewayRequestRoutingRules, appGatewayRequestRoutingRulesTigerSpeed)

probes: union(appGatewayProbes, appGatewayProbesTigerSpeed)

rewriteRuleSets: union(appGatewayRewriteRuleSets, appGatewayRewriteRuleSetsTigerSpeed)

}

Calling our local Bicep Registry

Referencing the Configuration

(sbx.bicepparam file)

30 of 33

30

env.bicepparam

In a nutshell

main.bicep

DevOps Trigger

Pipeline YAML

env.bicepparam

main.bicep

Microsoft Public Bicep Registry

Princeton Private Bicep Registry

Instantiated Azure Resources

common variables

YAML Templates

31 of 33

31

Take Aways

Infrastructure as Code

Developers do not have access to QA and Production Azure portals. No “click-ops”.
Updates to Azure resources involves updating the IAC code and running the pipeline.
Identical code is executed in all environments eliminating “drift”.
Enforces a “role based” and “least privilege access” security model.
Elevated privileges are temporarily assigned on an “as needed” basis and require approval.
Enforces standards and conventions.
Improves troubleshooting and remediation efforts.

32 of 33

32

Take Aways

Coding Framework

Enforces high quality standards for coding practices.
Enforces compliance with DevOps CI/CD principles.
Effectively leverage reusable code. Write once, use everywhere.
More efficient development, debugging, maintenance and administration of software.
Simplifies and homogenizes the code which improves cross team collaboration.

33 of 33

33

Questions?