Automate ESXi Host Patching with Jenkins, PowerCLI & Zerto API

🚀 Overview & Problem Statement

Patching ESXi hosts in a production vSphere DRS cluster looks simple on paper but involves a precise sequence of steps that must all succeed before the host can go into maintenance mode. You need to switch DRS to Manual so the cluster does not fight your controlled migrations, disable all DRS VM-Host affinity rules that would block vMotions, verify that every destination host has enough free memory and CPU headroom to absorb the incoming workloads, and — if you are running Zerto for replication — gracefully evacuate the Virtual Replication Appliance (VRA) on that host so that replication continues uninterrupted on another host.

That is before the ESXCLI patching command even runs. After the reboot you need to verify the correct software profile is installed, take the host out of maintenance mode, re-enable all DRS rules, and return DRS to Fully Automated. Done manually, one host takes 45–60 minutes of active babysitting. With a 10-node cluster and a monthly patch cycle that is a full working day per site, every month.

This post walks through a fully parameterised PowerShell and PowerCLI script that handles the complete end-to-end lifecycle, driven by a Jenkins Parameterised Build. Engineers supply the target ESXi hostname, trigger the build, and walk away. Email notifications arrive at every key milestone. The Jenkins console streams live progress throughout.

🏗️ Automation Flow — All Three Phases

The script is structured into three gated phases. If any step fails, an alert email is sent and the script exits with code 1 so Jenkins marks the build FAILED immediately rather than continuing into a broken state.

✅ Prerequisites

📜 Script Phases — Deep Dive

# Gate 1 — TCP reachability before attempting login
$VcenterNetTest   = Test-NetConnection -ComputerName $vcserver -Port 443
$VcenterReachable = $VcenterNetTest.TcpTestSucceeded

if (-not $VcenterReachable) {
    Write-Host "ERROR: Cannot reach $vcserver on port 443. Exiting." -ForegroundColor Red
    Exit 1
}

# Gate 2 — Validate authenticated session
$VcenterConnection = Connect-VIServer -Server $vcserver -User $vcusername -Password $vcpassword
if (-not $VcenterConnection.IsConnected) {
    Write-Host "ERROR: Failed to connect to $vcserver. Check credentials." -ForegroundColor Red
    Exit 1
}
Write-Host "Connected to vCenter: $vcserver" -ForegroundColor Green

$Cluster     = Get-Cluster -Name $ClusterName
$Cluster_DRS = $Cluster.DrsEnabled

if ($Cluster_DRS -eq $true -and $Cluster.Name -like $ClusterName) {
    Write-Host "Setting DRS to Manual on cluster: $ClusterName" -ForegroundColor Yellow
    Set-Cluster -Cluster $ClusterName -DRSEnabled:$true `
                -DrsAutomationLevel "Manual" -Confirm:$false
}
else {
    Write-Host "DRS not enabled on $ClusterName or name mismatch. Exiting." -ForegroundColor Red
    Exit 1
}

# Disable VM-to-VM DRS rules (keep-apart / keep-together) — skip Zerto
Get-Cluster -Name $ClusterName | Get-DrsRule |
    Where-Object { $_.Name -notlike "*Zerto*" } |
    Set-DrsRule -Enabled:$false

# Disable VM-to-Host affinity/anti-affinity rules — skip Zerto
Get-Cluster -Name $ClusterName | Get-DrsVMHostRule |
    Where-Object { $_.Name -notlike "Zerto*" } |
    Set-DrsVMHostRule -Enabled:$false

Write-Host "Non-Zerto DRS rules disabled successfully." -ForegroundColor Green

# All powered-on VMs on the source host, excluding Zerto VRA appliances
$VMsInHost = Get-VMHost -Name $FromVMHostName | Get-VM |
             Where-Object { $_.PowerState -like "*On" -and $_.Name -notlike "Z-VRA*" }

foreach ($SingleVM in $VMsInHost) {

    # Candidate hosts: not source, not in maintenance, sorted least-loaded first
    $WorkingNodes = Get-Cluster -Name $ClusterName | Get-VMHost |
        Where-Object {
            $_.Name            -notlike $FromVMHostName -and
            $_.ConnectionState -notlike "*ain*"         -and  # excludes Maintenance
            $_.ConnectionState -notlike "NotResponding"  -and
            $_.ConnectionState -notlike "Unknown"
        } | Sort-Object -Property MemoryUsageGB

    $Migrated = $false

    foreach ($DestHost in $WorkingNodes) {

        # Projected memory % after placing this VM on the destination host
        [int]$ProjectedMemGB = $DestHost.MemoryUsageGB + $SingleVM.MemoryGB
        [int]$MemPercent      = $ProjectedMemGB /
                              (Get-VMHost -Name $DestHost.Name -ErrorAction Stop).MemoryTotalGB * 100
        [int]$CPUPercent      = $DestHost.CpuUsageMhz / $DestHost.CpuTotalMhz * 100

        if ($MemPercent -lt 85 -and $CPUPercent -lt 95) {
            Move-VM -VM $SingleVM.Name -Destination $DestHost.Name -ErrorAction Stop
            Write-Host "[$($SingleVM.Name)] moved to [$($DestHost.Name)]" -ForegroundColor Green
            $Migrated = $true
            break   # found a suitable host — move to the next VM
        }
    }

    if (-not $Migrated) {
        Write-Host "WARNING: No suitable host found for [$($SingleVM.Name)]" -ForegroundColor Red
    }
}

$ESXiCLI           = Get-EsxCli -VMHost $FromVMHostName -V2
$ESXiInMaintenance = $ESXiCLI.system.maintenanceMode.get.Invoke()

# Enumerate profiles available in the offline depot ZIP
$TargetProfileName = $ESXiCLI.software.sources.profile.list.Invoke(
                       @{ 'depot' = $PatchDepotPath }).Name
$CurrentProfile    = ($ESXiCLI.software.profile.get.Invoke()).Name
$ExpectedPostPatch = "(Updated) $TargetProfileName"

if ($ESXiInMaintenance -eq "Enabled" -and $ExpectedPostPatch -ne $CurrentProfile) {

    # 'update' preserves VIB configuration — preferred over 'install' for incremental patches
    $UpdateResult = $ESXiCLI.software.profile.update.invoke(@{
        'depot'   = $PatchDepotPath
        'profile' = $TargetProfileName
    })
    Write-Host "Result: $($UpdateResult.Message)"
    Write-Host "Reboot required: $($UpdateResult.RebootRequired)"

    if ($UpdateResult.RebootRequired -eq $true) {
        Restart-VMhost -VMHost $FromVMHostName -Force:$true -Confirm:$false
    }
}

# Brief sleep to allow shutdown to begin before polling
Start-Sleep -Seconds 60

# Poll ICMP until the host responds after reboot
do {
    $HostPingable = Test-Connection -ComputerName $FromVMHostName -Quiet
} until ($HostPingable -eq $true)

Write-Host "$FromVMHostName is pingable. Waiting for ESXi services..."
Start-Sleep -Seconds 120  # Allow all ESXi services to fully initialise

# Re-verify installed profile after reboot
$ESXiCLI_Post     = Get-EsxCli -VMHost $FromVMHostName -V2
$PostPatchProfile = ($ESXiCLI_Post.software.profile.get.Invoke()).Name

if ($PostPatchProfile -eq $ExpectedPostPatch) {
    Write-Host "Verified: $FromVMHostName running $PostPatchProfile" -ForegroundColor Green
} else {
    Write-Host "WARNING: Expected [$ExpectedPostPatch] — Found [$PostPatchProfile]" -ForegroundColor Red
}

# Restore DRS — re-enable all non-Zerto rules, return to Fully Automated
Set-Cluster -Cluster $ClusterName -DRSEnabled:$true -DrsAutomationLevel "FullyAutomated" -Confirm:$false
Get-Cluster -Name $ClusterName | Get-DrsRule       | Where-Object { $_.Name -notlike "*Zerto*" } | Set-DrsRule -Enabled:$true
Get-Cluster -Name $ClusterName | Get-DrsVMHostRule | Where-Object { $_.Name -notlike "Zerto*"  } | Set-DrsVMHostRule -Enabled:$true

# Exit maintenance mode and reconnect host to cluster
Set-VMHost -VMHost $FromVMHostName -State Connected
Disconnect-VIServer -Server $vcserver -Confirm:$false

🔁 Zerto VRA Evacuation via REST API

If a host running a Zerto VRA goes into maintenance mode without first evacuating that VRA, all Virtual Protection Groups (VPGs) anchored to it enter an error state and replication is suspended. The Zerto REST API provides a changerecoveryvra/execute endpoint that migrates a VRA's protected VPGs to another VRA before the host goes offline.

Authentication — Get-ZertoSession

The script authenticates to the ZVM using HTTP Basic authentication over HTTPS on port 9669. The response header contains an x-zerto-session token that must be included in the header of every subsequent API call.

function Get-ZertoSession {
    param([string]$ZvmUser, [string]$ZvmPassword)

    $sessionURI = "https://$strZVMIP`:$strZVMPort/v1/session/add"

    # Encode credentials as Base64 for Basic auth header
    $authBytes  = [System.Text.Encoding]::UTF8.GetBytes("$ZvmUser`:$ZvmPassword")
    $authBase64 = [System.Convert]::ToBase64String($authBytes)
    $authHeader = @{ Authorization = "Basic $authBase64" }

    $response = Invoke-WebRequest -Uri $sessionURI -Headers $authHeader `
                    -Method POST -Body '{"AuthenticationMethod": "1"}' `
                    -ContentType "application/json" -SkipCertificateCheck

    # The session token is returned in the response header, not the body
    return $response.Headers.get_item("x-zerto-session")
}

Discover Your VRA Identifiers

Each VRA has a unique VraIdentifier string in Zerto. The script stores these in a hashtable keyed by ESXi host FQDN — adding a new host is a single line. Run the snippet below once against your ZVM to discover the identifiers for your environment:

$xZertoSession = Get-ZertoSession -ZvmUser $strZVMUser -ZvmPassword $strZVMPwd
$ZertoHeader   = @{ "x-zerto-session" = $xZertoSession }
$vraListURI    = "https://$strZVMIP`:$strZVMPort/v1/vras"

$vrasList = Invoke-RestMethod -Uri $vraListURI -Headers $ZertoHeader `
                -ContentType "application/json" -SkipCertificateCheck

# Outputs: HostDisplayName (matches ESXi FQDN) and VraIdentifier
$vrasList | Select-Object HostDisplayName, VraIdentifier | Format-Table -AutoSize

Hashtable Configuration — One Line Per Host

$ZertoVraIds = @{
    "esx101.yourdomain.local" = "2724388199262172525"
    "esx102.yourdomain.local" = "2724388199262170715"
    "esx103.yourdomain.local" = "2724388199262168037"
    "esx104.yourdomain.local" = "2724388199262150817"
    # Add one line per host — no other code changes needed
}

Evacuation Execution and Progress Polling

The evacuation endpoint returns a task ID. The script polls the task progress endpoint twice: first after 90 seconds, then again after a further 120 seconds if the task is not yet at 100%. If it still has not completed, the script exits with Exit 1 and sends an alert email rather than proceeding to maintenance mode. This prevents the host going dark while VPGs are mid-migration between VRAs.

# POST evacuation task — response is the task ID string
$executeURI = "https://$strZVMIP`:$strZVMPort/v1/vras/$VraId/changerecoveryvra/execute"
$taskId     = Invoke-RestMethod -Uri $executeURI -Headers $ZertoHeader `
                  -Body ($EvacuatePayload | ConvertTo-Json) `
                  -ContentType "application/json" -Method Post -SkipCertificateCheck
$taskURI    = "https://$strZVMIP`:$strZVMPort/v1/tasks/$taskId"

# Poll 1 — wait 90 seconds and check progress
Start-Sleep -Seconds 90
$p1 = (Invoke-RestMethod -Uri $taskURI -Headers $ZertoHeader `
           -ContentType "application/json" -SkipCertificateCheck).Status.Progress
if ($p1 -eq 100) { return $true }

# Poll 2 — wait an additional 120 seconds before giving up
Start-Sleep -Seconds 120
$p2 = (Invoke-RestMethod -Uri $taskURI -Headers $ZertoHeader `
           -ContentType "application/json" -SkipCertificateCheck).Status.Progress
if ($p2 -ne 100) {
    Write-Host "ERROR: VRA evacuation did not complete. Check ZVM." -ForegroundColor Red
    return $false   # caller sends alert email + Exit 1
}

📄 Evacuate.json — Payload Explained

The evacuation POST request body is loaded from a JSON file at C:\Patching\Evacuate.json on the Jenkins Windows agent. This file tells the Zerto API how to re-assign the VMs protected by the source VRA to other VRAs/hosts in the same Zerto site.

Correct Payload Structure

The Zerto API expects a vmsAllocations array. Each entry in the array maps a protected VM identifier (vmIdentifier) to a target host identifier (hostIdentifier). When both are set to null, Zerto automatically selects the best available VRA on another host — this is the recommended approach for routine host maintenance because it lets Zerto's placement logic decide.

{
  "vmsAllocations": [
    {
      "hostIdentifier": null,
      "vmIdentifier"  : null
    }
  ]
}

Optional: Target a Specific Host

If you need to direct all VPGs to a specific target host rather than letting Zerto choose, replace null with the Zerto host identifier for the target ESXi host. Retrieve host identifiers from GET /v1/virtualizationsites/{siteId}/hosts on your ZVM.

{
  "vmsAllocations": [
    {
      "hostIdentifier": "host-identifier-from-zerto-api",
      "vmIdentifier"  : null
    }
  ]
}

How the Script Loads It

# Load the evacuation payload from the JSON file on the Jenkins agent
$EvacuatePayload = Get-Content -Raw "C:\Patching\Evacuate.json" | ConvertFrom-Json

# The payload is then passed as the request body to the Zerto evacuation endpoint
# ConvertTo-Json serialises it back to a JSON string for Invoke-RestMethod
$taskId = Invoke-RestMethod -Uri $executeURI -Headers $ZertoHeader `
               -Body ($EvacuatePayload | ConvertTo-Json) `
               -ContentType "application/json" -Method Post -SkipCertificateCheck

🔧 Jenkins Setup — Parameterised Build

Running this from Jenkins gives you encrypted credentials, a complete audit trail of every patch run, real-time console output, archived transcript logs as build artefacts, and the ability to trigger or schedule without ever touching the script. Here is the exact configuration.

Set-ExecutionPolicy Bypass -Scope Process -Force
& "C:\Patching\VMware-ESXi-PrePatching-Automation.ps1"

📋 Build Parameters & Script Configuration Reference

The Jenkins build form exposes one user-typed parameter. All other values are configured in the script configuration block at the top of the file — update them once for your environment and they apply to every build run.

Jenkins Parameters

Script Configuration Block — Update Once Per Environment

🌟 Benefits

💡 Pro Tips & Best Practices

🔑 Use a Dedicated vCenter Service Account

Create a dedicated service account such as svc-vsphere@automatewithravi.com with the minimum vCenter roles: Virtual Machine → Migrate, Host → Maintenance, and Cluster → Modify. Never use a personal or domain admin account. When credentials rotate, update one Jenkins Secret Text entry — not hunt through scripts.

📅 Check VPG SLA Status Before Evacuating

Before triggering VRA evacuation, call GET /v1/vpgs and confirm all VPGs on the target host are in MeetingSLA status. Evacuating a VRA when a VPG is already degraded risks losing a recovery point. A quick pre-flight check saves a much longer incident conversation.

🔔 Add a Pre-Run Start Notification

The script sends notifications at completion milestones. Add one at the very start — before any DRS changes — so on-call engineers know a patching run has begun:

Send-Notification `
    -Subject ("ESXi Patching STARTED: $FromVMHostName — " + (Get-Date -Format dd-MM-yyyy)) `
    -Body "Hi Team,`n`nAutomated ESXi patching has started on $FromVMHostName.`nDRS is being set to Manual and VM migrations will begin shortly.`n`nThis is an automated notification."

⏱️ Adjust Zerto Polling for Larger Environments

The script polls Zerto at 90 seconds then 120 seconds. If your VRA protects many VPGs, evacuation can take longer. Increase the sleep durations or add a third polling attempt before Exit 1 to avoid failing a successful-but-slow evacuation.

📦 GitHub Repository

The complete sanitised script — VMware-ESXi-PrePatching-Automation.ps1 — along with the example Evacuate.json payload and notes on discovering your Zerto VRA identifiers, is published in the VMware automation repository on GitHub.

Clone it, update $vcserver, $ClusterName, $ZertoVraIds, $PatchDepotPath, and the SMTP settings, store your vCenter credentials in Jenkins, place Evacuate.json at C:\Patching\ on the Windows agent, and trigger your first parameterised patching build.