Powershell Script for Shutting Down your vSphere Environment

Every now and then I need to setup an environment so that if a power outage occurs out of business hours there is a sort of automation taking care of that gracefully shutting down all VMs and Hosts to prevent failures.

For some reason I can never use older scripts I already have because whether they are too old and need some rewriting or because they don’t take into account some aspect of the specific environment I’m working on; one way or another it always feels like starting over.

While I was about to write a new one I stumbled across this blog post by Mike Preston and I was happy to find a very straightforward, simple and yet very complete script which would cover most of your (and mine too) needs.

Here are the aspects I was positively impressed about:

– it keeps into account HA
– it keeps into account DRS and stops it from moving things around while shutting down everything (nice)
– it allows you to specify if vCenter is virtual and it takes care of it for last, including the host where vCenter resides (nice!!)
– it keeps into account that VMs might just not shutdown gracefully for some reason; this is a major reason why most scripts would fail (see VDI streamed VMs)
– it writes events in a log file which is nice to go and look at in case you want to know what happened once you’re back online
– it dumps to file the list of powered on VMs at the moment of shutdown so you can manually turn everything back on as it was or take advantage of another nice script wrote by Mike called “poweronvms”

Ever if this script is definitely one of the most complete it is still lacking some of the requirements I needed, so with Mike permission (thanks Mike!) I added some things:

– added check for loading VMware PowerCLI Snap-In
– support for vApps
– removed annoying error messages when variables are = null
– added support for encrypted password dumped on a file (I don’t like password in clear text in scripts)
– added date-time in the log file for each event
– minor cosmetic in the output
– tested with vsphere 5.5 U1

Before we start using the script we need to create a password file so that passwords are not stored in clear text in the script itself:

mkdir c:\shutdown
Read-Host -Prompt "Enter password" -AsSecureString | ConvertFrom-SecureString | out-file c:\shutdown\cred.txt
(type in the password of the user you want to use for running the script)
cat c:\shutdown\cred.txt

The output should be something like this:

01000000d08c9ddf0115d1118c7a00c04fc297eb0100000092295625f5c35b4bb8af99be46e6679d0000000002000000000003660000c0000000100
000007b8d21c33686ef6ec751c0f671e7ff510000000004800000a00000001000000022d4534785ecda5799047d48f0f79a4618000000a1376ef3bf
fa5a3a928bee66d68eab5d49b12308f9d0b63f140000002ca923ea8eb54ff11c5864d9f722931a4ec85597

This is a hash of the password calculated using the current Windows credentials and it’s also connected with the Windows machine you run the command from, so this password file is not portable and can be used only but the user who generated it.

The script is already set up to use such password file, just edit the variables in the top of the script accordingly.

Here it is:

#################################################################################
# Power off VMs (poweroffvms.ps1)
#
# This script does have a 'partner' script that powers the VMs back on, you can
# grab that script at http://blog.mwpreston.net/shares/
#
# Created By: Mike Preston, 2012 - With a whole lot of help from Eric Wright
#                                  (@discoposse)
#
# Variables:  $vcenter - The IP/DNS of your vCenter Server
#             $username/password - credentials for your vCenter Server
#             $filename - path to csv file to store powered on vms
#			  used for the poweronvms.ps1 script.
#             $cluster - Name of specific cluster to target within vCenter
#             $datacenter - Name of specific datacenter to target within vCenter
#
#################################################################################

if ( (Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null )
{
	Add-PsSnapin VMware.VimAutomation.Core
}

# Some variables
$vcenter = "<vcenter_fqdn>"
$username = "<vcenter_username>"
$password = get-content C:\shutdown\cred.txt | convertto-securestring
$credentials = new-object System.Management.Automation.PSCredential $username, $password
$cluster = "<cluster_name>"
$datacenter = "<datacenter_name>"
$filename = "c:\shutdown\poweredonvms.csv"
$time = ( get-date ).ToString('HH-mm-ss')
$date = ( get-date ).ToString('dd-MM-yyyy')
$logfile = New-Item -type file "C:\shutdown\ShutdownLog-$date-$time.txt" -Force

# For use with a virtualized vCenter - bring it down last 🙂 - if you don't need this functionality
# simply leave the variable set to NA and the script will work as it always had
$vCenterVMName = "NA"

Write-Host ""
Write-Host "Shutdown command has been sent to the vCenter Server." -Foregroundcolor yellow
Write-Host "This script will shutdown all of the VMs and hosts located in $datacenter." -Foregroundcolor yellow
Write-Host "Upon completion, this server ($vcenter) will also be shutdown gracefully." -Foregroundcolor yellow
Write-Host ""
Sleep 5

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) PowerOff Script Engaged"
Add-Content $logfile ""

# Connect to vCenter
Write-Host "Connecting to vCenter - $vcenter.... " -nonewline
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Connecting to vCenter - $vcenter"
$success = Connect-VIServer $vcenter -Credential $credentials -WarningAction:SilentlyContinue
if ($success) { Write-Host "Connected!" -Foregroundcolor Green }
else
{
    Write-Host "Something is wrong, Aborting script" -Foregroundcolor Red
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Something is wrong, Aborting script"
    exit
}
Write-Host ""
Add-Content $logfile  ""

# Turn Off vApps
Write-Host "Stopping VApps...." -Foregroundcolor Green
Write-Host ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Stopping VApps..."
Add-Content $logfile ""

$vapps = Get-VApp | Where { $_.Status -eq "Started" }

if ($vapps -ne $null)
{
	ForEach ($vapp in $vapps)
	{
			Write-Host "Processing $vapp.... " -ForegroundColor Green
			Write-Host ""
			Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Stopping $vapp."
			Add-Content $logfile ""
			Stop-VApp -VApp $vapp -Confirm:$false | out-null
			Write-Host "$vapp stopped." -Foregroundcolor Green
			Write-Host ""
	}
}

Write-Host "VApps stopped." -Foregroundcolor Green
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) VApps stopped."
Add-Content $logfile ""

# Get a list of all powered on VMs - used for powering back on....
Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV $filename

# Change DRS Automation level to partially automated...
Write-Host "Changing cluster DRS Automation Level to Partially Automated" -Foregroundcolor green
Get-Cluster $cluster | Set-Cluster -DrsAutomation PartiallyAutomated -confirm:$false 

# Change the HA Level
Write-Host ""
Write-Host "Disabling HA on the cluster..." -Foregroundcolor green
Write-Host ""
Add-Content $logfile "Disabling HA on the cluster..."
Add-Content $logfile ""
Get-Cluster $cluster | Set-Cluster -HAEnabled:$false -confirm:$false 

# Get VMs again (we will do this again instead of parsing the file in case a VM was powered in the nanosecond that it took to get here.... 🙂
Write-Host ""
Write-Host "Retrieving a list of powered on guests...." -Foregroundcolor Green
Write-Host ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Retrieving a list of powered on guests...."
Add-Content $logfile ""
$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Checking to see if vCenter is virtualized"

# Retrieve host info for vCenter
if ($vcenterVMName -ne "NA")
{
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) vCenter is indeed virtualized, getting ESXi host hosting vCenter Server"
    $vCenterHost = (Get-VM $vCenterVMName).Host.Name
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $vCenterVMName currently running on $vCenterHost - will process this last"
}
else
{
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) vCenter is not virtualized"
}

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Proceeding with VM PowerOff"
Add-Content $logfile ""

# And now, let's start powering off some guests....
ForEach ( $guest in $poweredonguests )
{
    if ($guest.Name -ne $vCenterVMName)
    {
        Write-Host "Processing $guest.... " -ForegroundColor Green
        Write-Host "Checking for VMware tools install" -Foregroundcolor Green
        $guestinfo = get-view -Id $guest.ID
        if ($guestinfo.config.Tools.ToolsVersion -eq 0)
        {
            Write-Host "No VMware tools detected in $guest , hard power this one" -ForegroundColor Yellow
            Write-Host ""
            Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - no VMware tools, hard power off"
            Stop-VM $guest -confirm:$false | out-null
        }
        else
        {
           write-host "VMware tools detected.  I will attempt to gracefully shutdown $guest"
           Write-Host ""
           Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - VMware tools installed, gracefull shutdown"
           $vmshutdown = $guest | shutdown-VMGuest -Confirm:$false | out-null
        }
    }
}

# Let's wait a minute or so for shutdowns to complete
Write-Host ""
Write-Host "Giving VMs 2 minutes before resulting in hard poweroff"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Waiting a couple minutes then hard powering off all remaining VMs"
Sleep 120

# Now, let's go back through again to see if anything is still powered on and shut it down if it is
Write-Host "Beginning Phase 2 - anything left on.... night night...." -ForegroundColor red
Write-Host ""

# Get our list of guests still powered on...

$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }

if ($poweredonguests -ne $null)
{
	ForEach ( $guest in $poweredonguests )
	{
		if ($guest.Name -ne $vCenterVMName)
		{
			Write-Host "Processing $guest ...." -ForegroundColor Green
			#no checking for toosl, we just need to blast it down...
			write-host "Shutting down $guest - I don't care, it just needs to be off..." -ForegroundColor Yellow
                        Write-Host ""
			Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - Hard Power Off"
			Stop-VM $guest -confirm:$false | out-null
		}
	}
}

# Wait 30 seconds
Write-Host "Waiting 30 seconds and then proceding with host power off"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Processing power off of all hosts now"
Sleep 30

# and now its time to slam down the hosts - I've chosen to go by datacenter here but you could put the cluster
# There are some standalone hosts in the datacenter that I would also like to shutdown, those vms are set to
# start and stop with the host, so i can just shut those hosts down and they will take care of the vm shutdown 😉

$esxhosts = Get-VMHost -Location $cluster
foreach ($esxhost in $esxhosts)
{
    if ($esxhost.Name -ne $vCenterHost)
    {
        #Shutem all down
        Write-Host "Shutting down $esxhost" -ForegroundColor Green
        Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Shutting down $esxhost"
        $esxhost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}
    }
}

# Finally shut down the host which vCenter resides on if using that functionality
if ($vcenterVMName -ne "NA")
{
    Add-Content $logfile ""
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Processing $vcenterHost - shutting down "
    Get-VMhost $vCenterHost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)} | out-null
	Write-Host "Shutdown Complete." -ForegroundColor Green
}

Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) All done!"
# That's a wrap

Just to say something obvious, test!

I used vCenter Simulator 2.0 for initial testing and once happy I used a real environment.

Now go and turn off all the vSphere environments you can! (just kidding)

(Seriously, I’m just kidding, don’t do it! Really! Just don’t)

Upgrading/Installing vSphere 5.5 Update 1 skipping the NFS bug

Lately I had to deal with a customer who run into the NFS bug without even knowing it.

Most people updated vSphere to 5.5 update 1 to fix the Heartbleed bug (or maybe because it was just the latest version available to install) but then they had to deal with the other problems; I blame the Heartbeat bug for the rush out of update 1 which lead to the NFS bug. Just my 2 cents.

Anyway, sooner or later you will have to get there but you definitely don’t want to deal with NFS problems, so I figured I would write a small how-to for building a customized vSphere ISO that is already patched for Heartbleed and NFS bugs.

In case you just have to update you can use VMware Update Manger or do it manually, but if  you have to do a fresh install you will need a two steps approach (and still get the NFS bug before you patch) or install an older version and jump straight to the latest patched version; one way or another it would just be better to start with a patched version.

First of all you need PowerCLI and the latest patch (ESXi550-201406001.zip).

Then in my case I created a folder in “C:\vSpherePatches” and did as follows:

Add-EsxSoftwareDepot C:\vSpherePatches\ESXi550-201406001.zip     # Adding the patch file to the repository
Get-EsxImageProfile | fl     # Listing Image Profile in the repository
Export-EsxImageProfile -ImageProfile ESXi-5.5.0-20140604001-standard -ExportToIso -FilePath C:\vSpherePatches\VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso     # Exporting the standard Image Profile to an iso file

If you want to have the feel of an official VMware release you can name the iso file like I did: “VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso”. At least I have the feeling this would be what they would call it based on their naming convention.

Now you can install the latest build with no fear of incurring in (known) bugs. 😉

How to upgrade Horizon Workspace 1.0 to 1.5

Since a new version of Horizon Workspace came out I’ve been curious about differences and improvements, so i upgraded it to version 1.5.

I have to say that, besides some differences here and there in the look and feel and the new functionality to manage VMware Ready Android devices, I have already found a nice improvement in the way you mount the NFS store in the data-va because finally the procedure simply works and there’s no need of crazy workarounds. I also suspect that they improved a lot the way you change self-signed certificates compared to what it used to be; more on this when i will have tested it.

Note: You can only upgrade if you have completed the setup procedure and you have a fully functional environment.

All commands need to be run from the configurator-va, just as usual, so let’s SSH into it, then:

su -
/usr/local/horizon/lib/menu/updatemgr.hzn check

You should get this output:

Image

As you can see it won’t recognize my vPostgres and this is normal. Since I’ve started this series of posts about Horizon Workspace installing the external database on vPostgres i also added the VM to the VApp to make sure it starts for first and stop for last. The first time i tried to upgrade i decided to ignore this error and went ahead with the upgrade which failed and left my Horizon Workspace installation in an unusable state so i reverted the snapshots, removed the vPostgres from the VApp and rerun the command and i could see no issue at that point.

Before going on and since it’s been so useful for me, i will show you how to snapshot all your VMs in the VApp at once so you can revert if you need. Since you might have more than just 5 VMs i like to use PowerCLI to take over this kind of task simply because it’s faster than working around the GUI:

Get-VApp Horizon-Workspace | Get-VM | New-Snapshot -Name "UpgradeHorizon" -Description "Horizon Workspace upgrade from version 1.0 to 1.5" -Memory -Quiesce -Confirm:$false -RunAsync

Given that you didn’t change the default name of the VApp (Horizon-Workspace) this will snapshot all the VMs included in it.

In the vSphere client you should see this:

Image

Once they are all completed we can go back to the configurator-va and start the upgrade with peace of mind:

/usr/local/horizon/lib/menu/updatemgr.hzn update

Now you should see a whole lot of downloading and installing. In my lab it took about 40 minutes for the whole procedure.

When the upgrade is done you have to shutdown and restart the VApp; since i still have the PowerCLI open i used it:

Stop-VApp -VApp Horizon-Workspace -Confirm:$false
Start-VApp -VApp Horizon-Workspace -Confirm:$false

You shouldn’t start it again until it’s stopped, so in order to see the status of the command i’ve omitted “RunAsync”.

Now check that your Horizon Workspace environment still works then you can check again for updates to see the current status:

Image

That looks al good, now you just have to remember to update your preview binaries in the data-va if you used the LibreOffice preview because there’s a new version, just do as usual.

If you’re happy with the result you can remove the snapshots we created earlier:

Get-VApp Horizon-Workspace | Get-VM | Get-SnapShot | Where { $_.Name.Contains("ToolsUpgrade") } | Remove-Snapshot -RemoveChildren -Confirm:$false -RunAsync

In my case i also had to put back the vPostgres VM in the VApp.

That’s pretty much it.

DISCLAIMER: PowerCLI is a powerful tool and should not be used to throw in some commands without understanding the effects. Before running PowerCLI commands in your production environment always test them and adjust them for your situation. I take no responsibility if you don’t test things out yourself first and your Horizon Workspace gets broken.

UPDATE: In the release notes it is clearly stated that in the documentation there are missing steps in case you have multiple service-va VMs. Please refer to the Horizon Workspace 1.5 release notes for the additional steps.

%d bloggers like this: