vSphere 6 Certificate Lifecycle Management

Recently I’ve been fighting with a vSphere environment and CA certificates and I thought a lot about certificate management and lifecycle in a VMware vSphere environment after that and how much it needs improvement. With the SSL Certificate Automation Tool VMware made a step in the right direction and even if the tools itself is sometimes a little buggy it is still very handy in automating a long and error prone process. In vSphere 6 VMware is taking another step in the right direction to help us create, apply and manage SSL certificates in a vSphere environment, but before talking about this we need to talk a bit about what’s new in SSO and vCenter architecture in vSphere 6. Since the introduction of SSO VMware changed its architecture in every major release, starting from 5.1 to 5.5 and now to 6.0 so let’s make a little bit of history:

Featured image

The new vSphere 6 management architecture introduces two main roles that you can deploy, these are the Management Node and the Platform Service Controller (PSC); the reason behind this separation is to have a logical entity that will take care of the main management features while another entity will hold the core and security features of the solution. What is nice about this separation is that you don’t need a 1:1 ratio between Management Nodes and PSCs so you can install PSC on separate boxes and replicate between them and then have as many Management nodes as you need (as long as you are within the same SSO domain)

Featured image

For HA scenario if you install PSC on separate boxes you will still need a load balancer. Supported solutions are Big-IP F5 and NetScaler so far.

You can obviously still install everything in one box:

Featured image

You might have noticed that the HA model for SSO was active/passive in 5.1, then active/active in 5.5 and now is active/passive again; this is due to the re-engineering of the Secure Token Service (STS) which is moving to a new and more robust method of STS (known as WebSSO) which is the same already used by vCAC (or vRealize Automation if you will) and that will be used from now forward instead of the old 5.5 method (WS-Trust). Let’s see how services are spread out on each role:

Let’s take a look to the services within the Management Node and the PSC:

Featured image

In the Management Node we can find services and features that every vSphere Admin feels very comfortable and familiar with such as vCenter Server, vSphere Web Client, Syslog Collector, etc., but two of them deserve a few words:

  • Virtual Datacenter Service: this service is new and it has been introduced to help mitigate the limitation connected with the Datacenter object in vCenter as a Management boundary.
  • (Optional) vPostgres: This component is obviously referring to the vCenter Appliance (thus optional) but I believe more and more new deployments or upgrades deserve to be considered a good fit for vCSA since VMware announced complete equality of features between vCenter installed on Windows and vCSA; leave alone the fuss of dedicating Windows licenses for vCenter which might not be a huge problem I just find the process of patching ad upgrading a vCSA simply amazing and it’s not a secret that products like EVO:RAIL make extensive use of vCSA. VMware wants to move all their services deployment model towards Virtual Appliances, this is not a secret and we need to get used to it, the sooner the better, but I’m digressing…

Featured image

In the Platform Service Controller or PSC we find our old friend SSO (we have had a rough past but now we are on better terms) and quite a few new services:

  • VMware Single Sign-On
    • Secure Token Service (STS)
    • Identity Management Service (IdM)
    • Directory Service (VMDir)
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • VMware Licensing Service
  • Authentication Framework Daemon (AFD)
  • Component Manager Service (CM)
  • HTTP Reverse Proxy

Describing all these services is out of the scope of this post but as you probably guess two of them will be our focus: the VMware Certificate Authority (or VMCA) and the VMware Endpoint Certificate Store (or VECS). But what are the roles of VMCA and VECS? The VMCA is no more or less than a CA, so you can:

  • Generate Certificates
  • Generate CRLs
  • Use the UI
  • Use the Command Line Interface to replace certificates

The VECS is where all certificates within the PSC are stored, with the only exception of the ESXi certificates that are stored locally on vSphere hosts, so here you can:

  • Store certificates and keys
  • Sync trusted certificates
  • Sync CRLs
  • Use the UI
  • Use the CLI to perform various actions

Since VMCA and VECS are part of the PSC, they will take advantage of the Multi-Master Replication Model which is offered by the Directory Service (VMDir) in order to achieve HA. In the past every service had its own user and required its own certificate but this is not the case anymore since we now have Solution Users (SU); since the number of services has increased significantly it would be impractical to manage the lifecycle of this many certificates so now we have 4 main SU that will hold the certificate used for a number of services.

Voila_Capture 2015-01-08_08-14-37_pm_white_background

What about use cases/scenarios in which I can implement VMCA? In what ways you can use this new tool?

Featured image

Scenario 1 and 2 are similar: the VMCA is the CA that releases certificates for all Solution Users (SU), the only difference is that in scenario 1 the VMCA is the root CA and you will need to distribute the Root CA Certificate so that all corporate browsers will trust it, while in scenario 2 the VMCA becomes part of an existing PKI as a subordinate CA and you certificate trust.

Featured image

In scenario 3 VMCA is installed but not used, CSRs are created and submitted to an external CA and VECS will be used to store certificates in PEM format.

Featured image

My favorite is scenario 2 because most enterprises I see already have a PKI (Microsoft CA usually) and all clients already trust the CA certificates, so adding the VMCA as s subordinate is a non disruptive process with a very low maintenance impact on the PKI itself, it protects investments already made to implement the current PKI and  preserves the knowledge to run the PKI.

Replacing certificates is still a CLI task (looks like Powershell will be involved) but VMCA and VECS are a very promising step toward the right direction for simplifying certificate lifecycle management in a vSphere environment.


ESXi 5.5 Update 2 build 2143827 break Citrix NetScaler

If you happen to run NetScaler and upgrade to build 2143827 of vSphere 5.5 you will run in very nasty issues.

Specifically, your NetScaler will go bonkers if you enter in the management web interface or even via SSH; in general all things that start a network increase in term of traffic directed to the NetScaler will make it crash.

When it doesn’t crash you will still experience network communication drops, hangs, connectivity issues and such and in a very random fashion too (just to add some fun).

Fortunately VMware gave us a workaround.

To work around this issue, add the line hw.em.txd=512 in loader.conf file.

To add the line hw.em.txd=512 in loader.conf file:

    1. Log in to the Citrix NetScaler virtual machine appliance as root
    2. Locate the loader.conf file on the Netscaler virtual machine appliance by running this command:find / -name loader.conf

Note: There are two loader.conf files. The two files are ./flash/boot/defaults/loader.conf and ./flash/boot/loader.conf.
Modify the first one and ensure to take a backup of the file prior to making the changes.

  1. Add this line in the loader.conf filehw.em.txd=512
  2. Save the changes
  3. Restart the NetScaler virtual machine appliance

For deeper details about the issue read KB VMware KB 2092809

Powershell Script for Shutting Down your vSphere Environment

Every now and then I need to setup an environment so that if a power outage occurs out of business hours there is a sort of automation taking care of that gracefully shutting down all VMs and Hosts to prevent failures.

For some reason I can never use older scripts I already have because whether they are too old and need some rewriting or because they don’t take into account some aspect of the specific environment I’m working on; one way or another it always feels like starting over.

While I was about to write a new one I stumbled across this blog post by Mike Preston and I was happy to find a very straightforward, simple and yet very complete script which would cover most of your (and mine too) needs.

Here are the aspects I was positively impressed about:

– it keeps into account HA
– it keeps into account DRS and stops it from moving things around while shutting down everything (nice)
– it allows you to specify if vCenter is virtual and it takes care of it for last, including the host where vCenter resides (nice!!)
– it keeps into account that VMs might just not shutdown gracefully for some reason; this is a major reason why most scripts would fail (see VDI streamed VMs)
– it writes events in a log file which is nice to go and look at in case you want to know what happened once you’re back online
– it dumps to file the list of powered on VMs at the moment of shutdown so you can manually turn everything back on as it was or take advantage of another nice script wrote by Mike called “poweronvms”

Ever if this script is definitely one of the most complete it is still lacking some of the requirements I needed, so with Mike permission (thanks Mike!) I added some things:

– added check for loading VMware PowerCLI Snap-In
– support for vApps
– removed annoying error messages when variables are = null
– added support for encrypted password dumped on a file (I don’t like password in clear text in scripts)
– added date-time in the log file for each event
– minor cosmetic in the output
– tested with vsphere 5.5 U1

Before we start using the script we need to create a password file so that passwords are not stored in clear text in the script itself:

mkdir c:\shutdown
Read-Host -Prompt "Enter password" -AsSecureString | ConvertFrom-SecureString | out-file c:\shutdown\cred.txt
(type in the password of the user you want to use for running the script)
cat c:\shutdown\cred.txt

The output should be something like this:


This is a hash of the password calculated using the current Windows credentials and it’s also connected with the Windows machine you run the command from, so this password file is not portable and can be used only but the user who generated it.

The script is already set up to use such password file, just edit the variables in the top of the script accordingly.

Here it is:

# Power off VMs (poweroffvms.ps1)
# This script does have a 'partner' script that powers the VMs back on, you can
# grab that script at http://blog.mwpreston.net/shares/
# Created By: Mike Preston, 2012 - With a whole lot of help from Eric Wright
#                                  (@discoposse)
# Variables:  $vcenter - The IP/DNS of your vCenter Server
#             $username/password - credentials for your vCenter Server
#             $filename - path to csv file to store powered on vms
#			  used for the poweronvms.ps1 script.
#             $cluster - Name of specific cluster to target within vCenter
#             $datacenter - Name of specific datacenter to target within vCenter

if ( (Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null )
	Add-PsSnapin VMware.VimAutomation.Core

# Some variables
$vcenter = "<vcenter_fqdn>"
$username = "<vcenter_username>"
$password = get-content C:\shutdown\cred.txt | convertto-securestring
$credentials = new-object System.Management.Automation.PSCredential $username, $password
$cluster = "<cluster_name>"
$datacenter = "<datacenter_name>"
$filename = "c:\shutdown\poweredonvms.csv"
$time = ( get-date ).ToString('HH-mm-ss')
$date = ( get-date ).ToString('dd-MM-yyyy')
$logfile = New-Item -type file "C:\shutdown\ShutdownLog-$date-$time.txt" -Force

# For use with a virtualized vCenter - bring it down last 🙂 - if you don't need this functionality
# simply leave the variable set to NA and the script will work as it always had
$vCenterVMName = "NA"

Write-Host ""
Write-Host "Shutdown command has been sent to the vCenter Server." -Foregroundcolor yellow
Write-Host "This script will shutdown all of the VMs and hosts located in $datacenter." -Foregroundcolor yellow
Write-Host "Upon completion, this server ($vcenter) will also be shutdown gracefully." -Foregroundcolor yellow
Write-Host ""
Sleep 5

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) PowerOff Script Engaged"
Add-Content $logfile ""

# Connect to vCenter
Write-Host "Connecting to vCenter - $vcenter.... " -nonewline
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Connecting to vCenter - $vcenter"
$success = Connect-VIServer $vcenter -Credential $credentials -WarningAction:SilentlyContinue
if ($success) { Write-Host "Connected!" -Foregroundcolor Green }
    Write-Host "Something is wrong, Aborting script" -Foregroundcolor Red
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Something is wrong, Aborting script"
Write-Host ""
Add-Content $logfile  ""

# Turn Off vApps
Write-Host "Stopping VApps...." -Foregroundcolor Green
Write-Host ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Stopping VApps..."
Add-Content $logfile ""

$vapps = Get-VApp | Where { $_.Status -eq "Started" }

if ($vapps -ne $null)
	ForEach ($vapp in $vapps)
			Write-Host "Processing $vapp.... " -ForegroundColor Green
			Write-Host ""
			Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Stopping $vapp."
			Add-Content $logfile ""
			Stop-VApp -VApp $vapp -Confirm:$false | out-null
			Write-Host "$vapp stopped." -Foregroundcolor Green
			Write-Host ""

Write-Host "VApps stopped." -Foregroundcolor Green
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) VApps stopped."
Add-Content $logfile ""

# Get a list of all powered on VMs - used for powering back on....
Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV $filename

# Change DRS Automation level to partially automated...
Write-Host "Changing cluster DRS Automation Level to Partially Automated" -Foregroundcolor green
Get-Cluster $cluster | Set-Cluster -DrsAutomation PartiallyAutomated -confirm:$false 

# Change the HA Level
Write-Host ""
Write-Host "Disabling HA on the cluster..." -Foregroundcolor green
Write-Host ""
Add-Content $logfile "Disabling HA on the cluster..."
Add-Content $logfile ""
Get-Cluster $cluster | Set-Cluster -HAEnabled:$false -confirm:$false 

# Get VMs again (we will do this again instead of parsing the file in case a VM was powered in the nanosecond that it took to get here.... 🙂
Write-Host ""
Write-Host "Retrieving a list of powered on guests...." -Foregroundcolor Green
Write-Host ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Retrieving a list of powered on guests...."
Add-Content $logfile ""
$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Checking to see if vCenter is virtualized"

# Retrieve host info for vCenter
if ($vcenterVMName -ne "NA")
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) vCenter is indeed virtualized, getting ESXi host hosting vCenter Server"
    $vCenterHost = (Get-VM $vCenterVMName).Host.Name
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $vCenterVMName currently running on $vCenterHost - will process this last"
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) vCenter is not virtualized"

Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Proceeding with VM PowerOff"
Add-Content $logfile ""

# And now, let's start powering off some guests....
ForEach ( $guest in $poweredonguests )
    if ($guest.Name -ne $vCenterVMName)
        Write-Host "Processing $guest.... " -ForegroundColor Green
        Write-Host "Checking for VMware tools install" -Foregroundcolor Green
        $guestinfo = get-view -Id $guest.ID
        if ($guestinfo.config.Tools.ToolsVersion -eq 0)
            Write-Host "No VMware tools detected in $guest , hard power this one" -ForegroundColor Yellow
            Write-Host ""
            Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - no VMware tools, hard power off"
            Stop-VM $guest -confirm:$false | out-null
           write-host "VMware tools detected.  I will attempt to gracefully shutdown $guest"
           Write-Host ""
           Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - VMware tools installed, gracefull shutdown"
           $vmshutdown = $guest | shutdown-VMGuest -Confirm:$false | out-null

# Let's wait a minute or so for shutdowns to complete
Write-Host ""
Write-Host "Giving VMs 2 minutes before resulting in hard poweroff"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Waiting a couple minutes then hard powering off all remaining VMs"
Sleep 120

# Now, let's go back through again to see if anything is still powered on and shut it down if it is
Write-Host "Beginning Phase 2 - anything left on.... night night...." -ForegroundColor red
Write-Host ""

# Get our list of guests still powered on...

$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }

if ($poweredonguests -ne $null)
	ForEach ( $guest in $poweredonguests )
		if ($guest.Name -ne $vCenterVMName)
			Write-Host "Processing $guest ...." -ForegroundColor Green
			#no checking for toosl, we just need to blast it down...
			write-host "Shutting down $guest - I don't care, it just needs to be off..." -ForegroundColor Yellow
                        Write-Host ""
			Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) $guest - Hard Power Off"
			Stop-VM $guest -confirm:$false | out-null

# Wait 30 seconds
Write-Host "Waiting 30 seconds and then proceding with host power off"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Processing power off of all hosts now"
Sleep 30

# and now its time to slam down the hosts - I've chosen to go by datacenter here but you could put the cluster
# There are some standalone hosts in the datacenter that I would also like to shutdown, those vms are set to
# start and stop with the host, so i can just shut those hosts down and they will take care of the vm shutdown 😉

$esxhosts = Get-VMHost -Location $cluster
foreach ($esxhost in $esxhosts)
    if ($esxhost.Name -ne $vCenterHost)
        #Shutem all down
        Write-Host "Shutting down $esxhost" -ForegroundColor Green
        Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Shutting down $esxhost"
        $esxhost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}

# Finally shut down the host which vCenter resides on if using that functionality
if ($vcenterVMName -ne "NA")
    Add-Content $logfile ""
    Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) Processing $vcenterHost - shutting down "
    Get-VMhost $vCenterHost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)} | out-null
	Write-Host "Shutdown Complete." -ForegroundColor Green

Add-Content $logfile ""
Add-Content $logfile "$(get-date -f dd/MM/yyyy) $(get-date -f HH:mm:ss) All done!"
# That's a wrap

Just to say something obvious, test!

I used vCenter Simulator 2.0 for initial testing and once happy I used a real environment.

Now go and turn off all the vSphere environments you can! (just kidding)

(Seriously, I’m just kidding, don’t do it! Really! Just don’t)

Upgrading/Installing vSphere 5.5 Update 1 skipping the NFS bug

Lately I had to deal with a customer who run into the NFS bug without even knowing it.

Most people updated vSphere to 5.5 update 1 to fix the Heartbleed bug (or maybe because it was just the latest version available to install) but then they had to deal with the other problems; I blame the Heartbeat bug for the rush out of update 1 which lead to the NFS bug. Just my 2 cents.

Anyway, sooner or later you will have to get there but you definitely don’t want to deal with NFS problems, so I figured I would write a small how-to for building a customized vSphere ISO that is already patched for Heartbleed and NFS bugs.

In case you just have to update you can use VMware Update Manger or do it manually, but if  you have to do a fresh install you will need a two steps approach (and still get the NFS bug before you patch) or install an older version and jump straight to the latest patched version; one way or another it would just be better to start with a patched version.

First of all you need PowerCLI and the latest patch (ESXi550-201406001.zip).

Then in my case I created a folder in “C:\vSpherePatches” and did as follows:

Add-EsxSoftwareDepot C:\vSpherePatches\ESXi550-201406001.zip     # Adding the patch file to the repository
Get-EsxImageProfile | fl     # Listing Image Profile in the repository
Export-EsxImageProfile -ImageProfile ESXi-5.5.0-20140604001-standard -ExportToIso -FilePath C:\vSpherePatches\VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso     # Exporting the standard Image Profile to an iso file

If you want to have the feel of an official VMware release you can name the iso file like I did: “VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso”. At least I have the feeling this would be what they would call it based on their naming convention.

Now you can install the latest build with no fear of incurring in (known) bugs. 😉

How to replace default VCSA 5.5 certificates with Microsoft CA signed certificates

DISCLAIMER: This is a very lenghty procedure and I’ve changed some steps from the original KB trying to make it shorter; if I made some mistakes please let me know.

I don’t do this all the time but today I had to replace SSL certificates on a vCenter Virtual Appliance and since I know this will happen more and more often I thought I should write a shorter procedure since VMware KB is very detailed and, yet again, very long. At least it’s not as long as the infamous 96 steps of version 5.1.

Before proceding it’s good practice to shutdown your vCSA and take a snapshot.

Go to http://vcenter_ip_address:5480 or http://fqdn:5480 and chack that the “Certificate regeneration enabled” setting in the Admin tab of the vCSA web interface is set to “No” or we will lose all our work at first reboot:


Also, since we are going to use a Microsoft CA for this tutorial, it would be a good idea to take a look at KB2062108 and complete those steps before proceeding.

Note: This procedure is specific for vCSA 5.5. If you have a previous version of vCSA please refer to KB2036744.

Download and install the latest build of OpenSSL 0.9.8 on a machine of your choice. For convenience I installed it on a Windows VM in “C:\OpenSSL”.

Create the following folders:


Open a text editor:

[ req ]
default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]
basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:vcva55, IP:, IP:ServerIPv6Address, DNS: vcva55.vmware.com

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = NY
localityName = New York
0.organizationName = VMware
organizationalUnitName = vCenterApplianceUniqueServer
commonName = vcva55.vmware.com

Change the following lines:

  • subjectAltName: insert here data about name and IP of your vCSA (you can omit IPv6 if you don’t use it)
  • commonName: this must be your vCSA FQDN
  • all section [req_distinguished_name]
  • leave organizationalUnitName as it is

Save the file as “C:\OpenSSL\Certs\openssl_generic.cfg”.

We need to generate one .cfg file for each service, changing the “organizationalUnitName” by opening the “openssl_generic.cfg” file we just created:

  • organizationalUnitName = VMware vCenter Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_vpxd.cfg”)
  • organizationalUnitName = VMware Inventory Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_inventoryservice.cfg”)
  • organizationalUnitName = VMware LogBrowser Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_logbrowser.cfg”)
  • organizationalUnitName = VMware vSphere Autodeploy Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_autodeploy.cfg”)

You should now have a .cfg file for each service in each folder with a different organizationalUnitName.

To generate the certificate requests, assuming you have the same path I have, you can use the following commands.

cd c:\OpenSSL\bin

openssl req -new -nodes -out c:\openssl\certs\vCenterSSO\rui_vpxd.csr -keyout c:\openssl\certs\vCenterSSO\rui_vpxd.key -config c:\openssl\certs\vCenterSSO\openssl_vpxd.cfg

openssl req -new -nodes -out c:\openssl\certs\InventoryService\rui_inventoryservice.csr -keyout c:\openssl\certs\InventoryService\rui_inventoryservice.key -config c:\openssl\certs\InventoryService\openssl_inventoryservice.cfg

openssl req -new -nodes -out c:\openssl\certs\LogBrowser\rui_logbrowser.csr -keyout c:\openssl\certs\LogBrowser\rui_logbrowser.key -config c:\openssl\certs\LogBrowser\openssl_logbrowser.cfg

openssl req -new -nodes -out c:\openssl\certs\AutoDeploy\rui_autodeploy.csr -keyout c:\openssl\certs\AutoDeploy\rui_autodeploy.key -config c:\openssl\certs\AutoDeploy\openssl_autodeploy.cfg

Now you should also have a .key file and a .csr file in each respective directory.

To generate certificates from the .csr file login your Microsoft CA web interface (by default it is http://servername/CertSrv/):

  1. Click the Request a certificate link.
  2. Click advanced certificate request.
  3. Click the Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file link.
  4. Open the certificate request (rui_service.csr, as generated above for each component) in a plain text editor and paste this text into the Saved Request box.
  5. Select the Certificate Template as VMware Certificate.
  6. Click Submit to submit the request.
  7. Click Base 64 encoded on the Certificate issued screen.
  8. Click the Download Certificate link.
  9. Save the certificate as rui_service.crt, in the appropriate C:\OpenSSL\Certs\<service>\ folder.  (for example rui_vpxd.crt)
  10. Repeat Steps 2 to 10 for each of the additional service.
  11. Navigate back to the home page of the certificate server and click Download a CA certificate, certificate chain or CRL.
  12. Click the Base 64 option.
  13. Click the Download CA Certificate chain link.
  14. Save the certificate chain as cachain.p7b in the c:\openssl\certs\ directory.

By default, Microsoft CA certificates are generated with the .cer format. Either use Save As or change it to .crt before continuing.

When complete, you have four certificates (rui_service.crt) for each of the services generated in their respective c:\openssl\certs\<services> folders and the cachain.p7b file in the c:\openssl\certs\ folder.

Copy the c:\openssl\certs folder on the root of the vCenter filesystem via SCP, rename it to “ssl”, SSH to the vCSA, then:

service vmware-stsd stop
service vmware-vpxd stop

Rename all files in the service folders so that the .key file is named “rui.key” and the .crt file is named “rui.crt”.

On the vCenter Appliance, move where the cachain.p7b file is, then convert it to cachain.pem:

openssl pkcs7 -print_certs -in cachain.p7b -out cachain.pem

Now open cachain.pem with a text editor and remove any text before the first “—–BEGIN CERTIFICATE—–” and after “—–END CERTIFICATE—–“.

Note: This assumes there are no intermediate certificates in the Certificate Authority.

Copy the cachain.pem file in every service folder.

cd <vcenterservicefolder>
cat rui.crt cachain.pem > chain.pem
/usr/sbin/vpxd_servicecfg certificate change chain.pem rui.key

If all goes well you should receive this:


Check KB2057248 if you get a different result.

service vmware-stsd start
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk

Create the chain.pem file for every service:

cat rui.crt cachain.pem > chain.pem


cd <inventoryservicefolder>
openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword
cp rui.key /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.crt /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.pfx /usr/lib/vmware-vpx/inventoryservice/ssl
cd /usr/lib/vmware-vpx/inventoryservice/ssl/
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
</em>rm /var/vmware/vpxd/inventoryservice_registered
service vmware-inventoryservice stop
service vmware-vpxd stop
service vmware-inventoryservice start
service vmware-vpxd start

Note: As there is a plain-text password on the above command, to avoid the history file showing the contents of the password because it is in plain text in the command above, run the unset HISTFILE command prior to executing any step containing a password.

Note: The default SSO administrator username for vCenter Single Sign-On 5.5 is administrator@vSphere.local

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk
cd <logbrowserservicefolder>
<code>openssl pkcs12 -export –out rui.pfx –in chain.pem -inkey rui.key –name rui –passout pass:testpassword</code>
cp rui.key /usr/lib/vmware-logbrowser/conf
cp rui.crt /usr/lib/vmware-logbrowser/conf
cp rui.pfx /usr/lib/vmware-logbrowser/conf
cd /usr/lib/vmware-logbrowser/conf
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
service vmware-logbrowser stop
service vmware-logbrowser start

In this environment the AutoDeploy service is not started so I’m skipping this step. (refer to KB2057223 to complete this step)

You can now restart the vCenter Server Appliance and chek that the certificates have been successfully replaced.


Related documents
Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223)
Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108)
Decoding a non-zero VC_CFG_RESULT for failed vpxd_servicecfg certificate changes (2057248)
Configuring certificates signed by a Certificate Authority (CA) for vCenter Server Appliance 5.1 (2036744)

vSphere 5.5 Update 1 is out with vSAN support!

What’s new

vCenter server 5.5 Update1

  • vCenter Server is now supported on Windows Server 2012 R2
  • vCloud® Hybrid Service™ vSphere® Client Plug-in, is now available in vSphere Web Client.


vSphere ESXi 5.5 Update1

  • VMware Virtual SAN  Virtual SAN 5.5 is a new hypervisor-converged storage tier that extends the vSphere Hypervisor to pool server-side magnetic disks (HDDs) and solid-state drives (SSDs). By clustering server-side HDDs and SSDs, Virtual SAN creates a distributed shared datastore designed and optimized for virtual environments. Virtual SAN is a standalone product that is sold separate from vSphere and requires its own license key


vSphere replication 5.5 Update1

  • VMware Virtual SAN is a fully supported feature of vSphere 5.5u1. You can use Virtual SAN in production environments with vSphere Replication 5.5.1 and vSphere 5.5u1.
  • Configure vSphere Replication on virtual machines that reside on VMware vSphere Flash Read Cache storage. vSphere Flash Read Cache is disabled on virtual machines after recovery.


vCenter Orchestrator 5.5 Update1

  • Workflow tagging
  • Plug-ins improvements
  • Version control system support
  • Dynamic Types (Beta)

Using ESXTOP with OSX Terminal application displays incorrectly

If are a VMware Admin switching from Windows to Mac you will find some limitations, even if lately VMware solved one of the major problems making the vSphere Web Client fully compatible with Mac in version 5.5.

Speaking about running ESXTOP from Mac, if you tried it you know you won’t be able to get any understandable data out of it and the reason seems to be the default  type of emulation that needs to be changed from “xterm-256color” to “xterm”.

Thanks to this post on the PunchingClouds Blog (bookmark it, it’s one of the good ones!) now this is not a problem anymore.

%d bloggers like this: