Upgrading/Installing vSphere 5.5 Update 1 skipping the NFS bug

Lately I had to deal with a customer who run into the NFS bug without even knowing it.

Most people updated vSphere to 5.5 update 1 to fix the Heartbleed bug (or maybe because it was just the latest version available to install) but then they had to deal with the other problems; I blame the Heartbeat bug for the rush out of update 1 which lead to the NFS bug. Just my 2 cents.

Anyway, sooner or later you will have to get there but you definitely don’t want to deal with NFS problems, so I figured I would write a small how-to for building a customized vSphere ISO that is already patched for Heartbleed and NFS bugs.

In case you just have to update you can use VMware Update Manger or do it manually, but if  you have to do a fresh install you will need a two steps approach (and still get the NFS bug before you patch) or install an older version and jump straight to the latest patched version; one way or another it would just be better to start with a patched version.

First of all you need PowerCLI and the latest patch (ESXi550-201406001.zip).

Then in my case I created a folder in “C:\vSpherePatches” and did as follows:

Add-EsxSoftwareDepot C:\vSpherePatches\ESXi550-201406001.zip     # Adding the patch file to the repository
Get-EsxImageProfile | fl     # Listing Image Profile in the repository
Export-EsxImageProfile -ImageProfile ESXi-5.5.0-20140604001-standard -ExportToIso -FilePath C:\vSpherePatches\VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso     # Exporting the standard Image Profile to an iso file

If you want to have the feel of an official VMware release you can name the iso file like I did: “VMware-VMvisor-Installer-5.5.0.update01-1881737.x86_64.iso”. At least I have the feeling this would be what they would call it based on their naming convention.

Now you can install the latest build with no fear of incurring in (known) bugs. 😉

How to replace default VCSA 5.5 certificates with Microsoft CA signed certificates

DISCLAIMER: This is a very lenghty procedure and I’ve changed some steps from the original KB trying to make it shorter; if I made some mistakes please let me know.

I don’t do this all the time but today I had to replace SSL certificates on a vCenter Virtual Appliance and since I know this will happen more and more often I thought I should write a shorter procedure since VMware KB is very detailed and, yet again, very long. At least it’s not as long as the infamous 96 steps of version 5.1.

Before proceding it’s good practice to shutdown your vCSA and take a snapshot.

Go to http://vcenter_ip_address:5480 or http://fqdn:5480 and chack that the “Certificate regeneration enabled” setting in the Admin tab of the vCSA web interface is set to “No” or we will lose all our work at first reboot:

1

Also, since we are going to use a Microsoft CA for this tutorial, it would be a good idea to take a look at KB2062108 and complete those steps before proceeding.

Note: This procedure is specific for vCSA 5.5. If you have a previous version of vCSA please refer to KB2036744.

Download and install the latest build of OpenSSL 0.9.8 on a machine of your choice. For convenience I installed it on a Windows VM in “C:\OpenSSL”.

Create the following folders:

C:\OpenSSL\Certs
C:\OpenSSL\Certs\vCenterSSO
C:\OpenSSL\Certs\InventoryService
C:\OpenSSL\Certs\LogBrowser
C:\OpenSSL\Certs\AutoDeploy

Open a text editor:

[ req ]
default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]
basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:vcva55, IP: 10.0.0.10, IP:ServerIPv6Address, DNS: vcva55.vmware.com

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = NY
localityName = New York
0.organizationName = VMware
organizationalUnitName = vCenterApplianceUniqueServer
commonName = vcva55.vmware.com

Change the following lines:

  • subjectAltName: insert here data about name and IP of your vCSA (you can omit IPv6 if you don’t use it)
  • commonName: this must be your vCSA FQDN
  • all section [req_distinguished_name]
  • leave organizationalUnitName as it is

Save the file as “C:\OpenSSL\Certs\openssl_generic.cfg”.

We need to generate one .cfg file for each service, changing the “organizationalUnitName” by opening the “openssl_generic.cfg” file we just created:

  • organizationalUnitName = VMware vCenter Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_vpxd.cfg”)
  • organizationalUnitName = VMware Inventory Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_inventoryservice.cfg”)
  • organizationalUnitName = VMware LogBrowser Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_logbrowser.cfg”)
  • organizationalUnitName = VMware vSphere Autodeploy Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_autodeploy.cfg”)

You should now have a .cfg file for each service in each folder with a different organizationalUnitName.

To generate the certificate requests, assuming you have the same path I have, you can use the following commands.

cd c:\OpenSSL\bin

openssl req -new -nodes -out c:\openssl\certs\vCenterSSO\rui_vpxd.csr -keyout c:\openssl\certs\vCenterSSO\rui_vpxd.key -config c:\openssl\certs\vCenterSSO\openssl_vpxd.cfg

openssl req -new -nodes -out c:\openssl\certs\InventoryService\rui_inventoryservice.csr -keyout c:\openssl\certs\InventoryService\rui_inventoryservice.key -config c:\openssl\certs\InventoryService\openssl_inventoryservice.cfg

openssl req -new -nodes -out c:\openssl\certs\LogBrowser\rui_logbrowser.csr -keyout c:\openssl\certs\LogBrowser\rui_logbrowser.key -config c:\openssl\certs\LogBrowser\openssl_logbrowser.cfg

openssl req -new -nodes -out c:\openssl\certs\AutoDeploy\rui_autodeploy.csr -keyout c:\openssl\certs\AutoDeploy\rui_autodeploy.key -config c:\openssl\certs\AutoDeploy\openssl_autodeploy.cfg

Now you should also have a .key file and a .csr file in each respective directory.

To generate certificates from the .csr file login your Microsoft CA web interface (by default it is http://servername/CertSrv/):

  1. Click the Request a certificate link.
  2. Click advanced certificate request.
  3. Click the Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file link.
  4. Open the certificate request (rui_service.csr, as generated above for each component) in a plain text editor and paste this text into the Saved Request box.
  5. Select the Certificate Template as VMware Certificate.
  6. Click Submit to submit the request.
  7. Click Base 64 encoded on the Certificate issued screen.
  8. Click the Download Certificate link.
  9. Save the certificate as rui_service.crt, in the appropriate C:\OpenSSL\Certs\<service>\ folder.  (for example rui_vpxd.crt)
  10. Repeat Steps 2 to 10 for each of the additional service.
  11. Navigate back to the home page of the certificate server and click Download a CA certificate, certificate chain or CRL.
  12. Click the Base 64 option.
  13. Click the Download CA Certificate chain link.
  14. Save the certificate chain as cachain.p7b in the c:\openssl\certs\ directory.

By default, Microsoft CA certificates are generated with the .cer format. Either use Save As or change it to .crt before continuing.

When complete, you have four certificates (rui_service.crt) for each of the services generated in their respective c:\openssl\certs\<services> folders and the cachain.p7b file in the c:\openssl\certs\ folder.

Copy the c:\openssl\certs folder on the root of the vCenter filesystem via SCP, rename it to “ssl”, SSH to the vCSA, then:

service vmware-stsd stop
service vmware-vpxd stop

Rename all files in the service folders so that the .key file is named “rui.key” and the .crt file is named “rui.crt”.

On the vCenter Appliance, move where the cachain.p7b file is, then convert it to cachain.pem:

openssl pkcs7 -print_certs -in cachain.p7b -out cachain.pem

Now open cachain.pem with a text editor and remove any text before the first “—–BEGIN CERTIFICATE—–” and after “—–END CERTIFICATE—–“.

Note: This assumes there are no intermediate certificates in the Certificate Authority.

Copy the cachain.pem file in every service folder.

cd <vcenterservicefolder>
cat rui.crt cachain.pem > chain.pem
/usr/sbin/vpxd_servicecfg certificate change chain.pem rui.key

If all goes well you should receive this:

VC_CFG_RESULT = 0

Check KB2057248 if you get a different result.

service vmware-stsd start
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk

Create the chain.pem file for every service:

cat rui.crt cachain.pem > chain.pem

Then:

cd <inventoryservicefolder>
openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword
cp rui.key /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.crt /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.pfx /usr/lib/vmware-vpx/inventoryservice/ssl
cd /usr/lib/vmware-vpx/inventoryservice/ssl/
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
</em>rm /var/vmware/vpxd/inventoryservice_registered
service vmware-inventoryservice stop
service vmware-vpxd stop
service vmware-inventoryservice start
service vmware-vpxd start

Note: As there is a plain-text password on the above command, to avoid the history file showing the contents of the password because it is in plain text in the command above, run the unset HISTFILE command prior to executing any step containing a password.

Note: The default SSO administrator username for vCenter Single Sign-On 5.5 is administrator@vSphere.local

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk
cd <logbrowserservicefolder>
<code>openssl pkcs12 -export –out rui.pfx –in chain.pem -inkey rui.key –name rui –passout pass:testpassword</code>
cp rui.key /usr/lib/vmware-logbrowser/conf
cp rui.crt /usr/lib/vmware-logbrowser/conf
cp rui.pfx /usr/lib/vmware-logbrowser/conf
cd /usr/lib/vmware-logbrowser/conf
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
service vmware-logbrowser stop
service vmware-logbrowser start

In this environment the AutoDeploy service is not started so I’m skipping this step. (refer to KB2057223 to complete this step)

You can now restart the vCenter Server Appliance and chek that the certificates have been successfully replaced.

 

Related documents
Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223)
Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108)
Decoding a non-zero VC_CFG_RESULT for failed vpxd_servicecfg certificate changes (2057248)
Configuring certificates signed by a Certificate Authority (CA) for vCenter Server Appliance 5.1 (2036744)

vSphere 5.5 Update 1 is out with vSAN support!

What’s new

vCenter server 5.5 Update1

  • vCenter Server is now supported on Windows Server 2012 R2
  • vCloud® Hybrid Service™ vSphere® Client Plug-in, is now available in vSphere Web Client.

 

vSphere ESXi 5.5 Update1

  • VMware Virtual SAN  Virtual SAN 5.5 is a new hypervisor-converged storage tier that extends the vSphere Hypervisor to pool server-side magnetic disks (HDDs) and solid-state drives (SSDs). By clustering server-side HDDs and SSDs, Virtual SAN creates a distributed shared datastore designed and optimized for virtual environments. Virtual SAN is a standalone product that is sold separate from vSphere and requires its own license key

 

vSphere replication 5.5 Update1

  • VMware Virtual SAN is a fully supported feature of vSphere 5.5u1. You can use Virtual SAN in production environments with vSphere Replication 5.5.1 and vSphere 5.5u1.
  • Configure vSphere Replication on virtual machines that reside on VMware vSphere Flash Read Cache storage. vSphere Flash Read Cache is disabled on virtual machines after recovery.

 

vCenter Orchestrator 5.5 Update1

  • Workflow tagging
  • Plug-ins improvements
  • Version control system support
  • Dynamic Types (Beta)

Using ESXTOP with OSX Terminal application displays incorrectly

If are a VMware Admin switching from Windows to Mac you will find some limitations, even if lately VMware solved one of the major problems making the vSphere Web Client fully compatible with Mac in version 5.5.

Speaking about running ESXTOP from Mac, if you tried it you know you won’t be able to get any understandable data out of it and the reason seems to be the default  type of emulation that needs to be changed from “xterm-256color” to “xterm”.

Thanks to this post on the PunchingClouds Blog (bookmark it, it’s one of the good ones!) now this is not a problem anymore.

%d bloggers like this: