Using vCSA 6.0 as a Subordinate CA of a Microsoft Root CA

One of the nicest improvements in vSphere 6 is the ability to use the VMware Certificate Authority (VMCA) as a subordinate CA.
In most cases enterprises already have some form of PKI deployed in house and very often it is Microsoft based so I will show you how I did it with a Microsoft Enterprise CA.

I give for granted that the Microsoft PKI is already in place, in my case it is a single VM with an Enterprise Microsoft CA installed.

The vCSA should also be already be in place.

As first step I edit the certool config file but first I make a backup of the default configuration:

mkdir /root/backup
cp /usr/lib/vmware-vmca/share/config/certool.cfg /root/backup
vi /usr/lib/vmware-vmca/share/config/certool.cfg

Compile the config file with the parameters that are good for your setup then save the file and exit.

Now we have to generate a certificate request for the VMCA to pass to the Microsoft CA and there are many ways to do that, I am going to use the vSphere Certificate Manager Utility that will automatically take most steps for me:

/usr/lib/vmware-vmca/bin/certificate-manager

Screen Shot 2015-03-29 at 23.20.25
At this point I have the .csr file (/root/root_signing_cert.csr) and the private key (/root/root_signing_cert.key) so let’s feed it to the Microsoft CA as you normally would for any certificate request using the “Subordinate Certification Authority” template:

Screen Shot 2015-03-29 at 23.25.43Now you have to take the crt file in base64 format on the vCSA and also the Microsoft CA root certificate in base64 format as well; copying files with SCP will be a challenge because the root user on the vCSA by default doesn’t use the bash shell so if you want to use this method you need to edit the “/etc/passwd” and set the root user to use bash as a shell and then you can put it back as it was once you are done transferring the files.

It could be just simpler to open the certs on your computer and the connect to the vCSA via SSH and copy the content inside new files; one way or another you need to take the certificates on the vCSA, in my case they are “root_signing_cert.pem” and “cam.pem”.

Now we need to combine the two files in a chain file:

cp root_signing_cert.pem caroot.pem
cat ca.pem >> caroot.pem

If you open the “caroot.pem” file you should see a single cert file with both ca and certificate one after another.

Now we can go back to the vSphere Certificate Manager Utility to apply this certificate:

Screen Shot 2015-03-29 at 23.36.16

Since we have already edited the certool.cfg file we just have to confirm the values that the wizard proposes, just remember to enter the FQDN of the vCenter server:

Screen Shot 2015-03-29 at 23.37.18
If you have a successful outcome you can connect via browser to your vSphere Web Client and check the certificate:


Screen Shot 2015-03-29 at 23.39.59

Screen Shot 2015-03-29 at 23.40.08

 

As you can see now this is a trusted connection and the VMCA has released certificates for the Solution Users on behalf of the Microsoft Root CA.

You can check the active certificate in the vSphere Web Client in the Administration section:

Screen Shot 2015-03-29 at 23.42.30

In case you decide to remove the original root certificate then you will have to refresh the Security Token Service (STS) Root Certificate, and replace the VMware Directory Service Certificate following the vSphere 6 documentation.

Now the VMCA is capable of signing certificates that are valid in you PKI chain and are trusted by default in you Windows domain by all clients.

 

vSphere 6 Certificate Lifecycle Management

Recently I’ve been fighting with a vSphere environment and CA certificates and I thought a lot about certificate management and lifecycle in a VMware vSphere environment after that and how much it needs improvement. With the SSL Certificate Automation Tool VMware made a step in the right direction and even if the tools itself is sometimes a little buggy it is still very handy in automating a long and error prone process. In vSphere 6 VMware is taking another step in the right direction to help us create, apply and manage SSL certificates in a vSphere environment, but before talking about this we need to talk a bit about what’s new in SSO and vCenter architecture in vSphere 6. Since the introduction of SSO VMware changed its architecture in every major release, starting from 5.1 to 5.5 and now to 6.0 so let’s make a little bit of history:

Featured image

The new vSphere 6 management architecture introduces two main roles that you can deploy, these are the Management Node and the Platform Service Controller (PSC); the reason behind this separation is to have a logical entity that will take care of the main management features while another entity will hold the core and security features of the solution. What is nice about this separation is that you don’t need a 1:1 ratio between Management Nodes and PSCs so you can install PSC on separate boxes and replicate between them and then have as many Management nodes as you need (as long as you are within the same SSO domain)

Featured image

For HA scenario if you install PSC on separate boxes you will still need a load balancer. Supported solutions are Big-IP F5 and NetScaler so far.

You can obviously still install everything in one box:

Featured image

You might have noticed that the HA model for SSO was active/passive in 5.1, then active/active in 5.5 and now is active/passive again; this is due to the re-engineering of the Secure Token Service (STS) which is moving to a new and more robust method of STS (known as WebSSO) which is the same already used by vCAC (or vRealize Automation if you will) and that will be used from now forward instead of the old 5.5 method (WS-Trust). Let’s see how services are spread out on each role:

Let’s take a look to the services within the Management Node and the PSC:

Featured image

In the Management Node we can find services and features that every vSphere Admin feels very comfortable and familiar with such as vCenter Server, vSphere Web Client, Syslog Collector, etc., but two of them deserve a few words:

  • Virtual Datacenter Service: this service is new and it has been introduced to help mitigate the limitation connected with the Datacenter object in vCenter as a Management boundary.
  • (Optional) vPostgres: This component is obviously referring to the vCenter Appliance (thus optional) but I believe more and more new deployments or upgrades deserve to be considered a good fit for vCSA since VMware announced complete equality of features between vCenter installed on Windows and vCSA; leave alone the fuss of dedicating Windows licenses for vCenter which might not be a huge problem I just find the process of patching ad upgrading a vCSA simply amazing and it’s not a secret that products like EVO:RAIL make extensive use of vCSA. VMware wants to move all their services deployment model towards Virtual Appliances, this is not a secret and we need to get used to it, the sooner the better, but I’m digressing…

Featured image

In the Platform Service Controller or PSC we find our old friend SSO (we have had a rough past but now we are on better terms) and quite a few new services:

  • VMware Single Sign-On
    • Secure Token Service (STS)
    • Identity Management Service (IdM)
    • Directory Service (VMDir)
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • VMware Licensing Service
  • Authentication Framework Daemon (AFD)
  • Component Manager Service (CM)
  • HTTP Reverse Proxy

Describing all these services is out of the scope of this post but as you probably guess two of them will be our focus: the VMware Certificate Authority (or VMCA) and the VMware Endpoint Certificate Store (or VECS). But what are the roles of VMCA and VECS? The VMCA is no more or less than a CA, so you can:

  • Generate Certificates
  • Generate CRLs
  • Use the UI
  • Use the Command Line Interface to replace certificates

The VECS is where all certificates within the PSC are stored, with the only exception of the ESXi certificates that are stored locally on vSphere hosts, so here you can:

  • Store certificates and keys
  • Sync trusted certificates
  • Sync CRLs
  • Use the UI
  • Use the CLI to perform various actions

Since VMCA and VECS are part of the PSC, they will take advantage of the Multi-Master Replication Model which is offered by the Directory Service (VMDir) in order to achieve HA. In the past every service had its own user and required its own certificate but this is not the case anymore since we now have Solution Users (SU); since the number of services has increased significantly it would be impractical to manage the lifecycle of this many certificates so now we have 4 main SU that will hold the certificate used for a number of services.

Voila_Capture 2015-01-08_08-14-37_pm_white_background

What about use cases/scenarios in which I can implement VMCA? In what ways you can use this new tool?

Featured image

Scenario 1 and 2 are similar: the VMCA is the CA that releases certificates for all Solution Users (SU), the only difference is that in scenario 1 the VMCA is the root CA and you will need to distribute the Root CA Certificate so that all corporate browsers will trust it, while in scenario 2 the VMCA becomes part of an existing PKI as a subordinate CA and you certificate trust.

Featured image

In scenario 3 VMCA is installed but not used, CSRs are created and submitted to an external CA and VECS will be used to store certificates in PEM format.

Featured image

My favorite is scenario 2 because most enterprises I see already have a PKI (Microsoft CA usually) and all clients already trust the CA certificates, so adding the VMCA as s subordinate is a non disruptive process with a very low maintenance impact on the PKI itself, it protects investments already made to implement the current PKI and  preserves the knowledge to run the PKI.

Replacing certificates is still a CLI task (looks like Powershell will be involved) but VMCA and VECS are a very promising step toward the right direction for simplifying certificate lifecycle management in a vSphere environment.

How to replace default VCSA 5.5 certificates with Microsoft CA signed certificates

DISCLAIMER: This is a very lenghty procedure and I’ve changed some steps from the original KB trying to make it shorter; if I made some mistakes please let me know.

I don’t do this all the time but today I had to replace SSL certificates on a vCenter Virtual Appliance and since I know this will happen more and more often I thought I should write a shorter procedure since VMware KB is very detailed and, yet again, very long. At least it’s not as long as the infamous 96 steps of version 5.1.

Before proceding it’s good practice to shutdown your vCSA and take a snapshot.

Go to http://vcenter_ip_address:5480 or http://fqdn:5480 and chack that the “Certificate regeneration enabled” setting in the Admin tab of the vCSA web interface is set to “No” or we will lose all our work at first reboot:

1

Also, since we are going to use a Microsoft CA for this tutorial, it would be a good idea to take a look at KB2062108 and complete those steps before proceeding.

Note: This procedure is specific for vCSA 5.5. If you have a previous version of vCSA please refer to KB2036744.

Download and install the latest build of OpenSSL 0.9.8 on a machine of your choice. For convenience I installed it on a Windows VM in “C:\OpenSSL”.

Create the following folders:

C:\OpenSSL\Certs
C:\OpenSSL\Certs\vCenterSSO
C:\OpenSSL\Certs\InventoryService
C:\OpenSSL\Certs\LogBrowser
C:\OpenSSL\Certs\AutoDeploy

Open a text editor:

[ req ]
default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]
basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:vcva55, IP: 10.0.0.10, IP:ServerIPv6Address, DNS: vcva55.vmware.com

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = NY
localityName = New York
0.organizationName = VMware
organizationalUnitName = vCenterApplianceUniqueServer
commonName = vcva55.vmware.com

Change the following lines:

  • subjectAltName: insert here data about name and IP of your vCSA (you can omit IPv6 if you don’t use it)
  • commonName: this must be your vCSA FQDN
  • all section [req_distinguished_name]
  • leave organizationalUnitName as it is

Save the file as “C:\OpenSSL\Certs\openssl_generic.cfg”.

We need to generate one .cfg file for each service, changing the “organizationalUnitName” by opening the “openssl_generic.cfg” file we just created:

  • organizationalUnitName = VMware vCenter Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_vpxd.cfg”)
  • organizationalUnitName = VMware Inventory Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_inventoryservice.cfg”)
  • organizationalUnitName = VMware LogBrowser Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_logbrowser.cfg”)
  • organizationalUnitName = VMware vSphere Autodeploy Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_autodeploy.cfg”)

You should now have a .cfg file for each service in each folder with a different organizationalUnitName.

To generate the certificate requests, assuming you have the same path I have, you can use the following commands.

cd c:\OpenSSL\bin

openssl req -new -nodes -out c:\openssl\certs\vCenterSSO\rui_vpxd.csr -keyout c:\openssl\certs\vCenterSSO\rui_vpxd.key -config c:\openssl\certs\vCenterSSO\openssl_vpxd.cfg

openssl req -new -nodes -out c:\openssl\certs\InventoryService\rui_inventoryservice.csr -keyout c:\openssl\certs\InventoryService\rui_inventoryservice.key -config c:\openssl\certs\InventoryService\openssl_inventoryservice.cfg

openssl req -new -nodes -out c:\openssl\certs\LogBrowser\rui_logbrowser.csr -keyout c:\openssl\certs\LogBrowser\rui_logbrowser.key -config c:\openssl\certs\LogBrowser\openssl_logbrowser.cfg

openssl req -new -nodes -out c:\openssl\certs\AutoDeploy\rui_autodeploy.csr -keyout c:\openssl\certs\AutoDeploy\rui_autodeploy.key -config c:\openssl\certs\AutoDeploy\openssl_autodeploy.cfg

Now you should also have a .key file and a .csr file in each respective directory.

To generate certificates from the .csr file login your Microsoft CA web interface (by default it is http://servername/CertSrv/):

  1. Click the Request a certificate link.
  2. Click advanced certificate request.
  3. Click the Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file link.
  4. Open the certificate request (rui_service.csr, as generated above for each component) in a plain text editor and paste this text into the Saved Request box.
  5. Select the Certificate Template as VMware Certificate.
  6. Click Submit to submit the request.
  7. Click Base 64 encoded on the Certificate issued screen.
  8. Click the Download Certificate link.
  9. Save the certificate as rui_service.crt, in the appropriate C:\OpenSSL\Certs\<service>\ folder.  (for example rui_vpxd.crt)
  10. Repeat Steps 2 to 10 for each of the additional service.
  11. Navigate back to the home page of the certificate server and click Download a CA certificate, certificate chain or CRL.
  12. Click the Base 64 option.
  13. Click the Download CA Certificate chain link.
  14. Save the certificate chain as cachain.p7b in the c:\openssl\certs\ directory.

By default, Microsoft CA certificates are generated with the .cer format. Either use Save As or change it to .crt before continuing.

When complete, you have four certificates (rui_service.crt) for each of the services generated in their respective c:\openssl\certs\<services> folders and the cachain.p7b file in the c:\openssl\certs\ folder.

Copy the c:\openssl\certs folder on the root of the vCenter filesystem via SCP, rename it to “ssl”, SSH to the vCSA, then:

service vmware-stsd stop
service vmware-vpxd stop

Rename all files in the service folders so that the .key file is named “rui.key” and the .crt file is named “rui.crt”.

On the vCenter Appliance, move where the cachain.p7b file is, then convert it to cachain.pem:

openssl pkcs7 -print_certs -in cachain.p7b -out cachain.pem

Now open cachain.pem with a text editor and remove any text before the first “—–BEGIN CERTIFICATE—–” and after “—–END CERTIFICATE—–“.

Note: This assumes there are no intermediate certificates in the Certificate Authority.

Copy the cachain.pem file in every service folder.

cd <vcenterservicefolder>
cat rui.crt cachain.pem > chain.pem
/usr/sbin/vpxd_servicecfg certificate change chain.pem rui.key

If all goes well you should receive this:

VC_CFG_RESULT = 0

Check KB2057248 if you get a different result.

service vmware-stsd start
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk

Create the chain.pem file for every service:

cat rui.crt cachain.pem > chain.pem

Then:

cd <inventoryservicefolder>
openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword
cp rui.key /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.crt /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.pfx /usr/lib/vmware-vpx/inventoryservice/ssl
cd /usr/lib/vmware-vpx/inventoryservice/ssl/
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
</em>rm /var/vmware/vpxd/inventoryservice_registered
service vmware-inventoryservice stop
service vmware-vpxd stop
service vmware-inventoryservice start
service vmware-vpxd start

Note: As there is a plain-text password on the above command, to avoid the history file showing the contents of the password because it is in plain text in the command above, run the unset HISTFILE command prior to executing any step containing a password.

Note: The default SSO administrator username for vCenter Single Sign-On 5.5 is administrator@vSphere.local

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk
cd <logbrowserservicefolder>
<code>openssl pkcs12 -export –out rui.pfx –in chain.pem -inkey rui.key –name rui –passout pass:testpassword</code>
cp rui.key /usr/lib/vmware-logbrowser/conf
cp rui.crt /usr/lib/vmware-logbrowser/conf
cp rui.pfx /usr/lib/vmware-logbrowser/conf
cd /usr/lib/vmware-logbrowser/conf
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
service vmware-logbrowser stop
service vmware-logbrowser start

In this environment the AutoDeploy service is not started so I’m skipping this step. (refer to KB2057223 to complete this step)

You can now restart the vCenter Server Appliance and chek that the certificates have been successfully replaced.

 

Related documents
Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223)
Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108)
Decoding a non-zero VC_CFG_RESULT for failed vpxd_servicecfg certificate changes (2057248)
Configuring certificates signed by a Certificate Authority (CA) for vCenter Server Appliance 5.1 (2036744)

Balancing multiple Horizon Workspace gateway-va with HAProxy

When working with Horizon Workspace the first component you will scale to multiple instances is probably the gateway-va since this is the access point of all users, just to make sure it’s always available for connections.

In this case you need a load balancer to direct all users to all the gateway-va you have in your environment; i wrote about commercial and open source load balancers and also how to build one with HAProxy in this post.

I’m going to show you how i configure it with Horizon Workspace but remember that since I’ve learned about HAProxy only relatively recently by Luca Dell’Oca my configuration is just the way i do it and not necessarily the best so use the comments if you want to contribute.

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------

global
log 127.0.0.1 local2 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
option accept-invalid-http-request
retries 3
timeout http-request 60s
timeout queue 30m
timeout connect 1800s
timeout client 30m
timeout server 30m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats :9000
stats realm Haproxy\ Statistics
stats uri /stats

#---------------------------------------------------------------------
# Redirect to secured
#---------------------------------------------------------------------
frontend unsecured
bind :80
redirect scheme https if !{ ssl_fc }

#---------------------------------------------------------------------
# frontend secured
#---------------------------------------------------------------------
frontend front
bind :443 ssl crt /etc/haproxy/reverseproxy.pem
mode http

acl workspace hdr_beg(host) -i workspace.myvirtualife.net
use_backend workspace if workspace

#---------------------------------------------------------------------
# balancing between the various backends
#---------------------------------------------------------------------
backend workspace
mode http
server workspace1 192.168.110.10:443 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl
server workspace2 192.168.110.11:443 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl

Try to add a gateway-va and experiment with HAProxy to test HAProxy as load balancer. You can use this article if you want to know how to do it.

There are few more things worth of noting:

  • timeouts are really long here otherwise users will experience disconnects because this is the kind of web app you keep open quite a lot;
  • on port 9000 on the HAProxy host you will find statistics, for example “lb.yourcompany.yourdomain:9000/stats”, that will give numbers about state of connections and state of backends, problems, etc…
  • “log 127.0.0.1 local2 info” is necessary if you want logging enabled which is so important when troubleshooting problems; a lot on how to read logs in the HAProxy documentation

if you intend to put a SSL cert like in my configuration, know that it has to be a chain of cert and private key like this:

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

To make logging work and write to a separate file instead of putting everything in “/var/log/messages”, edit your “/etc/rsyslog.conf” file and make sure these lines are present:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# HAProxy
local2.* /var/log/haproxy.log

How to build a load balancer with HAProxy

If you’ve been reading my previous articles you must have noticed that in Horizon Workspace there is often the hidden assumption that you need and/or you already have in place a load balancer.

Load balancers are usually appliances sold in hardware that are put in front of your workloads to distribute load to multiple backend machines delivering the same services. The reason why you want to do that is to provide performance and availability to your service as it grows.

Horizon Workspace is no difference and since it’s pretty easy to have multiple gateway-va for redundancy and scalability then you are going to need a load balancer.

I don’t want to get much into details about how many vendors are out there and what is good and bad about them, nor what I see in production environments; what I am going to say is that:

  • load balancers can be an expensive combination of hardware and software;
  • nowadays they do a whole bunch of things besides just load balancing connections, like SSL offloading, caching, content inspection, etc.
  • since virtualization has become so mainstream we now have load balancers solutions all in software coming as virtual appliances

Some time ago I just happened to bump into a nice blog post by Luca Dell’Oca about a piece of software called HAProxy.

HAProxy is a opensource software that does HTTP/TCP load balancing with a lot of nice features including for example SSL Offloading; also HAProxy seems to be used in production in very large environments with no problems at all. Check their website for reference.

At the time I was looking for a way to load balance a VMware View environment and after reading Luca’s post about how to do it with HAProxy I became a real funboy. If a customer has no load balancing solution or needs to load balance only a small subset of services I always go with HAProxy now because I found it to be very reliable and it delivers great performance consuming very little resources. What can you ask for more?

The documentation is pretty broad and precise which is always good when it comes to learn your way through things.

Enough with evangelizing HAProxy, I will just get down to business and show you how I build my load balancers.

First let’s clear out some goals and assumptions:

  • I like to use CentOS to do this but it’s not mandatory
  • I’m a big fun of RPMs but i prefer to build HAProxy from source code
  • in this post i will provide with a basic installation just to start-up
  • in future posts i will publish specific configs i use for Horizon Workspace and about how to deploy more than one HAProxy virtual appliance for redundancy
  • by no means this is the best way to do it, it’s just what i do
  • by no means I’m discouraging you from buying commercial load balancers; always remember you are the only support for solutions you build!

What I do is downloading a CentOS iso for minimal install, it’s good for this task and it’s a small download. Pick x86 or x64. Whatever. Just install it as you normally would, connect it to the internet and install VMware Tools as well.

For this tutorial I used the latest CentOS which at the time of writing is 6.4.

After getting a ‘root’ prompt this is what I do:

yum install wget openssl-devel pcre-devel make gcc -y     # this installs prerequisites
wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev19.tar.gz     # download the package
tar xzvf haproxy-1.5-dev19.tar.gz     # extracting
cd haproxy-1.5-dev19     # enter the extracted directory
make TARGET=linux2628 CPU=i686 USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE=1     # i compile it with compression and ssl support; use CPU=x86_64 for CentOS x64
make install     # install
cp /usr/local/sbin/haproxy* /usr/sbin/     # copy binaries to /usr/sbin
cp /root/haproxy-1.5-dev19/examples/haproxy.init /etc/init.d/haproxy     # copy init script in /etc/init.d
chmod 755 /etc/init.d/haproxy     # setting permission on init script
mkdir /etc/haproxy     # creating directory where the config file must reside
cp /root/haproxy-1.5-dev19/examples/examples.cfg /etc/haproxy/haproxy.cfg     # copy example config file
mkdir /var/lib/haproxy     # create directory for stats file
touch /var/lib/haproxy/stats     # creating stats file
useradd haproxy     # i like to make haproxy run with a specific user
service haproxy check     # checking configuration file is valid
service haproxy start     # starting haproxy to verify it is working
chkconfig haproxy on     # setting haproxy to start with VM

The main reason why I like to build HAProxy myself is that when I was learning about it I had troubles to make SSL offloading work even if I was sure I was configuring it right. Turns out most RPMs out there are built without SSL support so I started just building it up by myself. In this way I can always use the last version and even if the current latest is a development version I can tell you it’s pretty stable.

Don’t forget to disable all unneeded services/daemons; most of them are not needed to run a load balancer.

If you intend to leave the firewall on, go check Luca’s post which will give you a good insight about how to configure iptables to work with HAProxy.

Don’t bother disabling SELinux, it seems to go by with HAProxy pretty well.

Have fun with your new shiny (and free) load balancer.

How to deal with Horizon Workspace 1.5 FQDN and certificates

In the past I’ve written about how to install Horizon Workspace 1.0 and one of the tricky parts was about the decision of Workspace FQDN.

In version 1.0 you couldn’t change it afterwards so you had to do it right from the start, in version 1.5 you can change it but it’s still pretty tricky so you should still know in advance how it should look in the end. It’s not mandatory but it would help a lot.

In Horizon Workspace 1.5 during setup you won’t be asked for the FQDN but it will be automatically set as the name you’ve chosen for your gateway-va when you’ve created DNS entries for the VApp.

After deploying and configuring Horizon Workspace just as it used to be, you can go here and change the FQDN and/or certificates:

1

In this screen you are able to:

  • Configure an external load balancer
  • Install custom certificates
  • Change Horizon Workspace FQDN
  • Regenerate self-signed certificates

A lot of people reported errors when trying to change the FQDN, the most common is this:

Invalid IDP host/port

The reason for the above is that during the change of FQDN a check is performed on the SSL certificate for the new hostname to confirm that its common name matches the new FQDN. Here’s what i get in the configurator-va logs when i experience the issue:

ERROR [tomcat-http–29] com.vmware.horizon.configurator.vm.remote.impl.ConnectorRemoteImpl – Error when updating Connector “connector_va” with new IDP Url. Response from server: “Hostname is invalid or not reachable”. Could not connect to the URL. hostname in certificate didn’t match: “old_workspace_fqdn” != “new_workspace_fqdn”

This would mean that if we are just changing the FQDN but hosts remain the same we would have to replace certificates before actually making the change, a lot like during version 1.0 installation, but the same procedure doesn’t seem to help because even if the new certificate gets applied it still exposes the old one at least in one page that gets checked and this generates the error above.

At the end of the day we want to get the job done in the best way possible, so i can share the 2 ways i found to take over this task. The goal i am setting to myself as final result include the following:

  • Making sure self-signed certificates are replaced
  • Every virtual machine has numbered hostname naming convention based on role to help scaling (eg. gateway01, data01, etc.)
  • Horizon Workspace FQDN is set as i wish
  • Horizon Workspace FQDN is not the hostname of one of the virtual machine

So here’s my two cents on how to do that.

Method 1
In this method we create DNS records in our internal DNS so that the FQDN is pointing at a load balancer:

192.168.110.20			cofigurator-15.vsphere.lab
192.168.110.21			service-15.vsphere.lab
192.168.110.22			connector-15.vsphere.lab
192.168.110.23			data-15.vsphere.lab
192.168.110.24			gateway-15.vsphere.lab
172.16.110.2		        workspace-15.myvirtualife.net

My FQDN is “workspace-15.myvirtualife.net” since this is testing version 1.5.

My load balancer is configured to redirect all requests for “workspace-15.myvirtualife.net” to “192.168.110.24” and i’ve installed a certificate on it with the common name “workspace-15.myvirtualife.net” as it should be. To generate certificates i’ve used my internal Microsoft CA.

Complete deploy and configuration, accepting default for FQDN and certificates; the FQDN now is “gateway-15.vsphere.lab”.

Now log in to “https://gateway-15.vsphere.lab/admin&#8221; and reach the screen where you can change the FQDN and configure it as follows:

2

I pasted the certificate of my internal Microsoft CA since that is what i used to generate the cert for the load balancer.

Clicking save will change FQDN on all virtual machines plus adding my internal Microsoft CA as a trusted CA. This is a nice improvement since in version 1.0 it was a manual process where you had to SSH all VMs one by one… go read previous posts to see how much fun that was. Thank you VMware for this improvement.

In the end it should all look like this:

3

Now, given that your users can access your load balancer, just connect to “https://workspace-15.myvirtualife.net/admin&#8221; and you should be good to go.

If you need users to access from outside the network, like from the internet, just publish the load balancer and create a DNS record with the Workspace FQDN that points to the public IP used to publish the load balancer.

Method 2
In this method we create DNS records so that the FQDN is actually the name of our gateway-va:

192.168.110.25			cofigurator-15-2.vsphere.lab
192.168.110.26			service-15-2.vsphere.lab
192.168.110.27			connector-15-2.vsphere.lab
192.168.110.28			data-15-2.vsphere.lab
192.168.110.29		        workspace-15-2.myvirtualife.net

My FQDN is “workspace-15-2.myvirtualife.net” since this is method 2 for version 1.5.

You can proceed with the installation as you did for 1.0 version.

During configuration just accept all default when it comes to FQDN and certificates.

If you haven’t generated certificates for “workspace-15-2.myvirtualife.net” do it now.

Now reach the “FQDN & SSL” section in the configurator-va as we did for method 1 and set it like this:

5

The certificate you are pasting needs to be a chain of certificates including the CA certificate as well.

As the documentation states, the certificate chain has to look as follows:

-----BEGIN CERTIFICATE-----

SSL Cert - Workspace SSL cert

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Intermediate/Issuing CA Cert

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Root CA Cert

-----END CERTIFICATE-----

Clicking save will change FQDN on all virtual machines plus adding my internal Microsoft CA as a trusted CA. I already stated how annoying that was to do this manually so once more thank you VMware.

After a few minutes you should find yourself back at the “FQDN & SSL” screen. Don’t expect any successful confirmation because you won’t get any but if you get no error it’s all good. Close your browser and connect back to your Workspace FQDN and you should see no SSL warning no more.

Now since i don’t want my gateway-va hostname to be the same as the FQDN let’s open the console, login as root and fire up yast to change it:

Before

After

We also need to create a DNS record that points “gateway-15-2.vsphere.lab” to gateway-va IP address. Make sure the PTR for that IP points to the new record and not to the FQDN record.

Reboot the gateway-va and go back to the “FQDN & SSL”:

6

Since the gateway-va and the Horizon Workspace FQDN are not the same, the configurator-va assumes there will be a load balancer. You still have DNS pointing the Horizon Workspace FQDN to the gateway-va and your internal users should not have the warning for the self-signed certificate anymore. You can add a reverse proxy/load balancer if you want for outside access. This is a good way if you don’t want multiple gateway-va but you can still add more if you also add a load balancer and point the Workspace FQDN to that.

This should take you where you want to go. Comment section is open if you have doubts about the procedure i described I’ll try to answer fast enough and help if I can.

How to replace Horizon Workspace 1.0 self-signed certificates with Microsoft CA certificates

UPDATE: If you are deploying Horizon Workspace 1.5 you should look at this post.

In the last post we generated new certificates from an internal Microsoft CA to use them as replacement of the Horizon Workspace self-signed certificates that are created during the setup process.

For certificates to work correctly, all parties in the process need to trust the Certification Authority; this include all servers and clients involved in the Horizon Workspace deployment.

Because of this, before applying the new certificates to Workspace virtual appliances we need to add our internal Microsoft CA to the list of trusted Certification Authorities; this step is not needed if you are buying certificates from a public CA that is already trusted, Verisign can be an example.

In this phase you will need to connect via ssh to all 5 virtual appliances with the user ‘sshuser’ (password is the same as ‘root’) and raise to ‘root’ with “su -“; you will then copy the CA certificate (ca.pem if you followed my previous post) via SCP in the home directory of user ‘sshuser’ then do the following:

cp /home/sshuser/ca.pem /etc/ssl/certs
c_rehash


Then do the following on the service-va and connector-va virtual machines:

/usr/java/jre1.6.0_37/bin/keytool -import -trustcacerts -file /etc/ssl/certs/horizon_private_root_ca.pem -alias horizon_private_root_ca -keystore /usr/java/jre-vmware/lib/security/cacerts


In my case:

/usr/java/jre1.6.0_37/bin/keytool -import -trustcacerts -file /etc/ssl/certs/ca.pem -alias vsphere-va -keystore /usr/java/jre-vmware/lib/security/cacerts


And run the following on the data-va:

/opt/zimbra/jdk1.7.0_05/jre/bin/keytool -import -trustcacerts -file /etc/ssl/certs/horizon_private_root_ca.pem -alias horizon_private_root_ca -keystore /opt/zimbra/jdk1.7.0_05/jre/lib/security/cacerts


In my case:

/opt/zimbra/jdk1.7.0_05/jre/bin/keytool -import -trustcacerts -file /etc/ssl/certs/ca.pem -alias vsphere-va -keystore /opt/zimbra/jdk1.7.0_05/jre/lib/security/cacerts


Note: The password to import the CA in the store is “changeit”.

Note: If you have an intermediate CA certificate you will have to run the same commands for that certificate too.

At this point your internal CA should be trusted but at times I’ve seen this happening only after a reboot of all virtual machines, so let’s just stop the vApp and restart it.

Changing the certificates is a less tedious process and it can be performed entirely using the web interface. Open your browser and connect to the Workspace admin page, in my case https://workspace.myvirtualife.net/admin and go to “Settings” -> “View Virtual Appliances System Configuration”:

1

Then click on “SSL Certificate” and paste certificate (horizon.pem) and private key (key.pem) from the files we created earlier, then press “Save”:

2

You will get a green box as a confirmation.
Now go to “Module Configuration” -> “Go To Connector”:

3

Now go to “SSL Certificate” and do the same as you did before pasting certificate and private key:

4

Now you should be able to connect back to the Workspace admin page and notice that you are running with the new certificates, and in my case i have no certificate warning because my workstation is domain joined and by default it trusts the Microsoft CA:

5

Well that’s great, isn’t it?

There’s still a lot of work to do to complete our environment but are well on our way.

More in the posts to come, see you there!

%d bloggers like this: