How to backup, restore and schedule vCenter Server Appliance vPostgres Database

Now that we are moving away from SQL Express in favor of vPostgres for vCenter simple install on Windows and since vPostgres is the default database engine for (not so simple) install of vCSA I thought it would be nice to learn how to backup and restore this database.

Since it’s easier to perform these tasks on Windows and since there are already many guides on the Internet I will focus on vCSA because I think that more and more production environment (small and big) will be using vCSA since now it’s just as functional as vCenter if not more. (more on this in another post…)

You will find all instructions for both Windows and vCSA versions of vCenter on KB2091961, but more important than that you will find there also the python scripts that will work all the magic for you so grab the “” file and copy it to the vCSA:

scp root@<vcenter>:/tmp

For the copy to work you must have previously changed the shell configuration for the root user in “/etc/passwd” from “/bin/appliancesh” to “/bin/bash”


chmod +x
mkdir /tmp/linux_backup_restore/backups
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/VCDB.bak

All you will see when the backup is completed is:

Backup completed successfully.

You should see the backup file now:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 912K
drwx------ 2 root root 4.0K Jun 3 19:41 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak

At this point I removed a folder in my vCenter VM and Templates view, then I logged off the vSphere WebClient and started a restore:

service vmware-vpxd stop
service vmware-vdcs stop
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/VCDB.bak
service vmware-vpxd start
service vmware-vdcs start

I logged back in the WebClient and my folder was back, so mission accomplished.

Now how do I schedule this thing? Using the good old crontab but before that I will write a script that will run the backup and also give a name to the backup file corresponding to the weekday so I can have a rotation of 7 days:

_dow="$(date +'%A')"
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/${_bak}

I saved it as “backup_vcdb” and made it executable with “chmod +x backup_vcdb”.

Now to schedule it just run “crontab -e” and enter a single line just like this:

0 23 * * * python /tmp/linux_backup_restore/backup_vcdb

This basically means that the system will execute the script every day of every week of every year at 11pm.

After the crontab job runs you should see a new backup with a name of this sort:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 1.8M
drwx------ 2 root root 4.0K Jun 3 19:46 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak
-rw------- 1 root root 900K Jun 3 19:46 VCDB_Wednesday.bak

You will also have the log files of these backups in “/var/mail/root”.

Enjoy your new backup routine 🙂

Balancing multiple Horizon Workspace gateway-va with HAProxy

When working with Horizon Workspace the first component you will scale to multiple instances is probably the gateway-va since this is the access point of all users, just to make sure it’s always available for connections.

In this case you need a load balancer to direct all users to all the gateway-va you have in your environment; i wrote about commercial and open source load balancers and also how to build one with HAProxy in this post.

I’m going to show you how i configure it with Horizon Workspace but remember that since I’ve learned about HAProxy only relatively recently by Luca Dell’Oca my configuration is just the way i do it and not necessarily the best so use the comments if you want to contribute.

# Global settings

log local2 info
chroot /var/lib/haproxy
pidfile /var/run/
maxconn 4000
user haproxy
group haproxy
stats socket /var/lib/haproxy/stats

# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except
option redispatch
option accept-invalid-http-request
retries 3
timeout http-request 60s
timeout queue 30m
timeout connect 1800s
timeout client 30m
timeout server 30m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats :9000
stats realm Haproxy\ Statistics
stats uri /stats

# Redirect to secured
frontend unsecured
bind :80
redirect scheme https if !{ ssl_fc }

# frontend secured
frontend front
bind :443 ssl crt /etc/haproxy/reverseproxy.pem
mode http

acl workspace hdr_beg(host) -i
use_backend workspace if workspace

# balancing between the various backends
backend workspace
mode http
server workspace1 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl
server workspace2 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl

Try to add a gateway-va and experiment with HAProxy to test HAProxy as load balancer. You can use this article if you want to know how to do it.

There are few more things worth of noting:

  • timeouts are really long here otherwise users will experience disconnects because this is the kind of web app you keep open quite a lot;
  • on port 9000 on the HAProxy host you will find statistics, for example “lb.yourcompany.yourdomain:9000/stats”, that will give numbers about state of connections and state of backends, problems, etc…
  • “log local2 info” is necessary if you want logging enabled which is so important when troubleshooting problems; a lot on how to read logs in the HAProxy documentation

if you intend to put a SSL cert like in my configuration, know that it has to be a chain of cert and private key like this:


To make logging work and write to a separate file instead of putting everything in “/var/log/messages”, edit your “/etc/rsyslog.conf” file and make sure these lines are present:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# HAProxy
local2.* /var/log/haproxy.log

How to enable SSH root access in Horizon Workspace Virtual Machines

If you followed my previous posts you know how often and how useful is to SSH into a virtual appliance in Horizon Workspace and most of the time the commands you issue need to be run as ‘root’.

By default root access is not allowed via SSH and in order to get ‘root’ prompt you have to SSH as the user ‘sshuser’ with the same password as ‘root’ and then run:

su -

This can be annoying in particular using SCP to copy files because you are limited to ‘sshuser’ home directory and this force us to log back in as ‘root’ again to move the files we just copies.

In actuality there is a way to enable ‘root’ access straight from SSH to make things faster.


  1. Connect to each VA console via vSphere Client or WebClient
  2. Select “Login” and enter as “root”
  3. vi /etc/ssh/sshd_config
  4. Find “PermitRootLogin”
  5. Change “PermitRootLogin” from “no” to “yes” and save the file
  6. service sshd restart

The “/etc/ssh/sshd_config” file should look like this:

#HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes	# default is no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes

Now you can log in as ‘root’ directly from SSH.

How to replace Horizon Workspace 1.0 self-signed certificates with Microsoft CA certificates

UPDATE: If you are deploying Horizon Workspace 1.5 you should look at this post.

In the last post we generated new certificates from an internal Microsoft CA to use them as replacement of the Horizon Workspace self-signed certificates that are created during the setup process.

For certificates to work correctly, all parties in the process need to trust the Certification Authority; this include all servers and clients involved in the Horizon Workspace deployment.

Because of this, before applying the new certificates to Workspace virtual appliances we need to add our internal Microsoft CA to the list of trusted Certification Authorities; this step is not needed if you are buying certificates from a public CA that is already trusted, Verisign can be an example.

In this phase you will need to connect via ssh to all 5 virtual appliances with the user ‘sshuser’ (password is the same as ‘root’) and raise to ‘root’ with “su -“; you will then copy the CA certificate (ca.pem if you followed my previous post) via SCP in the home directory of user ‘sshuser’ then do the following:

cp /home/sshuser/ca.pem /etc/ssl/certs

Then do the following on the service-va and connector-va virtual machines:

/usr/java/jre1.6.0_37/bin/keytool -import -trustcacerts -file /etc/ssl/certs/horizon_private_root_ca.pem -alias horizon_private_root_ca -keystore /usr/java/jre-vmware/lib/security/cacerts

In my case:

/usr/java/jre1.6.0_37/bin/keytool -import -trustcacerts -file /etc/ssl/certs/ca.pem -alias vsphere-va -keystore /usr/java/jre-vmware/lib/security/cacerts

And run the following on the data-va:

/opt/zimbra/jdk1.7.0_05/jre/bin/keytool -import -trustcacerts -file /etc/ssl/certs/horizon_private_root_ca.pem -alias horizon_private_root_ca -keystore /opt/zimbra/jdk1.7.0_05/jre/lib/security/cacerts

In my case:

/opt/zimbra/jdk1.7.0_05/jre/bin/keytool -import -trustcacerts -file /etc/ssl/certs/ca.pem -alias vsphere-va -keystore /opt/zimbra/jdk1.7.0_05/jre/lib/security/cacerts

Note: The password to import the CA in the store is “changeit”.

Note: If you have an intermediate CA certificate you will have to run the same commands for that certificate too.

At this point your internal CA should be trusted but at times I’ve seen this happening only after a reboot of all virtual machines, so let’s just stop the vApp and restart it.

Changing the certificates is a less tedious process and it can be performed entirely using the web interface. Open your browser and connect to the Workspace admin page, in my case and go to “Settings” -> “View Virtual Appliances System Configuration”:


Then click on “SSL Certificate” and paste certificate (horizon.pem) and private key (key.pem) from the files we created earlier, then press “Save”:


You will get a green box as a confirmation.
Now go to “Module Configuration” -> “Go To Connector”:


Now go to “SSL Certificate” and do the same as you did before pasting certificate and private key:


Now you should be able to connect back to the Workspace admin page and notice that you are running with the new certificates, and in my case i have no certificate warning because my workstation is domain joined and by default it trusts the Microsoft CA:


Well that’s great, isn’t it?

There’s still a lot of work to do to complete our environment but are well on our way.

More in the posts to come, see you there!

How to install Horizon Workspace using an external database

In the previous posts we’ve taken care of all the preparation steps so now we should be ready to get down to business and install Horizon Workspace.

First of all download the Horizon Workspace OVA from the VMware website and get a product key; trial is good if you didn’t purchase one yet and it should be ok for proof-of-concept.

Once you have the OVA file you can import it in vCenter using the usual “Deploy OVF from template” menu.

While we wait for the upload to complete we need to create records in our DNS server for every virtual appliance. Here’s how i configured mine in my lab:			cofigurator.vsphere.lab			service01.vsphere.lab			connector01.vsphere.lab			data01.vsphere.lab		        gateway01.vsphere.lab

You also need to create PTR records, if you have a Windows based DNS this can be done simply by selecting a flag while creating A records:


Note: You must have created the reverse zone in DNS before using this flag. Otherwise you can manually create PTR records in the reverse zone.

Go through the wizard, once you get to setting up the network for the virtual appliances fill gateway, DNS, subnet mask and pick a port group:

Ova network

If you followed previous posts you should have no problems filling up all the information. Make sure all virtual appliances are on the same network segment.

Now let’s assign IP addresses to different virtual appliances according to the DNS records we created:

Ova network 2

During setup the name of the virtual appliances will be assigned based on reverse lookup query of every IP address so it’s important to also to create PTR records because A records are not used for DNS reverse lookups and if they are missing the setup process will fail.
In my case i don’t have a specific timezone for my country but since i live in GMT+1 i can choose Paris. If you don’t know what to choose or make the wrong choice here you can change it later on and you will also have more options to choose from.

In case you want to change the timezone later, just finish the whole setup and before starting configuration ssh to all VAs, get ‘root’ prompt and run these commands:

rm /etc/localtime
ln -s /usr/share/zoneinfo/Europe/Rome /etc/localtime

The date command in the beginning and end are useful to see if the operation was successful.

Note: When you get to the end of the wizard remember to check the flag so that the vApp is powered on after deployment.

After the deployment you’ll notice that only the configurator-va will be powered on as this is where we will setup the whole Horizon Workspace solution, so let’s start by connecting to the console of the configurator-va with the vSphere Client or Web Client as you prefer. You will be asked to press enter to start setup and here is where your DNS reverse records will be checked:

Config 1

Make sure there is correspondence between what you see here and the names and IP addresses you wanted to assign. If everything is correct you can confirm and go ahead; you will be asked for ‘root’ password do be assigned to all Vas plus a bunch of settings we described when we compiled the checklist in previous posts:

Config 2

The interesting thing to note here is that the suggested FQDN would be the same name of the gateway that we set in the DNS records, but we want to put this out on the internet so we are choosing “”.

Remember: the FQDN cannot be changed after deployment. The only supported option to change it is redeploying the whole thing from scratch.

Note: You need to use a valid SMTP server or setup will stop.

After answering to all questions, which include SMTP and vCenter parameters, you will see quite a few things happening:

  • turning on and preparing VMs
  • setting root passwords
  • setting timesync
  • generating self-signed ssl certificated
  • setting workspace FQDN
  • configuring virtual appliances firewalls
  • starting webapps

Grab something to drink, make phone calls… this takes a while. At the end you will be instructed to press enter and connect to the configurator-va via HTTPS.

Config 4

Admin user is a local user that can access the configurator appliance in case of problems, but it’s not the Horizon Workspace administrator.

Fill up licensing details and click “Next”.

In the Database Connection Setup let’s pick the External Database option and use the vPostgres instance we created earlier:


click “Next”:

Db 1

Error while testing DB connection. I/O error:; nested exception is

Now, this is where it gets interesting!
We get an error, so we must have made some mistake in the vPostgres database… well, no we didn’t. The problem here is that the configurator-va doesn’t know the host “” because we didn’t create an A record for it in the DNS, so add it then try again:

Db 2

Error while testing DB connection. I/O error: No route to host; nested exception is No route to host

Ok, another error, we must have made a mistake. ACTUALLY NO. I pointed the Workspace record to the Load Balancer as suggested in the documentation but the setup doesn’t like this choice. After a while i figured it’s because in this phase we need to point it to the gateway-va ip address so it can correctly recognize itself, so go change the workspace record in DNS and try again:

Db 5

Error creating admin user. hostname in certificate didn’t match: !=

Crap. Error again. A different one.
I know it doesn’t look like it but we are making progresses here.
At this point i understood why the setup wanted me to use the gateway-va FQDN as Workspace FQDN. Our problem now is that the “” hostname doesn’t match with the common name of the certificate that has been generated for the gateway-va which in my case it “gateway01.vsphere.lab”, but we need something that can be used outside on the internet so what do we do now?

I found the solution in the VMTN communities which are always a great resource. First let’s connect to the configurator-va with the user ‘sshuser’ with the same password we chose for ‘root’ during setup, type “su -” and insert the ‘root’ password and once we have the “#” prompt the do the following:

cd /usr/local/horizon/lib/menu/secure
./wizardssl.hzn --makesslcert gateway-va 'workspace FQDN'

In my case:

cd /usr/local/horizon/lib/menu/secure
./wizardssl.hzn --makesslcert gateway-va

Now all certificates are generated again and pushed to all virtual appliances, but the gateway-va certificate will match to the Workspace FQDN, so now we should be set, let’s try again:

Db 3

Whooa! Finally. You don’t know how much it took me to figure this out. No really, stop guessing. You don’t WANT to know.
Another option i could think of is to configure the load balancer before setting up Horizon Workspace, and point “” record to it but it think it’s not practical to set up a load balancer if the application is not up yet because you would have no way to test it.

In other words what i do is:

  • standard setup
  • set internet FQDN as Workspace FQDN
  • temporary point Workspace FQDN to gateway-va
  • recreate certificates
  • complete setup

Later on we will complete the job configuring load balancers, changing DNS entries to point at them and generating new certificates that are not self-signed so all pieces fall into place. More on these activities in later posts.

Now it’s time to configure Active Directory integration:


Configure using your Active Directory LDAP structure.

The user ‘workspace’ is a user i created earlier in Active Directory and that is the user that will function as Horizon Workspace admin.

Click “Next”.

Accept defaults for user mapping.

Click “Next”.

Let’s discover the users:


Note: If you see an error tab it’s most likely because you didn’t compile fields Name, Last Name and Email for all users.

Now selecting groups:


Click “Add” next to the Active Directory groups that you want to add to the Horizon Workspace.

For SSL certificates just leave defaults and click “Next”.

In the “Select Modules” page enable all modules but the View module and click “Next”.


Click “Go to Horizon Workspace” and in the login screen use the credentials you’ve set during setup:


Note: The password was set on the user when it has been created in Active Directory.

You should get here:

Login 1

Congrats! You’ve setup Horizon Workspace and in the coming posts we will complete the job installing load balancers, taking care of SSL Certificates and so on.

Understanding Horizon Workspace components and installation prerequisites

In the last post i described in details how to prepare a vPostgres DB to host Horizon Workspace external database.

During the installation process, as we will see, you can choose to use an internal database or an external one but keep in mind that the internal database is ment only for testing purpose so if you are installing Horizon Workspace in a production environment you must have a VM with vPostgres installed as this is the only supported configuration, so you can understand why the first post was needed.

So now we are ready to install Horizon Workspace… well, not quite yet. It is very important to understand that to install this product there are number of preparation steps that need to be taken before actually getting our hands dirty and start having fun. Some of those steps include filling up some technical prerequisites and some are just decisions that need to be taken keeping in mind that during the installation phase there are some settings that cannot be changed afterwards unless redeploying the entire solution. This is something you definitely don’t want to find out after you’ve performed all the installation and configuration tasks and then have to start over again.

In this post we are going through all the prerequisites so with that out of the way we will be able to easily proceed with the deployment phase, but first let’s talk about the Horizon Workspace virtual appliances and their respective functions. The following is taken from the official documentation.

  • VMware Horizon Workspace Configurator Virtual Appliance (configurator-va): You start configuring Horizon Workspace with this virtual appliance, using both the Configurator virtual appliance interface and the Configurator Web interface. The configurations you make with the Configurator are distributed to the other virtual appliances in the vApp. Note: The configurator-va is the only component that cannot scale to multiple instances.
  • VMware Horizon Workspace Manager Virtual Appliance (service-va): Horizon Workspace Manager handles ThinApp package synchronization and gives you access to the Administrator Web interface, from which you can manage users, groups, and resources.
  • VMware Horizon Workspace Connector Virtual Appliance (connector-va): Horizon Workspace Connector provides the following services: user authentication (identity provider), directory synchronization, ThinApp-catalog loading, and View pool synchronization.
  • VMware Horizon Workspace Data Virtual Appliance (data-va): Horizon Workspace Data Virtual Appliance controls the file storage and sharing service, stores users’ data (files), and synchronizes users’ data across multiple devices.
  • VMware Horizon Workspace Gateway Virtual Appliance (gateway-va): Horizon Workspace Gateway Virtual Appliance is the single endpoint for all end-user communication. User requests come to the gateway-va virtual machine, which then routes the request to the appropriate virtual appliance.

System and Network Configuration Requirements
The preparation part is the longest and most important when deploying a distributed service such as Horizon Workspace, for this reason VMware prepared a detailed checklist to fill up before starting the installation process. The following is a list of all the things you will have to decide and mark down:

  • Horizon Workspace Fully Qualified Domain Name (FQDN)
  • Network Information for Configurator (configurator-va)
  • Network Information for Manager (service-va)
  • Network Information for Connector (connector-va)
  • Network Information for Data (data-va)
  • Network Information for Gateway (gateway-va)
  • Network Information for IP Pools
  • Active Directory Domain Controller
  • SMTP Server
  • vCenter Credentials
  • SSL Certificate (Optional)
  • Horizon Workspace License Key
  • Microsoft Windows Preview
  • External Database

Before getting into details let’s take a high level look at the architecture of Horizon Workspace as it’s meant to be in a production environment:


This picture (which is taken straight from the public documentation of the product) shows that every connection from users accessing the Horizon Workspace portal have to go through the Horizon gateway VM(s). The “(s)” easily shows how you can have one or multiple Horizon gateways, in which case you will also need some sort of load balancing mechanism in front of the gateways. The Horizon gateway virtual appliance runs nginx as web server that basically proxies every connection to the desired service so users actually need connectivity only to the gateways virtual appliances.

IMPORTANT: Placing the gateway VA in a separate network such as a DMZ network is not a supported configuration.

The following picture gives a better understanding of the network configuration requirements:


As you can see all communication go into the gateway VA and out to the other virtual appliances which are actually providing the services. Users will connect exclusively in HTTPS and the same is true also for most of communication between virtual appliances, so we will need to work a bit on SSL certificates at some point but it’s not mandatory in the setup phase as you can see form the above list since it is marked as optional in the prereqs.

Horizon Workspace FQDN
Choosing the FQDN is a tricky one because once you input it during the setup you can’t go back and change it, so it definitely deserves some thinking or you might find yourself redeploying from scratch. Most companies choose to have the same FQDN for both internal and external connections which makes it perfectly transparent for users to reach Workspace no matter where they are located; obviously the FQDN will resolve with a public IP for external users and with a private IP for internal users, hence the need of two sets of load balancers as you can see in the first picture.

Network configuration for virtual appliances
Just write down TCP/IP configurations that you intend to assign to the five virtual appliances, including DNS configuration. I encourage you to use consecutive addresses for simplicity.

IP Pools
Honestly this is a little obscure to me. IP Pools are used as a set of IP addresses that you define and assign to a network in vCenter so that they can be used when you deploy a vApp. Funny is the fact that those addresses must not be the ones you will use for setting up the virtual appliances. Even funnier is the fact that if you deploy the vApp from the Web Client you don’t even have to create an IP Pool. I have no problems admitting my ignorance here on the usefulness and meaning of this step.

Active Directory Domain Controller
Self explaining. Since Horizon Workspace integrates with your Active Directory you will need to have IP address, basic parameters and credentials handy during the setup. Just keep in mind that your users in AD will need to have Name, Last Name and email address compiled before importing them in Horizon Workspace or the import will fail.

SMTP Server
This is used by users when sharing documents. Note that you must specify a working SMTP since a check is performed during the setup and you won’t be able to proceed otherwise.

vCenter credentials
If you are deploying Horizon Workspace I’m pretty sure you have these. 🙂

SSL Certificate (optional)
I like to deal with this after the initial deployment and this is another tricky one, so during the setup we will use default self-signed certificates for simplicity.

Horizon Workspace Product Key
Yes, you need one. 🙂
For a proof-of-concept you can request a trial key that will work for 100 users.

Microsoft Windows Preview
When using Microsoft documents in Horizon Workspace web portal you can get a preview without having Microsoft Office installed. The preview can be generated with a LibreOffice add-on that runs directly on the data-va or they can be generated on a Microsoft Server with Microsoft Office installed; the first is a free option and it’s usually good enough, the latest will grant you a higher level of compatibility but you will have to pay Microsoft licenses.

External Database
If you read my last post you should know about this already.

Now that you have all handy you are ready to install Horizon Workspace.

%d bloggers like this: