How to backup, restore and schedule vCenter Server Appliance vPostgres Database

Now that we are moving away from SQL Express in favor of vPostgres for vCenter simple install on Windows and since vPostgres is the default database engine for (not so simple) install of vCSA I thought it would be nice to learn how to backup and restore this database.

Since it’s easier to perform these tasks on Windows and since there are already many guides on the Internet I will focus on vCSA because I think that more and more production environment (small and big) will be using vCSA since now it’s just as functional as vCenter if not more. (more on this in another post…)

You will find all instructions for both Windows and vCSA versions of vCenter on KB2091961, but more important than that you will find there also the python scripts that will work all the magic for you so grab the “linux_backup_restore.zip” file and copy it to the vCSA:

scp linux_backup_restore.zip root@<vcenter>:/tmp

For the copy to work you must have previously changed the shell configuration for the root user in “/etc/passwd” from “/bin/appliancesh” to “/bin/bash”

Then:

unzip linux_backup_restore.zip
chmod +x backup_lin.py
mkdir /tmp/linux_backup_restore/backups
python /tmp/linux_backup_restore/backup_lin.py -f /tmp/linux_backup_restore/backups/VCDB.bak

All you will see when the backup is completed is:

Backup completed successfully.

You should see the backup file now:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 912K
drwx------ 2 root root 4.0K Jun 3 19:41 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak

At this point I removed a folder in my vCenter VM and Templates view, then I logged off the vSphere WebClient and started a restore:

service vmware-vpxd stop
service vmware-vdcs stop
python /tmp/linux_backup_restore/restore_lin.py -f /tmp/linux_backup_restore/backups/VCDB.bak
service vmware-vpxd start
service vmware-vdcs start

I logged back in the WebClient and my folder was back, so mission accomplished.

Now how do I schedule this thing? Using the good old crontab but before that I will write a script that will run the backup and also give a name to the backup file corresponding to the weekday so I can have a rotation of 7 days:

#!/bin/bash
_dow="$(date +'%A')"
_bak="VCDB_${_dow}.bak"
python /tmp/linux_backup_restore/backup_lin.py -f /tmp/linux_backup_restore/backups/${_bak}

I saved it as “backup_vcdb” and made it executable with “chmod +x backup_vcdb”.

Now to schedule it just run “crontab -e” and enter a single line just like this:

0 23 * * * python /tmp/linux_backup_restore/backup_vcdb

This basically means that the system will execute the script every day of every week of every year at 11pm.

After the crontab job runs you should see a new backup with a name of this sort:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 1.8M
drwx------ 2 root root 4.0K Jun 3 19:46 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak
-rw------- 1 root root 900K Jun 3 19:46 VCDB_Wednesday.bak

You will also have the log files of these backups in “/var/mail/root”.

Enjoy your new backup routine 🙂

Preparing vPostgres 9.2 as external Database for Horizon Workspace 1.5

Today in my lab i tried to install the new version of Horizon Workspace from scratch to test if all previous steps about the external vPostgres database are still valid.

There are very few differences:

  • Supported version for Horizon Workspace 1.5 is vPostgres 9.2 instead of 9.1
  • Passwords for ‘root’ and ‘postgres’ users are not randomly generated anymore but you enter them during the OVA import process
  • After completing the importing process timezone is set to UTC; you can easily change it from the console of the virtual machine
  • When editing “postgresql.conf” you will find that the file is almost empty; all settings are in the “postgresql.conf.auto” file in the same directory which is not supposed to be changed. All the modifications that need to be applied simply need to be added as new lines in the “postgresql.conf” file.

Besides the above the procedure is identical and seems to work just fine.

How to upgrade Horizon Workspace 1.0 to 1.5

Since a new version of Horizon Workspace came out I’ve been curious about differences and improvements, so i upgraded it to version 1.5.

I have to say that, besides some differences here and there in the look and feel and the new functionality to manage VMware Ready Android devices, I have already found a nice improvement in the way you mount the NFS store in the data-va because finally the procedure simply works and there’s no need of crazy workarounds. I also suspect that they improved a lot the way you change self-signed certificates compared to what it used to be; more on this when i will have tested it.

Note: You can only upgrade if you have completed the setup procedure and you have a fully functional environment.

All commands need to be run from the configurator-va, just as usual, so let’s SSH into it, then:

su -
/usr/local/horizon/lib/menu/updatemgr.hzn check

You should get this output:

Image

As you can see it won’t recognize my vPostgres and this is normal. Since I’ve started this series of posts about Horizon Workspace installing the external database on vPostgres i also added the VM to the VApp to make sure it starts for first and stop for last. The first time i tried to upgrade i decided to ignore this error and went ahead with the upgrade which failed and left my Horizon Workspace installation in an unusable state so i reverted the snapshots, removed the vPostgres from the VApp and rerun the command and i could see no issue at that point.

Before going on and since it’s been so useful for me, i will show you how to snapshot all your VMs in the VApp at once so you can revert if you need. Since you might have more than just 5 VMs i like to use PowerCLI to take over this kind of task simply because it’s faster than working around the GUI:

Get-VApp Horizon-Workspace | Get-VM | New-Snapshot -Name "UpgradeHorizon" -Description "Horizon Workspace upgrade from version 1.0 to 1.5" -Memory -Quiesce -Confirm:$false -RunAsync

Given that you didn’t change the default name of the VApp (Horizon-Workspace) this will snapshot all the VMs included in it.

In the vSphere client you should see this:

Image

Once they are all completed we can go back to the configurator-va and start the upgrade with peace of mind:

/usr/local/horizon/lib/menu/updatemgr.hzn update

Now you should see a whole lot of downloading and installing. In my lab it took about 40 minutes for the whole procedure.

When the upgrade is done you have to shutdown and restart the VApp; since i still have the PowerCLI open i used it:

Stop-VApp -VApp Horizon-Workspace -Confirm:$false
Start-VApp -VApp Horizon-Workspace -Confirm:$false

You shouldn’t start it again until it’s stopped, so in order to see the status of the command i’ve omitted “RunAsync”.

Now check that your Horizon Workspace environment still works then you can check again for updates to see the current status:

Image

That looks al good, now you just have to remember to update your preview binaries in the data-va if you used the LibreOffice preview because there’s a new version, just do as usual.

If you’re happy with the result you can remove the snapshots we created earlier:

Get-VApp Horizon-Workspace | Get-VM | Get-SnapShot | Where { $_.Name.Contains("ToolsUpgrade") } | Remove-Snapshot -RemoveChildren -Confirm:$false -RunAsync

In my case i also had to put back the vPostgres VM in the VApp.

That’s pretty much it.

DISCLAIMER: PowerCLI is a powerful tool and should not be used to throw in some commands without understanding the effects. Before running PowerCLI commands in your production environment always test them and adjust them for your situation. I take no responsibility if you don’t test things out yourself first and your Horizon Workspace gets broken.

UPDATE: In the release notes it is clearly stated that in the documentation there are missing steps in case you have multiple service-va VMs. Please refer to the Horizon Workspace 1.5 release notes for the additional steps.

How to install Horizon Workspace using an external database

In the previous posts we’ve taken care of all the preparation steps so now we should be ready to get down to business and install Horizon Workspace.

First of all download the Horizon Workspace OVA from the VMware website and get a product key; trial is good if you didn’t purchase one yet and it should be ok for proof-of-concept.

Once you have the OVA file you can import it in vCenter using the usual “Deploy OVF from template” menu.

While we wait for the upload to complete we need to create records in our DNS server for every virtual appliance. Here’s how i configured mine in my lab:

192.168.110.6			cofigurator.vsphere.lab
192.168.110.7			service01.vsphere.lab
192.168.110.8			connector01.vsphere.lab
192.168.110.9			data01.vsphere.lab
192.168.110.10		        gateway01.vsphere.lab

You also need to create PTR records, if you have a Windows based DNS this can be done simply by selecting a flag while creating A records:

DNS

Note: You must have created the reverse zone in DNS before using this flag. Otherwise you can manually create PTR records in the reverse zone.

Go through the wizard, once you get to setting up the network for the virtual appliances fill gateway, DNS, subnet mask and pick a port group:

Ova network

If you followed previous posts you should have no problems filling up all the information. Make sure all virtual appliances are on the same network segment.

Now let’s assign IP addresses to different virtual appliances according to the DNS records we created:

Ova network 2

During setup the name of the virtual appliances will be assigned based on reverse lookup query of every IP address so it’s important to also to create PTR records because A records are not used for DNS reverse lookups and if they are missing the setup process will fail.
In my case i don’t have a specific timezone for my country but since i live in GMT+1 i can choose Paris. If you don’t know what to choose or make the wrong choice here you can change it later on and you will also have more options to choose from.

In case you want to change the timezone later, just finish the whole setup and before starting configuration ssh to all VAs, get ‘root’ prompt and run these commands:

date
rm /etc/localtime
ln -s /usr/share/zoneinfo/Europe/Rome /etc/localtime


The date command in the beginning and end are useful to see if the operation was successful.

Note: When you get to the end of the wizard remember to check the flag so that the vApp is powered on after deployment.

After the deployment you’ll notice that only the configurator-va will be powered on as this is where we will setup the whole Horizon Workspace solution, so let’s start by connecting to the console of the configurator-va with the vSphere Client or Web Client as you prefer. You will be asked to press enter to start setup and here is where your DNS reverse records will be checked:

Config 1

Make sure there is correspondence between what you see here and the names and IP addresses you wanted to assign. If everything is correct you can confirm and go ahead; you will be asked for ‘root’ password do be assigned to all Vas plus a bunch of settings we described when we compiled the checklist in previous posts:

Config 2

The interesting thing to note here is that the suggested FQDN would be the same name of the gateway that we set in the DNS records, but we want to put this out on the internet so we are choosing “workspace.myvirtualife.net”.

Remember: the FQDN cannot be changed after deployment. The only supported option to change it is redeploying the whole thing from scratch.

Note: You need to use a valid SMTP server or setup will stop.

After answering to all questions, which include SMTP and vCenter parameters, you will see quite a few things happening:

  • turning on and preparing VMs
  • setting root passwords
  • setting timesync
  • generating self-signed ssl certificated
  • setting workspace FQDN
  • configuring virtual appliances firewalls
  • starting webapps

Grab something to drink, make phone calls… this takes a while. At the end you will be instructed to press enter and connect to the configurator-va via HTTPS.

Config 4

Admin user is a local user that can access the configurator appliance in case of problems, but it’s not the Horizon Workspace administrator.

Fill up licensing details and click “Next”.

In the Database Connection Setup let’s pick the External Database option and use the vPostgres instance we created earlier:

jdbc:postgresql://192.168.110.16/saas?stringtype=unspecified


click “Next”:

Db 1

Error while testing DB connection. I/O error: workspace.myvirtualife.net; nested exception is java.net.UnknownHostException: workspace.myvirtualife.net

Now, this is where it gets interesting!
We get an error, so we must have made some mistake in the vPostgres database… well, no we didn’t. The problem here is that the configurator-va doesn’t know the host “workspace.myvirtualife.net” because we didn’t create an A record for it in the DNS, so add it then try again:

Db 2

Error while testing DB connection. I/O error: No route to host; nested exception is java.net.NoRouteToHostException: No route to host

Ok, another error, we must have made a mistake. ACTUALLY NO. I pointed the Workspace record to the Load Balancer as suggested in the documentation but the setup doesn’t like this choice. After a while i figured it’s because in this phase we need to point it to the gateway-va ip address so it can correctly recognize itself, so go change the workspace record in DNS and try again:

Db 5

Error creating admin user. hostname in certificate didn’t match: !=

Crap. Error again. A different one.
I know it doesn’t look like it but we are making progresses here.
At this point i understood why the setup wanted me to use the gateway-va FQDN as Workspace FQDN. Our problem now is that the “workspace.myvirtualife.net” hostname doesn’t match with the common name of the certificate that has been generated for the gateway-va which in my case it “gateway01.vsphere.lab”, but we need something that can be used outside on the internet so what do we do now?

I found the solution in the VMTN communities which are always a great resource. First let’s connect to the configurator-va with the user ‘sshuser’ with the same password we chose for ‘root’ during setup, type “su -” and insert the ‘root’ password and once we have the “#” prompt the do the following:

cd /usr/local/horizon/lib/menu/secure
./wizardssl.hzn --makesslcert gateway-va 'workspace FQDN'
./wizardssl.hzn


In my case:

cd /usr/local/horizon/lib/menu/secure
./wizardssl.hzn --makesslcert gateway-va workspace.myvirtualife.net
./wizardssl.hzn


Now all certificates are generated again and pushed to all virtual appliances, but the gateway-va certificate will match to the Workspace FQDN, so now we should be set, let’s try again:

Db 3

Whooa! Finally. You don’t know how much it took me to figure this out. No really, stop guessing. You don’t WANT to know.
Another option i could think of is to configure the load balancer before setting up Horizon Workspace, and point “workspace.myvirtualife.net” record to it but it think it’s not practical to set up a load balancer if the application is not up yet because you would have no way to test it.

In other words what i do is:

  • standard setup
  • set internet FQDN as Workspace FQDN
  • temporary point Workspace FQDN to gateway-va
  • recreate certificates
  • complete setup

Later on we will complete the job configuring load balancers, changing DNS entries to point at them and generating new certificates that are not self-signed so all pieces fall into place. More on these activities in later posts.

Now it’s time to configure Active Directory integration:

Directory

Configure using your Active Directory LDAP structure.

The user ‘workspace’ is a user i created earlier in Active Directory and that is the user that will function as Horizon Workspace admin.

Click “Next”.

Accept defaults for user mapping.

Click “Next”.

Let’s discover the users:

Users

Note: If you see an error tab it’s most likely because you didn’t compile fields Name, Last Name and Email for all users.

Now selecting groups:

Groups

Click “Add” next to the Active Directory groups that you want to add to the Horizon Workspace.

For SSL certificates just leave defaults and click “Next”.

In the “Select Modules” page enable all modules but the View module and click “Next”.

Summary

Click “Go to Horizon Workspace” and in the login screen use the credentials you’ve set during setup:

Login

Note: The password was set on the user when it has been created in Active Directory.

You should get here:

Login 1

Congrats! You’ve setup Horizon Workspace and in the coming posts we will complete the job installing load balancers, taking care of SSL Certificates and so on.

Preparing vPostgres as external Database for Horizon Workspace

Since most of blog posts around the internet speak about how to make a basic installation of Horizon Workspace i decided to start a series of articles with the intention to show a full blown Horizon Workspace setup suited for production use.

A basic installation is good for a proof-of-concept or a lab environment but you will see very fast how it won’t scale and how it won’t even be supported by VMware itself since, for example, it uses an internal DB.

Even if the internal DB is based on VMware vFabric Postgres, or vPostgres, you still need to have a separate VM with vPostgres dedicated to be the Horizon Workspace database if you want a supported configuration in your production environment.

If you want to install vPostgres you have 2 options: download RPM packages and install them on a supported platform or download the OVA appliance and import it in vCenter.

I am no DBA and i find it not fun at all to learn database stuff if i don’t have to so it was a pretty easy choice for me. I also found that in the documents section of the Horizon Workspace page in the VMTN community there is a nice guide about that specific scenario.

The following is my experience deploying the OVA and preparing vPostgres for Horizon Workspace.

Install and Setup Notes
First thing first you have to download the OVA from the VMware website. There is a trial version to use which if i’m not mistaking last 60 days before expiring. Note that we will need version 9.1 even if there is a newer version simply because this is the version supported in Horizon Workspace at the moment of writing.

Once the download is complete we can use standard tools and procedures to import the virtual appliance in your environment:

Deploy ovf
Use vSphere Client or Webclient to deploy the OVF file.

Deploy ovf 2
The process is pretty straightforward all the way down to the network configuration.

Now we have to log in as ‘root’ and this can be sometimes tricky because we don’t get to choose what the password is since it’s randomly generated when the OVA is imported but no need to worry, the password is shown on the login screen. It’s tricky also because the password is really long and complex and sometimes i haven’t been able to type it in correctly, no matter what (there is also a time window for typing it which doesn’t help either). If this happens to you as well just redeploy the virtual appliance and hope for a more friendly password, i’m sure you won’t need to deploy it more than twice to successfully log in.

Root login

Once you are successfully logged in as ‘root’ we need to change the password as indicated in the login screen using the following command:

/opt/aurora/sbin/set_password

Note that the same password will be set to both the ‘root’ and ‘postgres’ users; this is worth noting because by default you can SSH into the virtual appliance only with the ‘postgres’ user.

If you want to change the network configuration you can run the following command:

/opt/vmware/share/vami/vami_config_net

While logged in as ‘root’ edit the following config file:

vi /var/vmware/vpostgres/current/pgdata/postgresql.conf

Find the “max_connections” value and change it from 100 to 600:

max_connections = 600

Add a line right under the “max_connections” line:

search_path = 'saas'

Save the file and change the directory into “/opt/vmware/vpostgres/current/bin” and su to the ‘postgres’ user:

cd /opt/vmware/vpostgres/current/bin
su postgres

Verify that you are indeed in the “/opt/vmware/vpostgres/current/bin” folder:

pwd

Now we are going to create a script that will take care of all the operations needed to prepare the database for Horizon Workspace, paste the following into a text file:

CREATE ROLE horizon LOGIN
        PASSWORD '123456'
        NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION;

ALTER ROLE horizon;

CREATE DATABASE saas
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
CONNECTION LIMIT = -1;
GRANT CONNECT, TEMPORARY ON DATABASE saas TO public;
GRANT ALL ON DATABASE saas TO postgres;
GRANT ALL ON DATABASE saas TO horizon;

\connect saas;
CREATE SCHEMA saas AUTHORIZATION horizon;
SET search_path = saas;
CREATE EXTENSION citext SCHEMA saas;

Change the ‘PASSWORD’ to whatever you want to use when you will make a connection to this DB with the ‘horizon’ user and save the text into a file, for example “/tmp/createdb.do”.

Now let’s execute the script:

./psql -f /tmp/createdb.do

If all goes well you should get an output similar to this:

CREATE ROLE
ALTER ROLE
CREATE DATABASE
GRANT
GRANT
GRANT
You are now connected to database "saas2" as user "postgres".
CREATE SCHEMA
CREATE EXTENSION

At this point the documentation states to reboot the appliance so go ahead.

Before installing Horizon Workspace we will use a tool called pgAdmin to check that all went as expected; pgAdmin is available for almost every platform so just pick what you prefer and install it on your management workstation.

Let’s open pgAdmin and check everything is all right:

Pgadmin 1
Let’s connect to the DB with the credentials we’ve set.

Pgadmin 2
Check the user ‘horizon’ attributes which should look quite familiar.

Pgadmin 3
Same thing for the DB properties.

All right! We are now ready to install Horizon Workspace and point it to this newly created database.

In the coming articles i’ll show you just how to do that.

Licensing Notes
vPostgres is not part of the Horizon Workspace product or the Horizon Suite so you will have to purchase it separately.

If you want to check the status of the trial license you can do it with the following command:

cat /opt/vmware/vFabric/vf.vpg.log

If you see the following in the log file it means you are running on trial license:

[Mon Jul 22 15:32:35 2013] WARNING getComponentLicenseState: No local serial numbers, and the server hasn't yet provided a license state, using default evaluation licenses, exception: No server license state available
[Mon Jul 22 15:32:35 2013] INFO getComponentLicenseState: No local serial numbers, nothing from server, using default evaluation licenses.

If you purchased a valid license you can use it by creating the following file and pasting your licensing information in it:

vi /etc/opt/vmware/vfabric/vf.vpg-serial-numbers.txt

Here’s a sample of what a valid license information should look like:

XXXXX-XXXXX-XXXXX-XXXXX-XXXXX    [quantity=20, expiration=Permament, addon=1]

After that we need to change permissions on the file so that vPostgres can read it:

chgrp vfabric /etc/opt/vmware/vfabric/vf.vpg-serial-numbers.txt
chmod 644 /etc/opt/vmware/vfabric/vf.vpg-serial-numbers.txt

Now a restart of the service is needed and then we can check the log again:

service aurora_mon stop
service aurora_mon start
cat /opt/vmware/vFabric/vf.vpg.log

You should be able to read that vPostgres found a valid serial number:

[Mon Jul 22 15:57:05 2013] INFO getComponentLicenseState: Using local serial numbers

Sizing Notes
In the community document i mentioned earlier you will also find some useful indications about how to size the vPostgres VM and how to edit the virtual appliance settings in details to support up to 30000 users with Horizon Workspace. You can also install a second instance of vPostgres and setup an HA configuration at the database level if vSphere HA is not good enough for your availability concearns.

Here’s a brief description of sizing requirements for 30000 users:

Sizing

Supported Platforms for manual RPM installation
If you want to build your own VM and manually install vPostgres using RPMs keep in mind the following list of supported operating systems:

  • Red Hat Linux RHEL 6.2 (64 bit)
  • SUSE Linux SLES 11 SP 1 (64 bit)

Refer to the official documents for detailed instructions on how to manually install vPostgres on supported platforms using RPMs.

%d bloggers like this: