How to backup, restore and schedule vCenter Server Appliance vPostgres Database

Now that we are moving away from SQL Express in favor of vPostgres for vCenter simple install on Windows and since vPostgres is the default database engine for (not so simple) install of vCSA I thought it would be nice to learn how to backup and restore this database.

Since it’s easier to perform these tasks on Windows and since there are already many guides on the Internet I will focus on vCSA because I think that more and more production environment (small and big) will be using vCSA since now it’s just as functional as vCenter if not more. (more on this in another post…)

You will find all instructions for both Windows and vCSA versions of vCenter on KB2091961, but more important than that you will find there also the python scripts that will work all the magic for you so grab the “” file and copy it to the vCSA:

scp root@<vcenter>:/tmp

For the copy to work you must have previously changed the shell configuration for the root user in “/etc/passwd” from “/bin/appliancesh” to “/bin/bash”


chmod +x
mkdir /tmp/linux_backup_restore/backups
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/VCDB.bak

All you will see when the backup is completed is:

Backup completed successfully.

You should see the backup file now:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 912K
drwx------ 2 root root 4.0K Jun 3 19:41 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak

At this point I removed a folder in my vCenter VM and Templates view, then I logged off the vSphere WebClient and started a restore:

service vmware-vpxd stop
service vmware-vdcs stop
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/VCDB.bak
service vmware-vpxd start
service vmware-vdcs start

I logged back in the WebClient and my folder was back, so mission accomplished.

Now how do I schedule this thing? Using the good old crontab but before that I will write a script that will run the backup and also give a name to the backup file corresponding to the weekday so I can have a rotation of 7 days:

_dow="$(date +'%A')"
python /tmp/linux_backup_restore/ -f /tmp/linux_backup_restore/backups/${_bak}

I saved it as “backup_vcdb” and made it executable with “chmod +x backup_vcdb”.

Now to schedule it just run “crontab -e” and enter a single line just like this:

0 23 * * * python /tmp/linux_backup_restore/backup_vcdb

This basically means that the system will execute the script every day of every week of every year at 11pm.

After the crontab job runs you should see a new backup with a name of this sort:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 1.8M
drwx------ 2 root root 4.0K Jun 3 19:46 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak
-rw------- 1 root root 900K Jun 3 19:46 VCDB_Wednesday.bak

You will also have the log files of these backups in “/var/mail/root”.

Enjoy your new backup routine 🙂

vSphere 6 Certificate Lifecycle Management

Recently I’ve been fighting with a vSphere environment and CA certificates and I thought a lot about certificate management and lifecycle in a VMware vSphere environment after that and how much it needs improvement. With the SSL Certificate Automation Tool VMware made a step in the right direction and even if the tools itself is sometimes a little buggy it is still very handy in automating a long and error prone process. In vSphere 6 VMware is taking another step in the right direction to help us create, apply and manage SSL certificates in a vSphere environment, but before talking about this we need to talk a bit about what’s new in SSO and vCenter architecture in vSphere 6. Since the introduction of SSO VMware changed its architecture in every major release, starting from 5.1 to 5.5 and now to 6.0 so let’s make a little bit of history:

Featured image

The new vSphere 6 management architecture introduces two main roles that you can deploy, these are the Management Node and the Platform Service Controller (PSC); the reason behind this separation is to have a logical entity that will take care of the main management features while another entity will hold the core and security features of the solution. What is nice about this separation is that you don’t need a 1:1 ratio between Management Nodes and PSCs so you can install PSC on separate boxes and replicate between them and then have as many Management nodes as you need (as long as you are within the same SSO domain)

Featured image

For HA scenario if you install PSC on separate boxes you will still need a load balancer. Supported solutions are Big-IP F5 and NetScaler so far.

You can obviously still install everything in one box:

Featured image

You might have noticed that the HA model for SSO was active/passive in 5.1, then active/active in 5.5 and now is active/passive again; this is due to the re-engineering of the Secure Token Service (STS) which is moving to a new and more robust method of STS (known as WebSSO) which is the same already used by vCAC (or vRealize Automation if you will) and that will be used from now forward instead of the old 5.5 method (WS-Trust). Let’s see how services are spread out on each role:

Let’s take a look to the services within the Management Node and the PSC:

Featured image

In the Management Node we can find services and features that every vSphere Admin feels very comfortable and familiar with such as vCenter Server, vSphere Web Client, Syslog Collector, etc., but two of them deserve a few words:

  • Virtual Datacenter Service: this service is new and it has been introduced to help mitigate the limitation connected with the Datacenter object in vCenter as a Management boundary.
  • (Optional) vPostgres: This component is obviously referring to the vCenter Appliance (thus optional) but I believe more and more new deployments or upgrades deserve to be considered a good fit for vCSA since VMware announced complete equality of features between vCenter installed on Windows and vCSA; leave alone the fuss of dedicating Windows licenses for vCenter which might not be a huge problem I just find the process of patching ad upgrading a vCSA simply amazing and it’s not a secret that products like EVO:RAIL make extensive use of vCSA. VMware wants to move all their services deployment model towards Virtual Appliances, this is not a secret and we need to get used to it, the sooner the better, but I’m digressing…

Featured image

In the Platform Service Controller or PSC we find our old friend SSO (we have had a rough past but now we are on better terms) and quite a few new services:

  • VMware Single Sign-On
    • Secure Token Service (STS)
    • Identity Management Service (IdM)
    • Directory Service (VMDir)
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • VMware Licensing Service
  • Authentication Framework Daemon (AFD)
  • Component Manager Service (CM)
  • HTTP Reverse Proxy

Describing all these services is out of the scope of this post but as you probably guess two of them will be our focus: the VMware Certificate Authority (or VMCA) and the VMware Endpoint Certificate Store (or VECS). But what are the roles of VMCA and VECS? The VMCA is no more or less than a CA, so you can:

  • Generate Certificates
  • Generate CRLs
  • Use the UI
  • Use the Command Line Interface to replace certificates

The VECS is where all certificates within the PSC are stored, with the only exception of the ESXi certificates that are stored locally on vSphere hosts, so here you can:

  • Store certificates and keys
  • Sync trusted certificates
  • Sync CRLs
  • Use the UI
  • Use the CLI to perform various actions

Since VMCA and VECS are part of the PSC, they will take advantage of the Multi-Master Replication Model which is offered by the Directory Service (VMDir) in order to achieve HA. In the past every service had its own user and required its own certificate but this is not the case anymore since we now have Solution Users (SU); since the number of services has increased significantly it would be impractical to manage the lifecycle of this many certificates so now we have 4 main SU that will hold the certificate used for a number of services.

Voila_Capture 2015-01-08_08-14-37_pm_white_background

What about use cases/scenarios in which I can implement VMCA? In what ways you can use this new tool?

Featured image

Scenario 1 and 2 are similar: the VMCA is the CA that releases certificates for all Solution Users (SU), the only difference is that in scenario 1 the VMCA is the root CA and you will need to distribute the Root CA Certificate so that all corporate browsers will trust it, while in scenario 2 the VMCA becomes part of an existing PKI as a subordinate CA and you certificate trust.

Featured image

In scenario 3 VMCA is installed but not used, CSRs are created and submitted to an external CA and VECS will be used to store certificates in PEM format.

Featured image

My favorite is scenario 2 because most enterprises I see already have a PKI (Microsoft CA usually) and all clients already trust the CA certificates, so adding the VMCA as s subordinate is a non disruptive process with a very low maintenance impact on the PKI itself, it protects investments already made to implement the current PKI and  preserves the knowledge to run the PKI.

Replacing certificates is still a CLI task (looks like Powershell will be involved) but VMCA and VECS are a very promising step toward the right direction for simplifying certificate lifecycle management in a vSphere environment.

%d bloggers like this: