How to backup, restore and schedule vCenter Server Appliance vPostgres Database

Now that we are moving away from SQL Express in favor of vPostgres for vCenter simple install on Windows and since vPostgres is the default database engine for (not so simple) install of vCSA I thought it would be nice to learn how to backup and restore this database.

Since it’s easier to perform these tasks on Windows and since there are already many guides on the Internet I will focus on vCSA because I think that more and more production environment (small and big) will be using vCSA since now it’s just as functional as vCenter if not more. (more on this in another post…)

You will find all instructions for both Windows and vCSA versions of vCenter on KB2091961, but more important than that you will find there also the python scripts that will work all the magic for you so grab the “linux_backup_restore.zip” file and copy it to the vCSA:

scp linux_backup_restore.zip root@<vcenter>:/tmp

For the copy to work you must have previously changed the shell configuration for the root user in “/etc/passwd” from “/bin/appliancesh” to “/bin/bash”

Then:

unzip linux_backup_restore.zip
chmod +x backup_lin.py
mkdir /tmp/linux_backup_restore/backups
python /tmp/linux_backup_restore/backup_lin.py -f /tmp/linux_backup_restore/backups/VCDB.bak

All you will see when the backup is completed is:

Backup completed successfully.

You should see the backup file now:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 912K
drwx------ 2 root root 4.0K Jun 3 19:41 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak

At this point I removed a folder in my vCenter VM and Templates view, then I logged off the vSphere WebClient and started a restore:

service vmware-vpxd stop
service vmware-vdcs stop
python /tmp/linux_backup_restore/restore_lin.py -f /tmp/linux_backup_restore/backups/VCDB.bak
service vmware-vpxd start
service vmware-vdcs start

I logged back in the WebClient and my folder was back, so mission accomplished.

Now how do I schedule this thing? Using the good old crontab but before that I will write a script that will run the backup and also give a name to the backup file corresponding to the weekday so I can have a rotation of 7 days:

#!/bin/bash
_dow="$(date +'%A')"
_bak="VCDB_${_dow}.bak"
python /tmp/linux_backup_restore/backup_lin.py -f /tmp/linux_backup_restore/backups/${_bak}

I saved it as “backup_vcdb” and made it executable with “chmod +x backup_vcdb”.

Now to schedule it just run “crontab -e” and enter a single line just like this:

0 23 * * * python /tmp/linux_backup_restore/backup_vcdb

This basically means that the system will execute the script every day of every week of every year at 11pm.

After the crontab job runs you should see a new backup with a name of this sort:

vcenter:/tmp/linux_backup_restore/backups # ls -lha
total 1.8M
drwx------ 2 root root 4.0K Jun 3 19:46 .
drwx------ 3 root root 4.0K Jun 3 19:28 ..
-rw------- 1 root root 898K Jun 3 19:29 VCDB.bak
-rw------- 1 root root 900K Jun 3 19:46 VCDB_Wednesday.bak

You will also have the log files of these backups in “/var/mail/root”.

Enjoy your new backup routine 🙂

Using vCSA 6.0 as a Subordinate CA of a Microsoft Root CA

One of the nicest improvements in vSphere 6 is the ability to use the VMware Certificate Authority (VMCA) as a subordinate CA.
In most cases enterprises already have some form of PKI deployed in house and very often it is Microsoft based so I will show you how I did it with a Microsoft Enterprise CA.

I give for granted that the Microsoft PKI is already in place, in my case it is a single VM with an Enterprise Microsoft CA installed.

The vCSA should also be already be in place.

As first step I edit the certool config file but first I make a backup of the default configuration:

mkdir /root/backup
cp /usr/lib/vmware-vmca/share/config/certool.cfg /root/backup
vi /usr/lib/vmware-vmca/share/config/certool.cfg

Compile the config file with the parameters that are good for your setup then save the file and exit.

Now we have to generate a certificate request for the VMCA to pass to the Microsoft CA and there are many ways to do that, I am going to use the vSphere Certificate Manager Utility that will automatically take most steps for me:

/usr/lib/vmware-vmca/bin/certificate-manager

Screen Shot 2015-03-29 at 23.20.25
At this point I have the .csr file (/root/root_signing_cert.csr) and the private key (/root/root_signing_cert.key) so let’s feed it to the Microsoft CA as you normally would for any certificate request using the “Subordinate Certification Authority” template:

Screen Shot 2015-03-29 at 23.25.43Now you have to take the crt file in base64 format on the vCSA and also the Microsoft CA root certificate in base64 format as well; copying files with SCP will be a challenge because the root user on the vCSA by default doesn’t use the bash shell so if you want to use this method you need to edit the “/etc/passwd” and set the root user to use bash as a shell and then you can put it back as it was once you are done transferring the files.

It could be just simpler to open the certs on your computer and the connect to the vCSA via SSH and copy the content inside new files; one way or another you need to take the certificates on the vCSA, in my case they are “root_signing_cert.pem” and “cam.pem”.

Now we need to combine the two files in a chain file:

cp root_signing_cert.pem caroot.pem
cat ca.pem >> caroot.pem

If you open the “caroot.pem” file you should see a single cert file with both ca and certificate one after another.

Now we can go back to the vSphere Certificate Manager Utility to apply this certificate:

Screen Shot 2015-03-29 at 23.36.16

Since we have already edited the certool.cfg file we just have to confirm the values that the wizard proposes, just remember to enter the FQDN of the vCenter server:

Screen Shot 2015-03-29 at 23.37.18
If you have a successful outcome you can connect via browser to your vSphere Web Client and check the certificate:


Screen Shot 2015-03-29 at 23.39.59

Screen Shot 2015-03-29 at 23.40.08

 

As you can see now this is a trusted connection and the VMCA has released certificates for the Solution Users on behalf of the Microsoft Root CA.

You can check the active certificate in the vSphere Web Client in the Administration section:

Screen Shot 2015-03-29 at 23.42.30

In case you decide to remove the original root certificate then you will have to refresh the Security Token Service (STS) Root Certificate, and replace the VMware Directory Service Certificate following the vSphere 6 documentation.

Now the VMCA is capable of signing certificates that are valid in you PKI chain and are trusted by default in you Windows domain by all clients.

 

How To Deploy vCSA 6.0 with a Mac

The new vCenter Server Appliance has a new deployment model, both architectural wise and installation wise.

I wrote extensively about the architectural changes in this post, so I will focus on how to deploy it with a Mac using command line tools since if you want to use the graphical setup you need to be running Windows.

In order to do this you need the ISO file of the vCSA mounted in your Mac.

In “/Volumes/VMware VCSA/vcsa-cli-installer/mac” you will find a script called “vcsa-deploy” that requires a JSON file with all the parameters needed to deploy and configure the VCSA on your host.

You can find templates of JSON files in “/Volumes/VMware VCSA/vcsa-cli-installer/templates”, here is how I compiled mine in order to obtain a single VM with all vCenter and PSC services:

{
    "__comments":
    [
        "Sample template to deploy a vCenter Server with an embedded Platform Services Controller."
    ],

    "deployment":
    {
        "esx.hostname":"192.168.1.107",
        "esx.datastore":"vsanDatastore",
        "esx.username":"root",
        "esx.password":"12345678",
        "deployment.option":"tiny",
        "deployment.network":"LAN",
        "appliance.name":"vCenter",
        "appliance.thin.disk.mode":true
    },

    "vcsa":
    {

        "system":
        {
            "root.password":"12345678",
            "ssh.enable":true
        },

        "sso":
        {
            "password":"12345678",
            "domain-name":"vsphere.local",
            "site-name":"Default-First-Site"
        },

        "networking":
        {
            "ip.family":"ipv4",
            "mode":"static",
            "ip":"192.168.110.2",
            "prefix":"24",
            "gateway":"192.168.110.254",
            "dns.servers":"8.8.8.8",
            "system.name":"192.168.110.2"
        }
    }
}

You can see how I used the newly created “vsanDatastore” as my destination datastore.

Your SSO password will be checked against complexity compliance by the script before starting the deployment process.
Passwords are stored in clear text so make sure not to leave around this file and possibly destroy it after use or change all the passwords right after deployment.
You might have noticed that as the system name I used the IP address: I had to do this because I have no DNS (yet) and if you enter a FQDN as system name you need to make sure that it can be resolved both with forward and reverse DNS calls so I had no choice; this will actually be a limitation later on because I will not be able to add the vCSA to a Windows domain so if I want to use Windows credentials to log in my vCenter I will need to setup LDAP authentication.

You just fire this command to start the deployment:

/Volumes/VMware\ VCSA/vcsa-cli-installer/mac/vcsa-deploy vcenter60.json

During the deployment process you will see the following:

Start vCSA command line installer to deploy vCSA "vCenter60", an embedded node.

Please see /var/folders/dp/xq_5cxlx2h71cgy2t83ghkd00000gn/T/vcsa-cli-installer-9wU8aB.log for logging information.

Run installer with "-v" or "--verbose" to log detailed information.

The SSO password meets the installation requirements.
Opening vCSA image: /Volumes/VMware VCSA/vcsa/vmware-vcsa
Opening VI target: vi://root@192.168.1.107:443/
Deploying to VI: vi://root@192.168.1.107:443/

Progress: 99%
Transfer Completed
Powering on VM: vCenter60

Progress: 18%
Power On Completed

Installing services...
Progress: 5%. Setting up storage
Progress: 50%. Installing RPMs
Progress: 56%. Installed oracle-instantclient11.2-odbc-11.2.0.2.0.x86_64.rpm
Progress: 62%. Installed vmware-identity-sts-6.0.0.5108-2499721.noarch.rpm
Progress: 70%. Installed VMware-Postgres-9.3.5.2-2444648.x86_64.rpm
Progress: 77%. Installed VMware-invsvc-6.0.0-2562558.x86_64.rpm
Progress: 79%. Installed VMware-vpxd-6.0.0-2559267.x86_64.rpm
Progress: 83%. Installed VMware-cloudvm-vimtop-6.0.0-2559267.x86_64.rpm
Progress: 86%. Installed VMware-sps-6.0.0-2559267.x86_64.rpm
Progress: 87%. Installed VMware-vdcs-6.0.0-2502245.x86_64.rpm
Progress: 89%. Installed vmware-vsm-6.0.0-2559267.x86_64.rpm
Progress: 95%. Configuring the machine
Service installations succeeded.

Configuring services for first time use...
Progress: 3%. Starting VMware Authentication Framework...
Progress: 11%. Starting VMware Identity Management Service...
Progress: 14%. Starting VMware Single Sign-On User Creation...
Progress: 18%. Starting VMware Component Manager...
Progress: 22%. Starting VMware License Service...
Progress: 25%. Starting VMware Service Control Agent...
Progress: 33%. Starting VMware System and Hardware Health Manager...
Progress: 44%. Starting VMware Common Logging Service...
Progress: 55%. Starting VMware Inventory Service...
Progress: 64%. Starting VMware vSphere Web Client...
Progress: 66%. Starting VMware vSphere Web Client...
Progress: 70%. Starting VMware ESX Agent Manager...
Progress: 74%. Starting VMware vSphere Auto Deploy Waiter...
Progress: 81%. Starting VMware Content Library Service...
Progress: 85%. Starting VMware vCenter Workflow Manager...
Progress: 88%. Starting VMware vService Manager...
Progress: 92%. Starting VMware Performance Charts...
Progress: 100%. Starting vsphere-client-postinstall...
First time configuration succeeded.

vCSA installer finished deploying "vCenter60", an embedded node:
System Name: 192.168.110.20
Login as: Administrator@vsphere.local

It's time to connecto the the new Web Client, just open your browser to "https://" and then select "Log In To the vSphere Web Client".

You should now log in but before starting the normal configuration process I suggest you take care of password expiration in which present in two separate areas in this version of vCSA: the SSO users and the root system user.
About the first one you can go to Administration -> Single Sign-On -> Configuration -> Password Policy and edit the Maximum Lifetime to “0” so effectively you are disabling expiration:

Featured image

For the root user you will need to drop to the vCSA command line, enable and access the Shell the issue the following:

localhost:~ # chage -l root        # show current password expiration settings

localhost:~ # chage -M -1 root     # set expiration to Never
Aging information changed.
localhost:~ # chage -l root
Minimum: 0
Maximum: -1
Warning: 7
Inactive: -1
Last Change: Mar 17, 2015
Password Expires: Never
Password Inactive: Never
Account Expires: Never

Now you could start deploying all your VMs but if you try that you will find that vSAN will complain about a policy violation!

Do you remember how we needed to change the default policy on the host before we could deploy vCSA?
We did that at the host level but when vCSA started managing the host the default policy has been overwritten to the original defaults so now we have to change it again to match our need but this time we can leverage the GUI for this task:

Featured image

Now all is set and you should be good to go… not really!
We’ve never set a network for the vSAN traffic, even if I’m running on a single node configuration this will still trigger a warning:

Featured image

All you have to do is create a new VMKernel portgroup and flag it for vSAN traffic and your will system be again a little happy vSphere host.

How to replace default VCSA 5.5 certificates with Microsoft CA signed certificates

DISCLAIMER: This is a very lenghty procedure and I’ve changed some steps from the original KB trying to make it shorter; if I made some mistakes please let me know.

I don’t do this all the time but today I had to replace SSL certificates on a vCenter Virtual Appliance and since I know this will happen more and more often I thought I should write a shorter procedure since VMware KB is very detailed and, yet again, very long. At least it’s not as long as the infamous 96 steps of version 5.1.

Before proceding it’s good practice to shutdown your vCSA and take a snapshot.

Go to http://vcenter_ip_address:5480 or http://fqdn:5480 and chack that the “Certificate regeneration enabled” setting in the Admin tab of the vCSA web interface is set to “No” or we will lose all our work at first reboot:

1

Also, since we are going to use a Microsoft CA for this tutorial, it would be a good idea to take a look at KB2062108 and complete those steps before proceeding.

Note: This procedure is specific for vCSA 5.5. If you have a previous version of vCSA please refer to KB2036744.

Download and install the latest build of OpenSSL 0.9.8 on a machine of your choice. For convenience I installed it on a Windows VM in “C:\OpenSSL”.

Create the following folders:

C:\OpenSSL\Certs
C:\OpenSSL\Certs\vCenterSSO
C:\OpenSSL\Certs\InventoryService
C:\OpenSSL\Certs\LogBrowser
C:\OpenSSL\Certs\AutoDeploy

Open a text editor:

[ req ]
default_md = sha512
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = testpassword
output_password = testpassword

[ v3_req ]
basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:vcva55, IP: 10.0.0.10, IP:ServerIPv6Address, DNS: vcva55.vmware.com

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = NY
localityName = New York
0.organizationName = VMware
organizationalUnitName = vCenterApplianceUniqueServer
commonName = vcva55.vmware.com

Change the following lines:

  • subjectAltName: insert here data about name and IP of your vCSA (you can omit IPv6 if you don’t use it)
  • commonName: this must be your vCSA FQDN
  • all section [req_distinguished_name]
  • leave organizationalUnitName as it is

Save the file as “C:\OpenSSL\Certs\openssl_generic.cfg”.

We need to generate one .cfg file for each service, changing the “organizationalUnitName” by opening the “openssl_generic.cfg” file we just created:

  • organizationalUnitName = VMware vCenter Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_vpxd.cfg”)
  • organizationalUnitName = VMware Inventory Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_inventoryservice.cfg”)
  • organizationalUnitName = VMware LogBrowser Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_logbrowser.cfg”)
  • organizationalUnitName = VMware vSphere Autodeploy Service Certificate (save as “C:\OpenSSL\Certs\vCenterSSO\openssl_autodeploy.cfg”)

You should now have a .cfg file for each service in each folder with a different organizationalUnitName.

To generate the certificate requests, assuming you have the same path I have, you can use the following commands.

cd c:\OpenSSL\bin

openssl req -new -nodes -out c:\openssl\certs\vCenterSSO\rui_vpxd.csr -keyout c:\openssl\certs\vCenterSSO\rui_vpxd.key -config c:\openssl\certs\vCenterSSO\openssl_vpxd.cfg

openssl req -new -nodes -out c:\openssl\certs\InventoryService\rui_inventoryservice.csr -keyout c:\openssl\certs\InventoryService\rui_inventoryservice.key -config c:\openssl\certs\InventoryService\openssl_inventoryservice.cfg

openssl req -new -nodes -out c:\openssl\certs\LogBrowser\rui_logbrowser.csr -keyout c:\openssl\certs\LogBrowser\rui_logbrowser.key -config c:\openssl\certs\LogBrowser\openssl_logbrowser.cfg

openssl req -new -nodes -out c:\openssl\certs\AutoDeploy\rui_autodeploy.csr -keyout c:\openssl\certs\AutoDeploy\rui_autodeploy.key -config c:\openssl\certs\AutoDeploy\openssl_autodeploy.cfg

Now you should also have a .key file and a .csr file in each respective directory.

To generate certificates from the .csr file login your Microsoft CA web interface (by default it is http://servername/CertSrv/):

  1. Click the Request a certificate link.
  2. Click advanced certificate request.
  3. Click the Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file link.
  4. Open the certificate request (rui_service.csr, as generated above for each component) in a plain text editor and paste this text into the Saved Request box.
  5. Select the Certificate Template as VMware Certificate.
  6. Click Submit to submit the request.
  7. Click Base 64 encoded on the Certificate issued screen.
  8. Click the Download Certificate link.
  9. Save the certificate as rui_service.crt, in the appropriate C:\OpenSSL\Certs\<service>\ folder.  (for example rui_vpxd.crt)
  10. Repeat Steps 2 to 10 for each of the additional service.
  11. Navigate back to the home page of the certificate server and click Download a CA certificate, certificate chain or CRL.
  12. Click the Base 64 option.
  13. Click the Download CA Certificate chain link.
  14. Save the certificate chain as cachain.p7b in the c:\openssl\certs\ directory.

By default, Microsoft CA certificates are generated with the .cer format. Either use Save As or change it to .crt before continuing.

When complete, you have four certificates (rui_service.crt) for each of the services generated in their respective c:\openssl\certs\<services> folders and the cachain.p7b file in the c:\openssl\certs\ folder.

Copy the c:\openssl\certs folder on the root of the vCenter filesystem via SCP, rename it to “ssl”, SSH to the vCSA, then:

service vmware-stsd stop
service vmware-vpxd stop

Rename all files in the service folders so that the .key file is named “rui.key” and the .crt file is named “rui.crt”.

On the vCenter Appliance, move where the cachain.p7b file is, then convert it to cachain.pem:

openssl pkcs7 -print_certs -in cachain.p7b -out cachain.pem

Now open cachain.pem with a text editor and remove any text before the first “—–BEGIN CERTIFICATE—–” and after “—–END CERTIFICATE—–“.

Note: This assumes there are no intermediate certificates in the Certificate Authority.

Copy the cachain.pem file in every service folder.

cd <vcenterservicefolder>
cat rui.crt cachain.pem > chain.pem
/usr/sbin/vpxd_servicecfg certificate change chain.pem rui.key

If all goes well you should receive this:

VC_CFG_RESULT = 0

Check KB2057248 if you get a different result.

service vmware-stsd start
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk

Create the chain.pem file for every service:

cat rui.crt cachain.pem > chain.pem

Then:

cd <inventoryservicefolder>
openssl pkcs12 -export -out rui.pfx -in chain.pem -inkey rui.key -name rui -passout pass:testpassword
cp rui.key /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.crt /usr/lib/vmware-vpx/inventoryservice/ssl
cp rui.pfx /usr/lib/vmware-vpx/inventoryservice/ssl
cd /usr/lib/vmware-vpx/inventoryservice/ssl/
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./02-inventoryservice --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
</em>rm /var/vmware/vpxd/inventoryservice_registered
service vmware-inventoryservice stop
service vmware-vpxd stop
service vmware-inventoryservice start
service vmware-vpxd start

Note: As there is a plain-text password on the above command, to avoid the history file showing the contents of the password because it is in plain text in the command above, run the unset HISTFILE command prior to executing any step containing a password.

Note: The default SSO administrator username for vCenter Single Sign-On 5.5 is administrator@vSphere.local

cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode uninstall --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk
cd <logbrowserservicefolder>
<code>openssl pkcs12 -export –out rui.pfx –in chain.pem -inkey rui.key –name rui –passout pass:testpassword</code>
cp rui.key /usr/lib/vmware-logbrowser/conf
cp rui.crt /usr/lib/vmware-logbrowser/conf
cp rui.pfx /usr/lib/vmware-logbrowser/conf
cd /usr/lib/vmware-logbrowser/conf
chmod 400 rui.key rui.pfx
chmod 644 rui.crt
cd /etc/vmware-sso/register-hooks.d
./09-vmware-logbrowser --mode install --ls-server https://<em>server.domain.com</em>:7444/lookupservice/sdk --user <em>sso_administrator</em> --password <em>sso_administrator_password
service vmware-logbrowser stop
service vmware-logbrowser start

In this environment the AutoDeploy service is not started so I’m skipping this step. (refer to KB2057223 to complete this step)

You can now restart the vCenter Server Appliance and chek that the certificates have been successfully replaced.

 

Related documents
Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223)
Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 5.x (2062108)
Decoding a non-zero VC_CFG_RESULT for failed vpxd_servicecfg certificate changes (2057248)
Configuring certificates signed by a Certificate Authority (CA) for vCenter Server Appliance 5.1 (2036744)

%d bloggers like this: