Building a higly available load balancing solution with HAProxy

When you start scaling your environment you will most likely need a load balancer but then again your load balancer will be your single point of failure which is one of the things you always want to avoid.

How do we go around that? Simply by scaling your load balancing solution as well. Most of the times in a production environment you will see load balancers in couples for redundancy. This is possible even with HAProxy using a software called keepalived.

Keepalived is not a tool specific for HAProxy but it does the job for us, since it will make it possible to share an IP address between our 2 load balancers. It does this using VRRP and you will get ownership of the IP address based on your keepalived configuration so you will end up with an active/passive architecture.

If you took the time to read the article i linked in the previous HAProxy post by Luca Dell’Oca you will know already how to build this.

First install keepalived and edit the config file:

yum install keepalived
vi /etc/keepalived/keepalived.conf

This is my config file, which you’ll notice is pretty much the same as Luca’s:

global_defs {
   notification_email {
     failover@myvirtualife.net
     sysadmin@myvirtualife.net
   }
   notification_email_from loadbalancer@myvirtualife.net
   smtp_server 192.168.100.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script chk_haproxy {
   script "killall -0 haproxy"
   interval 1                     # check every second
   Weight 2                       # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 12345678
    }

virtual_ipaddress {
    172.16.110.5
}

track_script {
33	chk_haproxy
}
}

After configuring keepalived let’s make a few more changes and then let’s check if the shared IP is active:

net.ipv4.ip_nonlocal_bind = 1
sysctl -p
service keepalived start
chkconfig keepalived on
ip addr sh eth0

keepalived

172.16.110.2 is the ip address of this load balancer.
172.16.110.5 is the shared ip address managed by keepelived.

Now you have to set up another HAProxy VM and configure it in the same way, just remember in the keepalived config file that ‘priority’ must be set to ‘100’.

To test if it works just hard power down the VM that holds the shared IP and test if communication still works.

You obviously also have to install and configure HAProxy on both VMs and remember to keep the two configurations aligned of you make any changes.

Most of the time i disable iptables but Luca does a better job than me and shows you how to configure iptables to happily get along with both keepalived and HAProxy, so if you intend to leave iptables on go check his post too.

Balancing multiple Horizon Workspace gateway-va with HAProxy

When working with Horizon Workspace the first component you will scale to multiple instances is probably the gateway-va since this is the access point of all users, just to make sure it’s always available for connections.

In this case you need a load balancer to direct all users to all the gateway-va you have in your environment; i wrote about commercial and open source load balancers and also how to build one with HAProxy in this post.

I’m going to show you how i configure it with Horizon Workspace but remember that since I’ve learned about HAProxy only relatively recently by Luca Dell’Oca my configuration is just the way i do it and not necessarily the best so use the comments if you want to contribute.

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------

global
log 127.0.0.1 local2 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
option accept-invalid-http-request
retries 3
timeout http-request 60s
timeout queue 30m
timeout connect 1800s
timeout client 30m
timeout server 30m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats :9000
stats realm Haproxy\ Statistics
stats uri /stats

#---------------------------------------------------------------------
# Redirect to secured
#---------------------------------------------------------------------
frontend unsecured
bind :80
redirect scheme https if !{ ssl_fc }

#---------------------------------------------------------------------
# frontend secured
#---------------------------------------------------------------------
frontend front
bind :443 ssl crt /etc/haproxy/reverseproxy.pem
mode http

acl workspace hdr_beg(host) -i workspace.myvirtualife.net
use_backend workspace if workspace

#---------------------------------------------------------------------
# balancing between the various backends
#---------------------------------------------------------------------
backend workspace
mode http
server workspace1 192.168.110.10:443 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl
server workspace2 192.168.110.11:443 weight 1 check port 443 inter 2000 rise 2 fall 5 ssl

Try to add a gateway-va and experiment with HAProxy to test HAProxy as load balancer. You can use this article if you want to know how to do it.

There are few more things worth of noting:

  • timeouts are really long here otherwise users will experience disconnects because this is the kind of web app you keep open quite a lot;
  • on port 9000 on the HAProxy host you will find statistics, for example “lb.yourcompany.yourdomain:9000/stats”, that will give numbers about state of connections and state of backends, problems, etc…
  • “log 127.0.0.1 local2 info” is necessary if you want logging enabled which is so important when troubleshooting problems; a lot on how to read logs in the HAProxy documentation

if you intend to put a SSL cert like in my configuration, know that it has to be a chain of cert and private key like this:

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

To make logging work and write to a separate file instead of putting everything in “/var/log/messages”, edit your “/etc/rsyslog.conf” file and make sure these lines are present:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# HAProxy
local2.* /var/log/haproxy.log

Using a Microsoft CA to generate certificates for Horizon Workspace

During installation of Horizon Workspace in the last post we used self-signed certificates for simplicity but when you will put Workspace in production you will definitely want to replace those certificates.

In this post we will use an internal Microsoft CA to request certificates for our Horizon Workspace implementation.

Note: The installation of a Microsoft CA is outside the scope of this article.

If you connect to your Horizon Workspace FQDN you will see the classic browser warning when you connect to an SSL website which certificate has been released by a Certificate Authority you don’t trust.

In fact if you take a close look at the certificate you will easily notice the following:

1

You can see how we don’t trust the CA as it is stated in red and as you can see from the certificate tree at the top.

We need to create a certificate request to pass to our Microsoft CA so that it can process it and spit out a certificate for us. There are several tools to create certificate requests but i like to use OpenSSL because it is available on almost every operating system so if you learn how to do it from that you will be able to do it in most situations.

The steps i am going to take will work on every platform, regardless the fact that i will do this on a Mac you will be able to take the same steps on a Windows box. You can find OpenSSL binaries for Windows here.

On a Mac, open a Terminal window, move to the “/bin” directory of your OpenSSL installation and run the following commands:

sudo openssl genrsa -out key.pem 2048
sudo openssl req -out horizon.csr -key key.pem -new


After running the second command we will be presented with a few questions to compile in order to create a certificate request:

Country Name (2 letter code) [AU]:IT
State or Province Name (full name) [Some-State]:Lazio
Locality Name (eg, city) []:Roma
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyVirtuaLife.Net
Organizational Unit Name (eg, section) []:IT Department
Common Name (e.g. server FQDN or YOUR name) []:workspace.myvirtualife.net
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:


The first command will generate a private key (key.pem) that we will use for our request, the second command will actually create a request file signed it with the private key we just created.

The request file (horizon.csr) can be opened as a text file and it should look like this:

-----BEGIN CERTIFICATE REQUEST-----
MIICuzCCAaMCAQAwdjELMAkGA1UEBhMCSVQxFzAVBgNVBAgTDkVtaWxpYSBSb21h
Z25hMRYwFAYDVQQHEw1SZWdnaW8gRW1pbGlhMRUwEwYDVQQKEwxNeVZpcnR1YUxp
ZmUxEzARBgNVBAsTCk5ldHdvcmtpbmcxCjAIBgNVBAMUASowggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQC6XP30PjkBqv7f1K9KKDm5gqdJLh8l9E8S28TQ
pEHzWcoIAnMezkFmGZUpIiQch53dABEAzJoNjQT5rGl8+Dfzghn6gK/0KreiqT7H
Yz+YQqiYofQ9u6yUOPo6I/ZsbagWUnTQ1E4n30IizY/UoVpt592dyy2+qay7xAZU
Yu0LG71dsG1TfclflgBh1PjRSGyrSaYTrm0ZlV2V5r74SZlLHtlGISxRG9Khsdmg
Fnkdybxr2CcM/lvwMN63rZXaYx9n9zr8/4qbOVNVo6o5PN9f+rllss1CKddr8Y03
n+IK9c1Y5H2BnK0RJBJELT2fRmulbVpV60Uo4u+bEFe/9oSXAgMBAAGgADANBgkq
hkiG9w0BAQUFAAOCAQEAQIw6kkW3VEY1gRUYqq/m2FxmpKlfJQzDxoe0DjSW7lJz
QcJlc25rPRi9ZsLe9/+1RcpAwJPMBii6ZyBIqtp2dwqpgSjn4bPa1nOE1YDnwlOe
NrX5wSe9JOnGL1FXhfyuALd+dVFRHhX/aAQU//klcfC8QHIVPtD78EffHVciENza
ckVl1L86CWAg2fWnZmvku9DYEbHNS1MhpJLgXBM3Yf7+kFt+PorO2AxF2SL82PBE
DVzdRT8nFeP9heaU0Jia0ByVKS873KoAuQFbm9cf++uNdCbZE02RRDTzYcREqD0Y
zNVsrfGVFJj8xRNJ1qgGu4wz5k9SnHnevMzq8DB9FQ==
-----END CERTIFICATE REQUEST-----

This will be correctly interpreted by the CA as a valid request but we won’t be able to read it. If you want to check if everything is ok you can do it like this:

openssl req -text -noout -in horizon.csr


You will see plenty of info and among that you will find those you inserted in the request:

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: C=IT, ST=Lazio, L=Roma, O=MyVirtuaLife.Net, OU=IT Department, CN=workspace.myvirtualife.net

Note: The common name value is what your browser checks to be the same of the website you are trying to access, if different it will throw an error.

To pass the request to a Microsoft CA just access the web portal of your CA and click “Request a certificate” -> “advanced certificate request” and then paste your request as follows:

2

Select “Web Server” then click “Submit” and download the Base 64 encoded certificate:

3

You should get a file called “certnew.cer” that i normally rename in “horizon.pem”.

You should also get the CA certificate file, to download it go back to the homepage of your CA and click on “Download a CA certificate, certificate chain, or CRL”, the you should be here:

4

Select “Base 64” and then “Download CA certificate”.
Whenever you download a certificate from a Microsoft CA it will be called “certnew.cer” so you can see why it’s a best practice to rename them, i usually call this “ca.pem”.

At this point we should have the following:

  • key.pem (private key)
  • horizon.pem (the horizon workspace certificate)
  • ca.pem (the certification authority certificate)

Clarifying the certificate formats chaos
Every guide you will find out there that instructs you how to generate certificates will most of the time do a bad job explaining the various kind of formats, the difference between them and when to use one kind or another. Since i don’t want to take credit for something i didn’t do i want you to know that the following is taken from this webpage where you will also be able to convert different types of certificates if you need.

When you are dealing with certificates you will find different formats such as pem, der, p7b, and pfx. A Windows server for example exports and imports .pfx files while an Apache server uses individual PEM (.crt, .cer) files. The following is a definition of the various formats i mentioned.

PEM Format:
The PEM format is the most common format that Certificate Authorities issue certificates in. PEM certificates usually have extensions such as .pem, .crt, .cer, and .key. They are Base64 encoded ASCII files and contain “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–” statements. Server certificates, intermediate certificates, and private keys can all be put into the PEM format.

Apache and other similar servers use PEM format certificates. Several PEM certificates, and even the private key, can be included in one file, one below the other, but most platforms, such as Apache, expect the certificates and private key to be in separate files.

DER Format:
The DER format is simply a binary form of a certificate instead of the ASCII PEM format. It sometimes has a file extension of .der but it often has a file extension of .cer so the only way to tell the difference between a DER .cer file and a PEM .cer file is to open it in a text editor and look for the BEGIN/END statements. All types of certificates and private keys can be encoded in DER format. DER is typically used with Java platforms.

PKCS#7/P7B Format:
The PKCS#7 or P7B format is usually stored in Base64 ASCII format and has a file extension of .p7b or .p7c. P7B certificates contain “—–BEGIN PKCS7—–” and “—–END PKCS7—–” statements. A P7B file only contains certificates and chain certificates, not the private key. Several platforms support P7B files including Microsoft Windows and Java Tomcat.

PKCS#12/PFX Format:
The PKCS#12 or PFX format is a binary format for storing the server certificate, any intermediate certificates, and the private key in one encryptable file. PFX files usually have extensions such as .pfx and .p12. PFX files are typically used on Windows machines to import and export certificates and private keys.

Understanding Horizon Workspace components and installation prerequisites

In the last post i described in details how to prepare a vPostgres DB to host Horizon Workspace external database.

During the installation process, as we will see, you can choose to use an internal database or an external one but keep in mind that the internal database is ment only for testing purpose so if you are installing Horizon Workspace in a production environment you must have a VM with vPostgres installed as this is the only supported configuration, so you can understand why the first post was needed.

So now we are ready to install Horizon Workspace… well, not quite yet. It is very important to understand that to install this product there are number of preparation steps that need to be taken before actually getting our hands dirty and start having fun. Some of those steps include filling up some technical prerequisites and some are just decisions that need to be taken keeping in mind that during the installation phase there are some settings that cannot be changed afterwards unless redeploying the entire solution. This is something you definitely don’t want to find out after you’ve performed all the installation and configuration tasks and then have to start over again.

In this post we are going through all the prerequisites so with that out of the way we will be able to easily proceed with the deployment phase, but first let’s talk about the Horizon Workspace virtual appliances and their respective functions. The following is taken from the official documentation.

  • VMware Horizon Workspace Configurator Virtual Appliance (configurator-va): You start configuring Horizon Workspace with this virtual appliance, using both the Configurator virtual appliance interface and the Configurator Web interface. The configurations you make with the Configurator are distributed to the other virtual appliances in the vApp. Note: The configurator-va is the only component that cannot scale to multiple instances.
  • VMware Horizon Workspace Manager Virtual Appliance (service-va): Horizon Workspace Manager handles ThinApp package synchronization and gives you access to the Administrator Web interface, from which you can manage users, groups, and resources.
  • VMware Horizon Workspace Connector Virtual Appliance (connector-va): Horizon Workspace Connector provides the following services: user authentication (identity provider), directory synchronization, ThinApp-catalog loading, and View pool synchronization.
  • VMware Horizon Workspace Data Virtual Appliance (data-va): Horizon Workspace Data Virtual Appliance controls the file storage and sharing service, stores users’ data (files), and synchronizes users’ data across multiple devices.
  • VMware Horizon Workspace Gateway Virtual Appliance (gateway-va): Horizon Workspace Gateway Virtual Appliance is the single endpoint for all end-user communication. User requests come to the gateway-va virtual machine, which then routes the request to the appropriate virtual appliance.

System and Network Configuration Requirements
The preparation part is the longest and most important when deploying a distributed service such as Horizon Workspace, for this reason VMware prepared a detailed checklist to fill up before starting the installation process. The following is a list of all the things you will have to decide and mark down:

  • Horizon Workspace Fully Qualified Domain Name (FQDN)
  • Network Information for Configurator (configurator-va)
  • Network Information for Manager (service-va)
  • Network Information for Connector (connector-va)
  • Network Information for Data (data-va)
  • Network Information for Gateway (gateway-va)
  • Network Information for IP Pools
  • Active Directory Domain Controller
  • SMTP Server
  • vCenter Credentials
  • SSL Certificate (Optional)
  • Horizon Workspace License Key
  • Microsoft Windows Preview
  • External Database

Before getting into details let’s take a high level look at the architecture of Horizon Workspace as it’s meant to be in a production environment:

Image

This picture (which is taken straight from the public documentation of the product) shows that every connection from users accessing the Horizon Workspace portal have to go through the Horizon gateway VM(s). The “(s)” easily shows how you can have one or multiple Horizon gateways, in which case you will also need some sort of load balancing mechanism in front of the gateways. The Horizon gateway virtual appliance runs nginx as web server that basically proxies every connection to the desired service so users actually need connectivity only to the gateways virtual appliances.

IMPORTANT: Placing the gateway VA in a separate network such as a DMZ network is not a supported configuration.

The following picture gives a better understanding of the network configuration requirements:

Image

As you can see all communication go into the gateway VA and out to the other virtual appliances which are actually providing the services. Users will connect exclusively in HTTPS and the same is true also for most of communication between virtual appliances, so we will need to work a bit on SSL certificates at some point but it’s not mandatory in the setup phase as you can see form the above list since it is marked as optional in the prereqs.

Horizon Workspace FQDN
Choosing the FQDN is a tricky one because once you input it during the setup you can’t go back and change it, so it definitely deserves some thinking or you might find yourself redeploying from scratch. Most companies choose to have the same FQDN for both internal and external connections which makes it perfectly transparent for users to reach Workspace no matter where they are located; obviously the FQDN will resolve with a public IP for external users and with a private IP for internal users, hence the need of two sets of load balancers as you can see in the first picture.

Network configuration for virtual appliances
Just write down TCP/IP configurations that you intend to assign to the five virtual appliances, including DNS configuration. I encourage you to use consecutive addresses for simplicity.

IP Pools
Honestly this is a little obscure to me. IP Pools are used as a set of IP addresses that you define and assign to a network in vCenter so that they can be used when you deploy a vApp. Funny is the fact that those addresses must not be the ones you will use for setting up the virtual appliances. Even funnier is the fact that if you deploy the vApp from the Web Client you don’t even have to create an IP Pool. I have no problems admitting my ignorance here on the usefulness and meaning of this step.

Active Directory Domain Controller
Self explaining. Since Horizon Workspace integrates with your Active Directory you will need to have IP address, basic parameters and credentials handy during the setup. Just keep in mind that your users in AD will need to have Name, Last Name and email address compiled before importing them in Horizon Workspace or the import will fail.

SMTP Server
This is used by users when sharing documents. Note that you must specify a working SMTP since a check is performed during the setup and you won’t be able to proceed otherwise.

vCenter credentials
If you are deploying Horizon Workspace I’m pretty sure you have these. 🙂

SSL Certificate (optional)
I like to deal with this after the initial deployment and this is another tricky one, so during the setup we will use default self-signed certificates for simplicity.

Horizon Workspace Product Key
Yes, you need one. 🙂
For a proof-of-concept you can request a trial key that will work for 100 users.

Microsoft Windows Preview
When using Microsoft documents in Horizon Workspace web portal you can get a preview without having Microsoft Office installed. The preview can be generated with a LibreOffice add-on that runs directly on the data-va or they can be generated on a Microsoft Server with Microsoft Office installed; the first is a free option and it’s usually good enough, the latest will grant you a higher level of compatibility but you will have to pay Microsoft licenses.

External Database
If you read my last post you should know about this already.

Now that you have all handy you are ready to install Horizon Workspace.

%d bloggers like this: