How to make an open source Load balancer (Stunnel + HAProxy)

[A #classic_but_useful tutorial from #phpontrax_com]

Intoduction

Our current setup we have multiple application and static content servers serving our ecommerce shopping software Veracart. Instead of spending thousands of dollars on an expensive load balancer appliance we decide to go the open source route. After looking at many different solutions what we finally ended up going with for our load balancer machine is stunnel for the ssl decryption and haproxy for the actual load balancing. stunnel is need because haproxy doesn’t support ssl in http mode.

Load balancer hardware: 1U, dual Intel Xeon 2.80GHz cpu’s, 4gigs of ram, sata drives.
Software: freebsd (or linux), stunnel 4.32 with xforwarded-for patch, haproxy 1.4.8

Stunnel Installation

First off if you want your web servers in your clusters to be able to know the connecting clients real ip address instead of the load balancers then you need to patch stunnel with xforwarded-for patch.

wget ftp://stunnel.org/stunnel/archive/4.x/stunnel-4.32.tar.gz
wget http://www.haproxy.org/download/patches/stunnel-4.32-xforwarded-for.diff
tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install

Haproxy Installation

download: http://www.haproxy.org/download/1.4/src/haproxy-1.4.8.tar.gz
or

cd /usr/ports/net/haproxy #(from freebsd ports)
make && make install

Stunnel Configuration

Stunnel will receive all the https connections on port 443 and forward as http requests to haproxy on port 81 (or any port you want). From there haproxy will send the http request to the webserver cluster. Below the ip 1.2.3.4 represents your public ip that will goto your cluster. You can have as many [secure.domain.com] sections as you want or have domains with ssl certs for.

stunnel.conf

sslVersion = all
options = NO_SSLv2
setuid = root
setgid = stunnel
pid = /var/run/stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log
 
[secure.domain.com]
cert = /usr/local/openssl/certs/secure.domain.com.crt
key = /usr/local/openssl/private/secure.domain.com.key
accept  = 1.2.3.4:443
connect = 1.2.3.4:81
xforwardedfor = yes
TIMEOUTclose = 0

Haproxy Configuration

Haproxy will receive the decrypted https request, now just an http request, from stunnel on port 81. It will then load balance those requests across however many servers you have listed. I forward the request to port 81 on our apache webservers.

haproxy.conf

global
    daemon
    maxconn 10000
    nbproc 2
    log 127.0.0.1 syslog
 
defaults
    log global
    clitimeout 60000
    srvtimeout 30000
    contimeout 4000
    retries 3
    option redispatch
    option httpclose
    option abortonclose
 
listen domain_cluster_http 1.2.3.4:80
    mode http
    balance roundrobin
    cookie SERVERID insert nocache
    option forwardfor except 1.2.3.4
    reqadd X-Forwarded-Proto:\ http
    server server1 10.1.1.2:80 cookie s1 weight 1 maxconn 5000 check
    server server2 10.1.1.3:80 cookie s2 weight 1 maxconn 5000 check
 
listen domain_cluster_https 1.2.3.4:81
    mode http
    balance roundrobin
    cookie SERVERID insert nocache
    option forwardfor except 1.2.3.4
    reqadd X-Forwarded-Proto:\ https
    server server1 10.1.1.2:81 cookie s1 weight 1 maxconn 5000 check
    server server2 10.1.1.3:81 cookie s2 weight 1 maxconn 5000 check   
 
listen  lb1_stats [load balancer's public ip]:80
    mode http
    stats uri /
    stats auth username:password
    #stats refresh 5s

Apache Configuration

Lastly there is some things you will want to do to apache so it will know the real clients ip address. What I use is mod_rpaf, which changes the remote address of the client visible to other Apache modules from the X-Forwarded-For header. On freebsd there is a port for this.

cd /usr/ports/www/mod_rpaf2
make && make install

This will install the Apache2 version. After its installed open up your Apache’s http.conf file and add the following lines.

httpd.conf

# if DSO load module first:
LoadModule rpaf_module libexec/apache2/mod_rpaf-2.0.so
RPAFenable On
RPAFsethostname On
RPAFproxy_ips [load balancer's ip]
RPAFheader X-Forwarded-For

Next since we are forwarding https request to our webserver cluster nodes on port 81 we need to add the following line to http.conf

httpd.conf

Listen 81

Conclusion

We don’t have a huge amount of traffic about a steady 5Mbps, and the load balancer is always idling along with barely any load at all. This has worked out great for us. All the ssl decrypting is off loaded from the web servers and haproxy is super fast and has a great stats page to see what’s going on with your cluster’s.
[A #classic_but_useful tutorial from #phpontrax_com]