XtendWeb Cluster Installation

XtendWeb Cluster Introduction

Note

XtendWeb cluster does not replicate the httpd (LAMP Stack) configuration due to limitations imposed by cPanel itself. Although having a HA LAMP stack is one of the development goals, you will need to switch the domains to LEMP stack to get High Availability. XtendWeb makes switching a webapp to LEMP stack quite easy

Note

You will need XtendWeb license subscription for all the systems in a cluster. This means you will need a minimum of 2 XtendWeb license to deploy a cluster

XtendWeb cluster is the world’s easiest web application clustering solution featuring a fully automated deployment of a clustered DNS load balanced LEMP stack. It is specifically designed for multi-datacenter deployment and use encrypted communication between the server

XtendWeb cluster provides high availability, scalability and an inherent data backup (application files and database are replicated and normally reside on 2 servers at any time ) Coupled with the simplicity and intuitiveness of cPanel control panel , XtendWeb cluster is a must have for any modern enterprise web application deployment

What is clustered and High Available ?

1. Nginx+The application servers like php-fpm/Phusion passenger
2. MariaDB
3. SMTP (Incoming Email) - Multiple MX providing HA
4. SMTP (OutGoing Email) - All servers can send authenticated email

5. CronJobs - Cron job runs on one server at any time and starts running on the slave when master is shutdown
6. The file system - /home is clustered excluding the /home/$user/mail folder
7. DNS ( gdnsd or named )

What is not clustered and High Available?

1. HTTPD (cPanel httpd runs only on master cPanel node)
# Any domains set to "PROXY to httpd" will show a 5xx error code when master is down!

2. IMAP/POP
# /home/$user/mail reside only on master
# The secondary MX are configured as a store-and-forward backup MX
# note that an entry like below in /etc/postfix/recipient_bcc will auto_bcc emails to an alternate email while its being stored
# this  can be used as a method to access emails immediately when master server is down
# cat /etc/postfix/recipient_bcc
# [email protected] [email protected]

3. WHM, cPanel , WebMail
# These services run only on the master server

XtendWeb Cluster Requirements

1. Atleast 2 CentOS7 servers
# Only the master server need a valid cPanel LICENSE
# Slave servers need to be installed with cPanel DNS only which is a free product

2. MariaDB 10.2 or MariaDB 10.1
# Setup of MariaDB on the slaves are done automatically by the Ansible playbook run

3. Comparable Disk size for /home and /var as the data is synced
# The Ansible setup does not take into account additional home dirs like /home2,home3 etc
# Additonal home dirs need to be configured manually post setup

4. Addional TCP ports being open in the firewall (for example AWS Firewall security group)
# 4430,9999,13306,30865
#Note that these ports, only need to be open for inter-server communication
#They can be blocked for external traffic

5. More RAM for Unison file sync
# Unison is like a live backup program running 24x7
# It need at-least 1 GB of RAM  as extra for every 100GB of disk being synced

XtendWeb cluster setup

Note

Install CSF firewall on both servers and whitelist each others IP address (all ip’s) for access . The server’s hostname must be valid and should resolve correctly as many cluster components rely on hostname to connect

Note

Changing hostname once cluster is setup is chaotic. Choose a hostname and stick to it!

Preparing the servers

The Slaves

# Prepare any number of servers and install cPanel DNS only on it. The slave server is auto setup by the master
# So do nothing on it except install cPanel DNS only
cd /home && curl -o latest-dnsonly -L https://securedownloads.cpanel.net/latest-dnsonly && sh latest-dnsonly

The Master

#Install cPanel
cd /home && curl -o latest -L https://securedownloads.cpanel.net/latest && sh latest
ssh-keygen
ssh-copy-id root@slaves-fqdn
Select MariaDB 10.2

Ensure MariaDB 10.2 or 10.1 is selected at Home »Software »MySQL/MariaDB Upgrade and the upgrade is performed

Single quoted password in /root/.my.cnf

Ensure mysql password in /root/.my.cnf is bound by a single quote rather than no quotes or double quotes

setup XtendWeb cluster on master server

yum -y install epel-release
yum -y install https://github.com/AnoopAlias/XtendWeb/raw/ndeploy4/nDeploy-release-centos-1.0-7.noarch.rpm

yum -y --enablerepo=ndeploy install nginx-nDeploy nDeploy # For nginx as webserver
yum -y --enablerepo=ndeploy install openresty-nDeploy nDeploy # For openresty as webserver

/opt/nDeploy/scripts/cpanel-nDeploy-setup.sh enable

yum -y install python-pip libffi-devel python-paramiko python-jinja2
pip install ansible

cd /opt/nDeploy/conf/nDeploy-cluster
cp -p hosts.sample hosts
hosts file

Edit /opt/nDeploy/conf/nDeploy-cluster/hosts .The variables are self explanatory The example provided above is for a 2 server cluster setup

hosts file variables

webserver=nginx or webserver=openresty

mainip=
#The servers WAN or public IP. Do not use LAN ip here if you are on a NAT setup

dbip=
# The IP address MariaDB replication will use
#If your servers can talk over a LAN , it is best to use the LAN ip here

dbmode=rwsplit or dbmode=readonly
# rwsplit should work for most setups

dns=geodns or dns=named
# geodns will use gdnsd as server with geoLB
# named will use cPanel named with simple Round Robin LB
# If unsure use geodns as this is recommended for XtendWeb 4.6.x

repo=ndeploy or repo=ndeploy-edge
# Edge has newer release often in testing stage
# If in doubt use repo=ndeploy

latitude=
# Obtain latitude for servers WAN ip from https://www.maxmind.com/en/geoip-demo

longitude=
# Obtain longitude for servers WAN ip from https://www.maxmind.com/en/geoip-demo

# Enclose latitude and longitude in quotes as they may have negative values

Setup group_vars

# Setup homedir variable used to describe the names of home directories
# The example below assume you have 2 additional homedir /home2 and /home3
# cat /opt/nDeploy/conf/nDeploy-cluster/group_vars/all
---
homedir:
 - home
 - home2
 - home3

Run Ansible playbook

# It is recommended that you run the command below in screen as it may take time to complete
ansible-playbook -i ./hosts cluster.yml

# Once the Ansible play completes.The cluster is fully setup

Add Additonal IP mapping if required

The playbook will only setup the cluster to use with the mainip. If you have more IP address and wish to use them, you will need to setup the mapping The maps are setup in 2 yaml files

ipmap

The dnsmap section maps the masters IP address(LAN IP if NAT is used) to the WAN IP of the slave. the ipmap section maps the masters IP address(LAN IP if NAT is used) to the slaves IP (LAN IP if NAT is used). The dnsmap and ipmap will be same on a non-NAT machine, but will be different on NAT-ed machines In both case the key (Left hand side) will be the IP address that show up in ifconfig command on master

dnsmap

The key ( Left Hand side) of dnsmap is the masters IP address (LAN IP if NAT is used) and the Right hand side is the dns resource name that is to be configured in gdnsd, for example: ip1

Configure gdnsd for additional IP address

The resource name provided in the /opt/nDeploy/conf/ndeploy_master.yaml file should be used to configure both files below. The IP address used are the WAN IP’s (DNS IP’s)

The names used in dcmap can be obtained from /etc/gdnsd/config

[root@li931-182 ~]# cat /etc/gdnsd/geoip_resources
    ip1 => {
      map => auto_map,
      service_types => [ xtendweb ],
      dcmap => {
        li931-182 => 45.56.113.183,
        li1481-142 => 139.162.164.143,
                }
    },
[[email protected] ~]# cat /etc/gdnsd/metafo_resources
    ip1 => {
      datacenters => [ real ],
      dcmap => {
        real => %geoip!ip1
      }
    },

Ensure gdnsd checkconf does not throw an error

[root@li931-182 ~]# gdnsd checkconf
# info: Loading configuration from '/etc/gdnsd/config'
# info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:53
# info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:53
# info: plugin_geoip: map 'auto_map': Loading GeoIP2 database '/etc/gdnsd/geoip/GeoLite2-City.mmdb': Version: 2.0, Type: GeoLite2-City, IPVersion: 6, Timestamp: 2018-08-07 23:25:47 UTC
# info: plugin_geoip: map 'auto_map' runtime db updated. nets: 277591 dclists: 2
# info: Loading zone data...
# info: rfc1035: will use inotify for zone change detection
# warning: Zone 'lemurstack.com.': SOA Master does not match any NS records for this zone
# info: Zone lemurstack.com.: source rfc1035:lemurstack.com with serial 1 loaded as authoritative
# info: rfc1035: Loaded 1 zonefiles from '/etc/gdnsd/zones/'
# info: rfc1035: quiescence time is 3 seconds
# info: Configuration and zone data loads just fine
csync2 -xv
[root@li931-182 ~]# systemctl restart gdnsd

cPanel Horizontal scaling . Adding more web servers

You can add more servers to the cluster anytime. Adding more servers involve adding the appropriate hostnames and other variables in /opt/nDeploy/conf/nDeploy-cluster/hosts and re-running the Ansible playbook

Remove MySQL dump lock so that MySQL replication can be setup on the new slaves

rm -f /opt/nDeploy/conf/XTENDWEB_MYSQL_DUMP_LOCK_DO_NOT_REMOVE

The one extra thing you need to do if you are adding additional servers or if you are setting up XtendWeb cluster on an existing cPanel server with accounts and database’s already present is apply the necessary grants for the slave servers

mysqlaccess

Grants can be setup from Home »SQL Services »Additional MySQL Access Hosts section in WHM. Click on the “click here” link to setup additional grants required for the slaves

Testing domain on cluster setup

Since the web application runs on multiple servers that are independant of each other there is a chance that one of the server is not serving page correctly and this goes unnoticed because of round robin DNS

To test individual servers do

curl -v -I https://domain.com --resolve "domain.com:443:xxx.xxx.xxx.xxx"
# Where xxx.xxx.xxx.xxx is the master or slave servers IP that you need to test

Cluster Administration scripts

/opt/nDeploy/scripts/cluster_home_ensure_all.py
# Ensures the users and shell is in sync

/opt/nDeploy/scripts/cluster_dns_ensure_all.sh
#Ensure DNS changes are uptodate

/opt/nDeploy/scripts/cluster_sync_rpms.sh
#Ensure the entire LEMP stack is in sync
# Note that this does not include manually installed pecl modules etc

/opt/nDeploy/scripts/fix_unison_filesync.py restart|reset
# Reset starts unison cluster from scratch,so use it rarely if
# unison is not starting up
# restart as the name implies restarts Unison