AUTOM8N Cluster Installation

AUTOM8N Cluster Requirements: cPanel v72 or higher server with CentOS7/CloudLinux7 64 bit OS

AUTOM8N Cluster is compatible with CloudLinux LVE, CageFS, cPanel MultiPHP, AutoSSL, FleetSSL ,LSPHP(Apache Only)

AUTOM8N Cluster is NOT compatible with CloudLinux PHP Selector and Alt-PHP Use MultiPHP instead

Note

Starting from AUTOM8N version 4.7.x, if master server is CloudLinux, All slaves must be CloudLinux too

Note

setup lsphp(mod_lsapi) on Apache to be used in AUTOM8N Cluster https://autom8n.com/howtos/autom8n_cluster_on_cloudlinux.html

AUTOM8N cluster provide DNS LoadBalancing, High Availability, Multi Server Scalability

Additionally, it provide a queue-and-forward backup mail server

AUTOM8N Cluster Requirements

1. Atleast 2 CentOS7 servers
# Only the master server need a valid cPanel LICENSE
# Slave servers need to be installed with cPanel DNS only which is a free product

2. MariaDB 10.2 or MariaDB 10.1
# Setup of MariaDB on the slaves are done automatically by the Ansible playbook run

3. Comparable Disk size for /home and /var as the data is synced

4. Addional TCP ports being open in the firewall (for example AWS Firewall security group)
# 4430,9999,13306,30865
#Note that these ports, only need to be open for inter-server communication
#They can be blocked for external traffic

5. More RAM for Unison file sync
# Unison is like a live backup program running 24x7
# It need at-least 1 GB of RAM  as extra for every 100GB of disk being synced

AUTOM8N cluster setup

Note

Install CSF firewall on both servers and whitelist each others IP address (all ip’s) for access . The server’s hostname must be valid and should resolve correctly as many cluster components rely on hostname to connect

Note

Changing hostname once cluster is setup is chaotic. Choose a hostname and stick to it!

Preparing the servers

The Slaves

# Prepare any number of servers and install cPanel DNS only on it. The slave server is auto setup by the master
# So do nothing on it except install cPanel DNS only
cd /home && curl -o latest-dnsonly -L https://securedownloads.cpanel.net/latest-dnsonly && sh latest-dnsonly

The Master

#Install cPanel
cd /home && curl -o latest -L https://securedownloads.cpanel.net/latest && sh latest
ssh-keygen
ssh-copy-id root@slaves-fqdn
Select MariaDB 10.2

Ensure MariaDB 10.2 or 10.1 is selected at Home »Software »MySQL/MariaDB Upgrade and the upgrade is performed

setup AUTOM8N on master server

yum -y install epel-release
yum -y install https://github.com/AnoopAlias/AUTOM8N/raw/ndeploy4/nDeploy-release-centos-1.0-7.noarch.rpm

yum -y --enablerepo=ndeploy install nginx-nDeploy nDeploy # For nginx as webserver
yum -y --enablerepo=ndeploy install openresty-nDeploy nDeploy # For openresty as webserver

/opt/nDeploy/scripts/cpanel-nDeploy-setup.sh enable

cd /opt/nDeploy/conf/nDeploy-cluster
cp -p hosts.sample hosts
hosts file

Edit /opt/nDeploy/conf/nDeploy-cluster/hosts .The variables are self explanatory The example provided above is for a 2 server cluster setup

hosts file variables

webserver=nginx or webserver=openresty

mainip=
#The servers WAN or public IP. Do not use LAN ip here if you are on a NAT setup

dbip=
# The IP address MariaDB replication will use
#If your servers can talk over a LAN , it is best to use the LAN ip here

dbmode=readconnroute
# readconnroute is recommended, the other option is rwsplit

dns=geodns or dns=named
# geodns will use gdnsd as server with geoLB
# named will use cPanel named with simple Round Robin LB
# If unsure use geodns as this is recommended for AUTOM8N 4.6.x

repo=ndeploy or repo=ndeploy-edge
# Edge has newer release often in testing stage
# If in doubt use repo=ndeploy

latitude=
# Obtain latitude for servers WAN ip from https://www.maxmind.com/en/geoip-demo

longitude=
# Obtain longitude for servers WAN ip from https://www.maxmind.com/en/geoip-demo

# Enclose latitude and longitude in quotes as they may have negative values

Setup group_vars

mkdir /opt/nDeploy/conf/nDeploy-cluster/group_vars
# Setup homedir variable used to describe the names of home directories
# The example below assume you have 2 additional homedir /home2 and /home3
# cat /opt/nDeploy/conf/nDeploy-cluster/group_vars/all
---
homedir:
 - home
 - home2
 - home3

Run Ansible playbook

# It is recommended that you run the command below in screen as it may take time to complete
ansible-playbook -i ./hosts cluster.yml

# Once the Ansible play completes.The cluster is fully setup

Note

Following section is only required if you have more than one IP address assigned to the server

Add Additonal IP mapping if required

The playbook will only setup the cluster to use with the mainip. If you have more IP address and wish to use them, you will need to setup the mapping The maps are setup in 2 yaml files

ipmap

The dnsmap section maps the masters IP address(LAN IP if NAT is used) to the WAN IP of the slave. the ipmap section maps the masters IP address(LAN IP if NAT is used) to the slaves IP (LAN IP if NAT is used). The dnsmap and ipmap will be same on a non-NAT machine, but will be different on NAT-ed machines In both case the key (Left hand side) will be the IP address that show up in ifconfig command on master

dnsmap

The key ( Left Hand side) of dnsmap is the masters IP address (LAN IP if NAT is used) and the Right hand side is the dns resource name that is to be configured in gdnsd, for example: ip1

Configure gdnsd for additional IP address

The resource name provided in the /opt/nDeploy/conf/ndeploy_master.yaml file should be used to configure both files below. The IP address used are the WAN IP’s (DNS IP’s)

The names used in dcmap can be obtained from /etc/gdnsd/config

[root@li931-182 ~]# cat /etc/gdnsd/geoip_resources
    ip1 => {
      map => auto_map,
      service_types => [ xtendweb ],
      dcmap => {
        li931-182 => 45.56.113.183,
        li1481-142 => 139.162.164.143,
                }
    },
[[email protected] ~]# cat /etc/gdnsd/metafo_resources
    ip1 => {
      datacenters => [ real ],
      dcmap => {
        real => %geoip!ip1
      }
    },

Ensure gdnsd checkconf does not throw an error

[root@li931-182 ~]# gdnsd checkconf
# info: Loading configuration from '/etc/gdnsd/config'
# info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:53
# info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:53
# info: plugin_geoip: map 'auto_map': Loading GeoIP2 database '/etc/gdnsd/geoip/GeoLite2-City.mmdb': Version: 2.0, Type: GeoLite2-City, IPVersion: 6, Timestamp: 2018-08-07 23:25:47 UTC
# info: plugin_geoip: map 'auto_map' runtime db updated. nets: 277591 dclists: 2
# info: Loading zone data...
# info: rfc1035: will use inotify for zone change detection
# warning: Zone 'lemurstack.com.': SOA Master does not match any NS records for this zone
# info: Zone lemurstack.com.: source rfc1035:lemurstack.com with serial 1 loaded as authoritative
# info: rfc1035: Loaded 1 zonefiles from '/etc/gdnsd/zones/'
# info: rfc1035: quiescence time is 3 seconds
# info: Configuration and zone data loads just fine
csync2 -xv
[root@li931-182 ~]# systemctl restart gdnsd

cPanel Horizontal scaling . Adding more web servers

You can add more servers to the cluster anytime. Adding more servers involve adding the appropriate hostnames and other variables in /opt/nDeploy/conf/nDeploy-cluster/hosts and re-running the Ansible playbook

Remove MySQL dump lock so that MySQL replication can be setup on the new slaves

rm -f /opt/nDeploy/conf/XTENDWEB_MYSQL_DUMP_LOCK_DO_NOT_REMOVE

The one extra thing you need to do if you are adding additional servers or if you are setting up AUTOM8N cluster on an existing cPanel server with accounts and database’s already present is apply the necessary grants for the slave servers

mysqlaccess

Grants can be setup from Home »SQL Services »Additional MySQL Access Hosts section in WHM. Click on the “click here” link to setup additional grants required for the slaves

Testing domain on cluster setup

Since the web application runs on multiple servers that are independent of each other there is a chance that one of the server is not serving page correctly and this goes unnoticed because of round robin DNS

To test individual servers do

curl -v -I https://domain.com --resolve "domain.com:443:xxx.xxx.xxx.xxx"
# Where xxx.xxx.xxx.xxx is the master or slave servers IP that you need to test

Cluster Administration scripts

/opt/nDeploy/scripts/cluster_home_ensure_all.py
# Ensures the users and shell is in sync

/opt/nDeploy/scripts/cluster_dns_ensure_all.sh
#Ensure DNS changes are uptodate

/opt/nDeploy/scripts/cluster_sync_rpms.sh
#Ensure the entire LEMP stack is in sync
# Note that this does not include manually installed pecl modules etc

/opt/nDeploy/scripts/fix_unison_filesync.py restart|reset
# Reset starts unison cluster from scratch,so use it rarely if
# unison is not starting up
# restart as the name implies restarts Unison