High Availability Installations on Ubuntu

This article is for guidance on deploying the PortSIP PBX HA on Ubuntu.

Prerequisites

Servers

In the examples used in this article, a total of three servers are used. PBX nodes are used to refer to servers in the following

• The three PBX nodes or all PBX nodes, refer to all PBX servers

• Master node and node 1 refer to pbx01

• Node 2 refers to pbx02, and node 3 refers to pbx03

Disk

Deploying the HA requires having two disk volumes. In this example, the hard disk of each server is 80 GB, creating two volumes:

• One is 40G (/dev/sda) for Linux OS we called it "system volume". This volume is used to install the Linux system, it's can be smaller, in your production environment, you can allocate it such as 100G.

• The other one is 40G (/dev/sdb) for storing the PBX data; we called it "data volume". This volume usually should be large since it is used to store the PBX data such as DB, recording files, logs, and chat files, in your production environment, we suggest it be as big as possible, such as 500G, or 1000G.

You are free to decide the disk size, but the size must be the same for all servers.

For the three node servers, they must have the same disk volumes, and the size must be the same!

The /dev/sdb disk volume must remain uninitialized and in an unused state, don't initialize it; otherwise, the installation will fail.

Network

In this article, we deploy the HA on three servers that have the following IP addresses:

• 192.168.1.131

• 192.168.1.132

• 192.168.1.133

We used 192.168.1.130 as the VIP (Virtual IP).

Linux OS

The PortSIP PBX HA for Ubuntu supports the following versions:

• Ubuntu 20.04

Architecture

Figure 1-1 PortSIP PBX HA Architecture

Pacemaker

The Pacemaker is a high-availability Cluster Resource Manager (CRM) that can be used to manage resources and ensure that they remain available in the event of a node failure.

The PortSIP PBX HA uses the Pacemaker to do the resource management and monitoring, once the event of PBX node failure, the resources will automatically move to a working node in the cluster.

DRBD

DRBD is used for High Availability purposes. It is a software product used to replicate data in real time from one server to another. This ensures business continuity even in the event of hardware failure.

The DRBD is utilized in the PortSIP HA scenario to synchronize data (DB, recording files, log files, and prompt files) between the PBX nodes.

To connect to the PBX service, all SIP clients (IP Phone, Softphone, Mobile App, WebRTC Client) will access the Virtual IP of PortSIP PBX in the HA scenario.

Working Mechanisms

The PortSIP PBX HA working mechanisms are as follows:

• Three PBX nodes are running at the same time to prevent split-brain during HA switching.

• Use DRBD to synchronize data between nodes, including current active call information, recording files, logs, call records, and prompt voice files uploaded by users.
• Use virtual IP (VIP) as the access entry point of the system.

• Use the Pacemaker to monitor the status between servers.
• The primary node provides services. Once the primary node has a DOWN machine, the Pacemaker will immediately drift the VIP to the standby node that takes over after detecting it. User service requests will be automatically routed to the standby node, and the standby node will continue to provide services.
• The established call on the primary node will be automatically restored by the standby node.

Notes

• Ensure that your hardware has at least 4 cores and 4GB of memory

• Unless otherwise specified, perform the operations mentioned below only on the master node, which is named pbx01

• During the Linux OS installation, you will need to create a user named “pbx”. All Linux commands during the HA configuration process must be performed by that “pbx” user.

• During the configuration of the HA, you may be prompted to enter the password for the user "pbx" multiple times. Please provide the password as requested.

Please keep in mind that the username "pbx" used in this guide is just an example. You’re free to use a different username, but remember to create the same username and password on all three PBX servers.

Install Ubuntu OS

The steps in this section guide you through the process of installing and configuring the Ubuntu operating system on all three node servers.

In this article, we installed the Ubuntu 20.04.

Select language

Install available updates for the installer

Keyboard configuration

Choose type of install

Please choose the Ubuntu Server.

Network connections

Please configure a static IP address for the server manually.

For the PBX node 1 (pbx01):

For the PBX node 2 (pbx02):

For the PBX node 3 (pbx03):

Configure proxy

Keep it as the default settings.

Configure Ubuntu archive mirror

Keep it as the default settings.

Guided storage configuration

Keep it as the default settings.

Storage configuration

Keep it as the default setting, the /dev/sdb should be unused state.

The /dev/sdb disk must remain uninitialized and in an unused state; otherwise, the installation will fail.

Profile setup

For all three PBX servers, a new user named "pbx" needs to be created. This user should be granted sudo permissions and should have a consistent password across all servers.

For the PBX node 1 (pbx01):

For the PBX node 2 (pbx02):

For the PBX node 3 (pbx03):

Upgrade to Ubuntu Pro

Keep it as the default settings: Skip for now.

SSH Setup

Install the OpenSSH server.

Featured Server Snaps

Keep it as the default settings.

Installing system

Start the process of installing the operating system and remain patient while it completes.

Install complete

Once the system is successfully installed, reboot the system (Reboot Now).

After the server reboot, use the "pbx" user credentials created during installation to log in and continue with the following steps.

Resolve all these servers' hostnames

Perform the following commands at all three nodes: pbx01, pbx02, pbx03. Note: you need to replace the IP and hostname in the following command with your actual IP and hostname.

sudo bash -c 'cat >> /etc/hosts' << EOF
192.168.1.131 pbx01
192.168.1.132 pbx02
192.168.1.133 pbx03
EOF

Set password-free login

In this guide, "pbx01", "pbx02", and "pbx03" are the mean for node 1, node 2, and node 3, respectively. The command provided below should only be executed on the node "pbx01".

Generate the certificate

Perform the below command only on the pbx01:

ssh-keygen -t rsa

Follow the prompts to generate the certificate - press the enter button if required.

Set password-free login for all three servers

The following commands provided below should only be executed on the node "pbx01".

If you are prompted to choose an option (yes/no), please enter yes.

ssh-copy-id -i ~/.ssh/id_rsa.pub pbx01

ssh-copy-id -i ~/.ssh/id_rsa.pub pbx02

ssh-copy-id -i ~/.ssh/id_rsa.pub pbx03

Configuring PortSIP PBX HA

Download the PBX resource package

The following command should only be executed on the node "pbx01".

cd /opt/ && sudo wget -N https://www.portsip.com/downloads/ha/v16/portsip-pbx-ha-guide-16-online.tar.gz \ 
&& sudo tar xf portsip-pbx-ha-guide-16-online.tar.gz

Set variables

Please prepare the value for the below Variables.

Name	Type	Description
pbx01_hostname	string	The hostname of node 1, in this case is pbx01
pbx02_hostname	string	The hostname of node 2, in this case is pbx02
pbx03_hostname	string	The hostname of node 3, in this case is pbx03
pbx01_private_ip	string	The private static IP of node 1, in this case is 192.168.1.131
pbx02_private_ip	string	The private static IP of node 2, in this case is 192.168.1.132
pbx03_private_ip	string	The private static IP of node 3, in this case is 192.168.1.133
vip	string	The virtual IP in this case is 192.168.1.130
pbx_image	string	PortSIP PBX docker image in this case is portsip/pbx:16.4.2.2771-release
pbx_datapath_disk	string	The disk volume for PBX data store in this case is /dev/sdb
pbx_datapath_size	string	The disk volume size for PBX data store，in this case is 36G(If set as full 40G maybe report error); For example if the disk volume size is 500G , suggest set as 490G.

The following command is only executed on the node pbx01.

cd /opt/portsip-pbx-ha-guide && sudo bash -c 'cat > ./res/vars.yml' << EOF
pbx01_hostname: pbx01
pbx02_hostname: pbx02
pbx03_hostname: pbx03
pbx01_private_ip: 192.168.1.131
pbx02_private_ip: 192.168.1.132
pbx03_private_ip: 192.168.1.133
vip: 192.168.1.130
pbx_image: portsip/pbx:16.4.2.2771-release
pbx_datapath_disk: /dev/sdb
pbx_datapath_size: 36G
EOF

Install Dependencies

The following command is only executed on the node pbx01.

cd /opt/portsip-pbx-ha-guide/ && /bin/bash install_dependencies.sh

Reboot Servers

The following commands are only executed on the node pbx01 (Note, the pbx01 must be restarted after the pbx02, and pbx03 restarted).

Restart pbx03

ssh -t pbx03 "sudo reboot"

Restart pbx02

ssh -t pbx02 "sudo reboot"

Restart pbx01

ssh -t pbx01 "sudo reboot"

Initialize Resources

Only execute the following command on the node pbx01, the execution may take some time, so patience is required. Please do not interrupt, restart, or shut down while the process is in progress.

cd /opt/portsip-pbx-ha-guide/ && /bin/sh deploy_pbx.sh

Once the resource configuration is complete, you can access your PBX by opening https://192.168.1.130:8887 in a web browser. All future PBX management, configuration, and access will be done through the virtual IP 192.168.1.130.

Configuring PortSIP PBX

When you sign into the PBX Web portal, the setup wizard will automatically pop up. In its first step, it asks for setting up the IP addresses and entering the virtual IP 192.168.1.130 for the Private IPv4 filed. If your virtual IP is different from 192.168.1.130, please enter your actual virtual IP

Once successfully completed the Setup Wizard, select the menu Advanced > Settings > Advanced, select the Enable Call Recovery option, and confirm by clicking on the OK button.

Managing PBX HA

Once you successfully configured the PortSIP PBX HA as per the above steps, you can use the following commands to manage the PBX HA.

All commands should be performed only on the pbx01.

View PBX HA Status

cd /opt/portsip-pbx-ha-guide && /bin/bash ha_ctl.sh status

The following output indicates that the PBX HA is working properly.

 drbd_attr    (ocf::linbit:drbd-attr):    Started pbx01
 Master/Slave Set: drbd_devpath-clone [drbd_devpath]
     Masters: [ pbx01 ]
     Slaves: [ pbx02 pbx03 ]
 vip    (ocf::heartbeat:IPaddr2):    Started pbx01
 src_pkt_ip    (ocf::heartbeat:IPsrcaddr):    Started pbx01
 datapath_fs    (ocf::heartbeat:Filesystem):    Started pbx01
 pbx    (ocf::portsip:pbx):    Started pbx01

Restart PBX

Use the below command to restart the PBX.

cd /opt/portsip-pbx-ha-guide && /bin/bash ha_ctl.sh restart -s pbx

The following output indicates that the PBX is successfully restarted.

try to restart resource pbx
disable resource pbx
resource 'pbx' is not running on any node
enable resource pbx