=========================== |pfSense| High Availability =========================== .. meta:: :description: pfSense configuration recipes :keywords: pfSense, networking, security, firewall, network, firewalling :robots: noindex,nofollow In this configuration we will have 4 network interfaces. **SYNC**, dedicated to state and configurations synchronization. **WAN** for external connections. **MGMT**, dedicated to |pfsense| management. **DMZ**, for internal servers. I omit **LAN** interface because for this porpuse is not necessary, some tests will make from a management pc, on MGMT network. Ip Address and diagram ======================= First of all it's necessary to make plan of work. |br| 1- ip address assignments 2- network diagram (logical and fisical) 3- servers naming .. note:: All public IP addresses are taken from IP reserved by :abbr:`IETF (Internet Engineering Task Force)` (TEST-NET-3) documentation and examples. WAN Addressing --------------- The **WAN** network is assigned by your :abbr:`ISP (Internet Service Provider)`, tipically is a ```/29``` network. |br| For HA systems it's necessary to have 3 ip address, one for each interface and one for virtual interface called :abbr:`CARP (Common Address Redundancy Protocol)`. To connect each nodes to te internet it needed a Gateway and a pair of :abbr:`DNS (Domain Name System)` server, for this lab use a Google DNS: 8.8.8.8 and 8.8.4.4. .. list-table:: WAN IP Address :widths: 25 25 35 :header-rows: 1 * - IP Address - Interface - Node * - ``203.0.113.254/29`` - CARP - Shared on all Nodes * - ``203.0.113.253/29`` - WAN - Primary Node * - ``203.0.113.252/29`` - WAN - Secondary Node MGMT Addressing ----------------- The **MGMT** network is private and protected network used only for network mangement. |br| Best practice is to choose, a non-canonical ip address like ``192.168.0.0/24`` to avoid future overlaps. .. list-table:: MGMT IP Address :widths: 25 25 35 :header-rows: 1 * - IP Address - Interface - Node * - ``172.18.0.254/24`` - CARP - Shared on all Nodes * - ``172.18.0.253/24`` - MGMT - Primary Node * - ``172.18.0.252/24`` - MGMT - Secondary Node DMZ Addressing ----------------- The **DMZ** network is tipically a public network asigned by your :abbr:`ISP (Internet Service Provider)`, in this example we will use a private network with :abbr:`NAT (Network Address Traslation)`. |br| Best practice is to choose, a non-canonical ip address like ``192.168.0.0/24`` to avoid future overlaps. .. list-table:: DMZ IP Address :widths: 25 25 35 :header-rows: 1 * - IP Address - Interface - Node * - ``192.168.34.254/24`` - CARP - Shared on all Nodes * - ``192.168.34.253/24`` - DMZ - Primary Node * - ``192.168.34.252/24`` - DMZ - Secondary Node SYNC Addreessing ----------------- The **SYNC** network is a private and protected subnet, only |pfsense| nodes need comunicate with this interfaces, and not need :abbr:`CARP (Common Address Redundancy Protocol)` virtual ip. .. list-table:: SYNC IP Address :widths: 25 25 35 :header-rows: 1 * - IP Address - Interface - Node * - ``10.26.58.1/28`` - SYNC - Primary Node * - ``10.26.58.2/28`` - SYNC - Secondary Node Network diagram ---------------- The network diagram show how |pfsense| and network switchs are plugged. |br| This diagram is valid for both fisical and virtual devices. For complete High Availability system, the network must also be totally redundant, in this example only the :abbr:`ISP (Internet Service Provider)` router is not redundant. whether you use switchs for each subnet, or vlans the logical connection diagram remains the same. In this documents i provide two diagram; logical and fisical. .. attention:: This documents not provide how configure switchs for VLANs or :abbr:`STP (Spanning Tree Protocol)`. .. figure:: images/diagram-ha.png :alt: HA fisical diagram :width: 650px :class: with-shadow :align: center HA Fisical Diagram ------------------------------------------------------------------------------- .. figure:: images/diagram-ha-logical.png :alt: HA logical diagram :width: 650px :class: with-shadow :align: center HA Logical Diagram Server Naming -------------- Each nodes need a :abbr:`FQDN (Fully Qualified Domain Name)`. .. list-table:: Nodes FQDN :widths: 25 25 :header-rows: 1 * - FQDN - Node * - heimdall.oneos.it - Shared on all nodes * - heimdall01.oneos.it - Primary Node * - heimdall02.oneos.it - Secondary Node Cluster Configuration Basics ============================= Install the :abbr:`OS (Operating System)` and assign the interface at the same order on both nodes, like this, it is suggested to set the ip addresses in static mode. .. warning:: If the interface are not aligned, configurations synchronization and other task will not work properly. Every interface additionally configuration must be replicated on all nodes. Best practice is to assign interface and ip address from console for **WAN** and **LAN** as explain below and change **LAN** name in **MGMT** from web :abbr:`GUI (Grafical User Interface)`. |br| Set correct :abbr:`DHCP (Dynamic Host Configuration Protocol)` pool, only on Primary node. .. literalinclude:: primary-int.txt :caption: Console interface IP - Primary Node :language: bash :emphasize-lines: 20,27,29,32,40,44,46,53,80,87,90,98,112 :linenos: .. literalinclude:: secondary-int.txt :caption: Console interface IP - Secondary Node :language: bash :emphasize-lines: 20,27,29,32,40,44,46,53,80,87,90,98,112 :linenos: Interface on Primary Node --------------------------- Connect a management pc on **LAN** network and set it's network interface in :abbr:`DHCP (Dynamic Host Configuration Protocol)` mode, access to |pfsense| from browser at this :abbr:`URL (Uniform Resource Locator)`; https://172.18.0.253. When running wizard set admin password, hostname, domain and DNS, leave all as default for **WAN** interface, and change the **LAN** name to **MGMT**, leave the rest at default. Now set other interface from web :abbr:`GUI (Grafical User Interface)` and assign the chosen ip address. The final result must be as below. .. figure:: images/primary-int.png :alt: Primary Node Interfaces :width: 700px :align: center Primary Node Interfaces SYNC Setup on Primary Node ---------------------------- Firts step for activate Sync Setup is make a firewall rule on SYNC interface for communications between nodes. .. figure:: images/sync-rules.png :alt: SYNC Rules :width: 700px :align: center SYNC Rules on Primary Now create a user ``syncUSER`` with “System - HA node sync” privilege and keep in mind the password. Navigate to **System** |rarrow| **User Manager** |rarrow| **Users**. Click |btn-add-plus| .. figure:: images/syncuser1.png :alt: Sync User Creation :width: 700px :align: center Sync User Creation [part 1] Click |btn-save| Now edit user ``syncUSER`` by click on |fa-pencil| and add **Effective Privileges** to “System - HA node sync”. .. figure:: images/syncuser2.png :alt: Sync User Creation :width: 700px :align: center Sync User Creation [part 2] Interface on Secondary Node ---------------------------- Connect a management pc on **LAN** network and set it's network interface in :abbr:`DHCP (Dynamic Host Configuration Protocol)` mode, access to |pfsense| from browser at this :abbr:`URL (Uniform Resource Locator)`; https://172.18.0.252. |br| When running wizard set admin password and leave all as default for **WAN** interface, and change the **LAN** name to **MGMT**, leave the rest at default. Now set other interface from web :abbr:`GUI (Grafical User Interface)` and assign the chosen ip address. The final result must be as below. .. figure:: images/secondary-int.png :alt: Secondary Node Interfaces :width: 700px :align: center Secondary Node Interfaces SYNC Setup on Secondary Node ------------------------------- Firts step for activate Sync Setup is make a firewall rule on SYNC interface for communications between nodes. |br| Set rule as **premit any any**, this rule will overwritten after configuration synchronization. .. figure:: images/sync-rules-sec.png :alt: SYNC Rules :width: 700px :align: center SYNC Rules on Secondary Now create a user with “System - HA node sync” privilege identically at user created on Primary node, including the password. State Synchronization Settings (`pfsync`) on Primary Node ----------------------------------------------------------- For synchronization process to use ``pfsync``, an utility that perform states synchronizations between |pfsense|. |br| Navigate to **System** |rarrow| **High Avail. Sync**. |br| Synchronize States: checked Synchronize Interface: SYNC Sinchronize Peer IP: ``10.26.58.2`` Click |btn-save| State Synchronization Settings (pfsync) on Secondary Node ----------------------------------------------------------- For synchronization process to use ``pfsync``, an utility that perform states synchronizations between |pfsense|. |br| Navigate to **System** |rarrow| **High Avail. Sync**. |br| Synchronize States: checked Synchronize Interface: SYNC Filter Host ID: 02 Sinchronize Peer IP: ``10.26.58.1`` Click |btn-save| Configuration Synchronization Settings (XMLRPC Sync) on Primary Node ------------------------------------------------------------------------- The XMLRPC protocols synchronize |pfsense| configuration file (``config.xml``) with other nodes of cluster. |br| .. attention:: Configuration synchronization **must only be configured on the primary node**. Never activate options in this section on the secondary node of a two-member cluster. Navigate to **System** |rarrow| **High Avail. Sync**. |br| Synchronize Config IP: ``10.26.58.2`` Remote System Username: ``syncUSER`` Remote System Password: REGISTERED PASSWORD FOR USER ``syncUSER`` Select option to sync: Click |btn-toggle| button Checking configuration synchronization by view rules on interface **SYNC**. If synchronization works, the rule must be the same applied on Primary Node. On **MGMT** interface. Failover peer IP ``172.18.0.252`` On Secondary this settings will applied automatically on ``172.16.0.253``. .. Warning:: Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization. Configuring CARP Virtual IP on Primary Node ============================================ **BSD** systems use a protocol named :abbr:`CARP (Common Address Redundancy Protocol)` to elect a Primary Node in a cluster. |br| This protocol works like :abbr:`VRRP (Virtual Router Redundancy Protocol)`. |br| .. warning:: CARP interfaces must create at the same order you created interfaces, and only on Primary Node. Navigate to **Firewall** |rarrow| **Virtual IPs**. |br| Click |btn-add-plus| Configuring **WAN CARP** ------------------------- Click |btn-add-plus| Type: CARP Interface: WAN Address type: Single address Address(es) ``203.0.113.254/29`` Virtual IP Password: Chose a password for this CARP VHID Group: ``1`` Advertising frequency: Base: ``1`` Skew: ``0`` Description: WAN CARP Gateway Click |btn-save| Configuring **MGMT CARP** ------------------------- Click |btn-add-plus| Type: CARP Interface: MGMT Address type: Single address Address(es) ``172.18.0.254/24`` Virtual IP Password: Chose a password for this CARP VHID Group: ``2`` Advertising frequency: Base: ``1`` Skew: ``0`` Description: MGMT CARP Gateway Click |btn-save| Configuring **DMZ CARP** ------------------------- Click |btn-add-plus| Type: CARP Interface: DMZ Address type: Single address Address(es) ``192.168.34.254/24`` Virtual IP Password: Chose a password for this CARP VHID Group: ``3`` Advertising frequency: Base: ``1`` Skew: ``0`` Description: DMZ CARP Gateway Click |btn-save| Virtual ip address list looks like this. |br| .. figure:: images/carp-list.png :alt: Carp List :width: 700px :align: center Carp List To verify confiuguration, go to **Status** |rarrow| **CARP (failover)** on both nodes. .. figure:: images/carp-status-primary.png :alt: CARP Status :width: 700px :align: center CARP Staus Primary Node .. figure:: images/carp-status-secondary.png :alt: CARP Status :width: 700px :align: center CARP Staus Secondary Node There is two button: |btn-carp-temp| and |btn-carp-pers| The first one change the role from **MASTER** to **BACKUP** until the node was rebooted or when the new button |btn-carp-enable| is pressed. |br| The second one change the role from **MASTER** to **BACKUP** in a persitent mode, for return to **MASTER** role need to press the new button |btn-carp-pers-leave| .. attention:: When you use those buttons you must login to admin page not on **CARP Interface** but on the single node ip address, it is also a good practice to change the theme of the administration interface of the secondary nodes to understand on the fly where you are operating. Configure Outbound NAT ================================= When CARP is used it's necessary to manual adjust **Outbound NAT** for use of CARP IPs insted of interface IPs. |br| Navigate to **Firewall** |rarrow| **NAT** |rarrow| **Outbound**. Mode: Hybrid Outbound NAT rule generator. This metod generate automatically NAT rule and it's already possible to add/modify manual rules instead of automatic NAT rule generator. Click |btn-save| Now you must insert manual outbound NAT mappings, click |fa-level-up| **Add**. Interface: WAN Address Family: IPv4 Source: **MGMT** subnet ``172.18.0.0/24`` Traslation->Address WAN CARP Gateway ``203.0.113.254`` Description: MGMT to WAN Click |btn-save| Now click |fa-level-up| **Add**. Interface: WAN Address Family: IPv4 Source: **MGMT** subnet ``172.18.0.0/24`` Destination: Type: Any Port: ``500`` Traslation: Address: WAN CARP Gateway ``203.0.113.254`` Port or Range: Check **Static** Description: ISAKMP to WAN Click |btn-save| The first rule permit to translate **MGMT** IPs to **CARP WAN Gateway** instead of single interface IP. The second rule is for connection to external IPSEC VPN. Continue by creating the two rules for the **DMZ** interface, taking care to modify the **Source** field with the network ``192.168.34.0/24``. |br| The **Automatic Rules** remain in configuration only for the network than not mapped in **Mappings**, this section overlaps the default rules. When complete the result look like this: .. figure:: images/out-nat.png :alt: Outbound NAT :width: 700px :align: center Outbound NAT Rules Adjust DHCP Server ===================== Modify :abbr:`DHCP (Dynamic Host Configuration Protocol)` configuration on Primary for make service in **HA**. This setup need to be make on all interfaces with :abbr:`DHCP (Dynamic Host Configuration Protocol)` activated. Go to **Services** |rarrow| **DHCP Server** |rarrow| **MGMT**. DNS Server: MGMT CARP Gateway, ``172.18.0.254`` Gateway: MGMT CARP Gateway, ``172.18.0.254`` Failover Peer IP MGMT Secondary interface ip, ``172.16.0.252`` Click |btn-save| The Secondary will automatically adjust **Failover Peer IP** with correct value, ``172.18.0.253`` .. warning:: Those changes need to make on all interface that have DHCP Server activated and a CARP interface. Testing HA =========== Login on **Primary Node** by node ip address and not by **CARP Ip**. |br| On another browser tab login on **Backup Node** by node ip address and not by **CARP Ip**. Go to **Primary Node** tab. |br| It's time to perform a test. Navigate to **Status** |rarrow| **CARP (failover)**. There are two buttons; |btn-carp-temp| and |btn-carp-pers|, respectively; - CARP will be disable until reboot of the Node or when the new button |btn-carp-enable| is pressed. - CARP remain disabled until the button |btn-carp-pers-leave| is pressed. |br| .. tip:: |btn-carp-pers| is necessary when perform a system upgrade of a node or when makes change on interfaces. Test One ------------ Make an infinite ping to google DNS ``8.8.8.8`` from management pc. |br| On **Primary Node** click on button |btn-carp-temp|, if all works as aspected, you don't will lost any ping and the **Status** column will go in **BACKUP**. |br| On **Secondary Node** the **Status** column will go in **MASTER**. |br| If all works as aspected, you don't will lost any ping. |br| Test Two --------- Make an infinite ping to google DNS ``8.8.8.8`` from management pc. |br| On Primary Node go to **Diagnostics** |rarrow| **Reboot**. Select **Normal Reboot** and press **Submit**. |br| After reboot as finished the **Primary Node** will return as **MASTER**. |br| If all works as aspected, you don't will lost any ping. Test Three ------------ Make an infinite ping to google DNS ``8.8.8.8`` from management pc. |br| Perform a hot shutdown of **Primary Node** by removing the power supply. |br| If all works as aspected, you will lost only one or two ping. .. tip:: If all works as aspected you must perform a configuration backup of all nodes.