pfSense® High Availability

In this configuration we will have 4 network interfaces. SYNC, dedicated to state and configurations synchronization. WAN for external connections. MGMT, dedicated to pfSense® management. DMZ, for internal servers.

I omit LAN interface because for this porpuse is not necessary, some tests will make from a management pc, on MGMT network.

Ip Address and diagram

First of all it’s necessary to make plan of work.

1- ip address assignments

2- network diagram (logical and fisical)

3- servers naming

Note

All public IP addresses are taken from IP reserved by IETF (TEST-NET-3) documentation and examples.

WAN Addressing

The WAN network is assigned by your ISP, tipically is a `/29` network.
For HA systems it’s necessary to have 3 ip address, one for each interface and one for virtual interface called CARP. To connect each nodes to te internet it needed a Gateway and a pair of DNS server, for this lab use a Google DNS: 8.8.8.8 and 8.8.4.4.

WAN IP Address
IP Address	Interface	Node
`203.0.113.254/29`	CARP	Shared on all Nodes
`203.0.113.253/29`	WAN	Primary Node
`203.0.113.252/29`	WAN	Secondary Node

MGMT Addressing

The MGMT network is private and protected network used only for network mangement.
Best practice is to choose, a non-canonical ip address like 192.168.0.0/24 to avoid future overlaps.

MGMT IP Address
IP Address	Interface	Node
`172.18.0.254/24`	CARP	Shared on all Nodes
`172.18.0.253/24`	MGMT	Primary Node
`172.18.0.252/24`	MGMT	Secondary Node

DMZ Addressing

The DMZ network is tipically a public network asigned by your ISP, in this example we will use a private network with NAT.
Best practice is to choose, a non-canonical ip address like 192.168.0.0/24 to avoid future overlaps.

DMZ IP Address
IP Address	Interface	Node
`192.168.34.254/24`	CARP	Shared on all Nodes
`192.168.34.253/24`	DMZ	Primary Node
`192.168.34.252/24`	DMZ	Secondary Node

SYNC Addreessing

The SYNC network is a private and protected subnet, only pfSense® nodes need comunicate with this interfaces, and not need CARP virtual ip.

SYNC IP Address
IP Address	Interface	Node
`10.26.58.1/28`	SYNC	Primary Node
`10.26.58.2/28`	SYNC	Secondary Node

Network diagram

The network diagram show how pfSense® and network switchs are plugged.
This diagram is valid for both fisical and virtual devices. For complete High Availability system, the network must also be totally redundant, in this example only the ISP router is not redundant. whether you use switchs for each subnet, or vlans the logical connection diagram remains the same. In this documents i provide two diagram; logical and fisical.

Attention

This documents not provide how configure switchs for VLANs or STP.

HA Fisical Diagram

HA Logical Diagram

Server Naming

Each nodes need a FQDN.

Nodes FQDN
FQDN	Node
heimdall.oneos.it	Shared on all nodes
heimdall01.oneos.it	Primary Node
heimdall02.oneos.it	Secondary Node

Cluster Configuration Basics

Install the OS and assign the interface at the same order on both nodes, like this, it is suggested to set the ip addresses in static mode.

Warning

If the interface are not aligned, configurations synchronization and other task will not work properly. Every interface additionally configuration must be replicated on all nodes.

Best practice is to assign interface and ip address from console for WAN and LAN as explain below and change LAN name in MGMT from web GUI.
Set correct DHCP pool, only on Primary node.

Console interface IP - Primary Node

FreeBSD/amd64 (pfSense.home.arpa) (ttyu0)

pfSense - Netgate Device ID: 98f0e53206c36b104aff

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> 
 LAN (lan)       -> igb1       -> v4: 192.168.1.1/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 2

Available interfaces:

1 - WAN (igb0 - dhcp, dhcp6)
2 - LAN (igb1 - static)

Enter the number of the interface you wish to configure: 1

Configure IPv4 address WAN interface via DHCP? (y/n) n

Enter the new WAN IPv4 address.  Press <ENTER> for none:
> 203.0.113.253

Subnet masks are entered as bit counts (as in CIDR notation) in pfSense.
e.g. 255.255.255.0 = 24
     255.255.0.0   = 16
     255.0.0.0     = 8

Enter the new WAN IPv4 subnet bit count (1 to 32):
> 29

For a WAN, enter the new WAN IPv4 upstream gateway address.
For a LAN, press <ENTER> for none:
> 203.0.113.249

Configure IPv6 address WAN interface via DHCP6? (y/n) n

Enter the new WAN IPv6 address.  Press <ENTER> for none:
> 
Disabling IPv4 DHCPD...
Disabling IPv6 DHCPD...

Do you want to revert to HTTP as the webConfigurator protocol? (y/n) n

Please wait while the changes are saved to WAN...
 Reloading filter...
 Reloading routing configuration...
 DHCPD...

The IPv4 WAN address has been set to 203.0.113.253/29

Press <ENTER> to continue.
pfSense - Netgate Device ID: 98f0e53206c36b104aff

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> v4: 203.0.113.253/29
 LAN (lan)       -> igb1       -> v4: 192.168.1.1/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 2

Available interfaces:

1 - WAN (igb0 - static)
2 - LAN (igb1 - static)

Enter the number of the interface you wish to configure: 2

Enter the new LAN IPv4 address.  Press <ENTER> for none:
> 172.18.0.253

Subnet masks are entered as bit counts (as in CIDR notation) in pfSense.
e.g. 255.255.255.0 = 24
     255.255.0.0   = 16
     255.0.0.0     = 8

Enter the new LAN IPv4 subnet bit count (1 to 32):
> 24

For a WAN, enter the new LAN IPv4 upstream gateway address.
For a LAN, press <ENTER> for none:
> 

Enter the new LAN IPv6 address.  Press <ENTER> for none:
> 

Do you want to enable the DHCP server on LAN? (y/n) y
Enter the start address of the IPv4 client address range: 172.18.0.10
Enter the end address of the IPv4 client address range: 172.18.0.20
Disabling IPv6 DHCPD...

Do you want to revert to HTTP as the webConfigurator protocol? (y/n) n

Please wait while the changes are saved to LAN...
 Reloading filter...
 Reloading routing configuration...
 DHCPD...

The IPv4 LAN address has been set to 172.18.0.253/24
You can now access the webConfigurator by opening the following URL in your web browser:
                https://172.18.0.253/

Press <ENTER> to continue.
pfSense - Netgate Device ID: 98f0e53206c36b104aff

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> v4: 203.0.113.253/29
 LAN (lan)       -> igb1       -> v4: 172.18.0.253/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 

Console interface IP - Secondary Node

FreeBSD/amd64 (pfSense.home.arpa) (ttyu0)

pfSense - Netgate Device ID: 72278514ec4359e94ac9

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> 
 LAN (lan)       -> igb1       -> v4: 192.168.1.1/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 2

Available interfaces:

1 - WAN (igb0 - dhcp, dhcp6)
2 - LAN (igb1 - static)

Enter the number of the interface you wish to configure: 1

Configure IPv4 address WAN interface via DHCP? (y/n) n

Enter the new WAN IPv4 address.  Press <ENTER> for none:
> 203.0.113.252

Subnet masks are entered as bit counts (as in CIDR notation) in pfSense.
e.g. 255.255.255.0 = 24
     255.255.0.0   = 16
     255.0.0.0     = 8

Enter the new WAN IPv4 subnet bit count (1 to 32):
> 29

For a WAN, enter the new WAN IPv4 upstream gateway address.
For a LAN, press <ENTER> for none:
> 203.0.113.249

Configure IPv6 address WAN interface via DHCP6? (y/n) n

Enter the new WAN IPv6 address.  Press <ENTER> for none:
> 
Disabling IPv4 DHCPD...
Disabling IPv6 DHCPD...

Do you want to revert to HTTP as the webConfigurator protocol? (y/n) n

Please wait while the changes are saved to WAN...
 Reloading filter...
 Reloading routing configuration...
 DHCPD...

The IPv4 WAN address has been set to 203.0.113.252/29

Press <ENTER> to continue.
pfSense - Netgate Device ID: 72278514ec4359e94ac9

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> v4: 203.0.113.252/29
 LAN (lan)       -> igb1       -> v4: 192.168.1.1/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 2

Available interfaces:

1 - WAN (igb0 - static)
2 - LAN (igb1 - static)

Enter the number of the interface you wish to configure: 2

Enter the new LAN IPv4 address.  Press <ENTER> for none:
> 172.18.0.252

Subnet masks are entered as bit counts (as in CIDR notation) in pfSense.
e.g. 255.255.255.0 = 24
     255.255.0.0   = 16
     255.0.0.0     = 8

Enter the new LAN IPv4 subnet bit count (1 to 32):
> 24

For a WAN, enter the new LAN IPv4 upstream gateway address.
For a LAN, press <ENTER> for none:
> 

Enter the new LAN IPv6 address.  Press <ENTER> for none:
> 

Do you want to enable the DHCP server on LAN? (y/n) n
Disabling IPv4 DHCPD...
Disabling IPv6 DHCPD...

Do you want to revert to HTTP as the webConfigurator protocol? (y/n) n

Please wait while the changes are saved to LAN...
 Reloading filter...
 Reloading routing configuration...
 DHCPD...

The IPv4 LAN address has been set to 172.18.0.252/24
You can now access the webConfigurator by opening the following URL in your web browser:
                https://172.18.0.252/

Press <ENTER> to continue.
pfSense - Netgate Device ID: 72278514ec4359e94ac9

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on pfSense ***

 WAN (wan)       -> igb0       -> v4: 203.0.113.252/29
 LAN (lan)       -> igb1       -> v4: 172.18.0.252/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 

Interface on Primary Node

Connect a management pc on LAN network and set it’s network interface in DHCP mode, access to pfSense® from browser at this URL; https://172.18.0.253. When running wizard set admin password, hostname, domain and DNS, leave all as default for WAN interface, and change the LAN name to MGMT, leave the rest at default. Now set other interface from web GUI and assign the chosen ip address. The final result must be as below.

Primary Node Interfaces

SYNC Setup on Primary Node

Firts step for activate Sync Setup is make a firewall rule on SYNC interface for communications between nodes.

SYNC Rules on Primary

Now create a user syncUSER with “System - HA node sync” privilege and keep in mind the password. Navigate to System → User Manager → Users. Click

Sync User Creation [part 1]

Click

Now edit user syncUSER by click on and add Effective Privileges to “System - HA node sync”.

Sync User Creation [part 2]

Interface on Secondary Node

Connect a management pc on LAN network and set it’s network interface in DHCP mode, access to pfSense® from browser at this URL; https://172.18.0.252.
When running wizard set admin password and leave all as default for WAN interface, and change the LAN name to MGMT, leave the rest at default. Now set other interface from web GUI and assign the chosen ip address. The final result must be as below.

Secondary Node Interfaces

SYNC Setup on Secondary Node

Firts step for activate Sync Setup is make a firewall rule on SYNC interface for communications between nodes.
Set rule as premit any any, this rule will overwritten after configuration synchronization.

SYNC Rules on Secondary

Now create a user with “System - HA node sync” privilege identically at user created on Primary node, including the password.

State Synchronization Settings (pfsync) on Primary Node

For synchronization process to use pfsync, an utility that perform states synchronizations between pfSense®.

Navigate to System → High Avail. Sync.

Synchronize States:: checked
Synchronize Interface:: SYNC
Sinchronize Peer IP:: 10.26.58.2

Click

State Synchronization Settings (pfsync) on Secondary Node

For synchronization process to use pfsync, an utility that perform states synchronizations between pfSense®.

Navigate to System → High Avail. Sync.

Synchronize States:: checked
Synchronize Interface:: SYNC
Filter Host ID:: 02
Sinchronize Peer IP:: 10.26.58.1

Click

Configuration Synchronization Settings (XMLRPC Sync) on Primary Node

The XMLRPC protocols synchronize pfSense® configuration file (config.xml) with other nodes of cluster.

Attention

Configuration synchronization must only be configured on the primary node. Never activate options in this section on the secondary node of a two-member cluster.

Navigate to System → High Avail. Sync.

Synchronize Config IP:: 10.26.58.2
Remote System Username:: syncUSER
Remote System Password:: REGISTERED PASSWORD FOR USER syncUSER
Select option to sync:: Click button

Checking configuration synchronization by view rules on interface SYNC. If synchronization works, the rule must be the same applied on Primary Node.

On MGMT interface.

Failover peer IP: 172.18.0.252

On Secondary this settings will applied automatically on 172.16.0.253.

Warning

Do not make changes to the secondary in areas set to be synchronized! These changes will be overwritten the next time the primary node performs a synchronization.

Configuring CARP Virtual IP on Primary Node

BSD systems use a protocol named CARP to elect a Primary Node in a cluster.
This protocol works like VRRP.

Warning

CARP interfaces must create at the same order you created interfaces, and only on Primary Node.

Navigate to Firewall → Virtual IPs.
Click

Configuring WAN CARP

Click

Type:

CARP

Interface:

WAN

Address type:

Single address

Address(es)

203.0.113.254/29

Virtual IP Password:

Chose a password for this CARP

VHID Group:

1

Advertising frequency:

Base:: 1
Skew:: 0

Description:

WAN CARP Gateway

Click

Configuring MGMT CARP

Click

Type:

CARP

Interface:

MGMT

Address type:

Single address

Address(es)

172.18.0.254/24

Virtual IP Password:

Chose a password for this CARP

VHID Group:

2

Advertising frequency:

Base:: 1
Skew:: 0

Description:

MGMT CARP Gateway

Click

Configuring DMZ CARP

Click

Type:

CARP

Interface:

DMZ

Address type:

Single address

Address(es)

192.168.34.254/24

Virtual IP Password:

Chose a password for this CARP

VHID Group:

3

Advertising frequency:

Base:: 1
Skew:: 0

Description:

DMZ CARP Gateway

Click

Virtual ip address list looks like this.

Carp List

To verify confiuguration, go to Status → CARP (failover) on both nodes.

CARP Staus Primary Node

CARP Staus Secondary Node

There is two button:

and

The first one change the role from MASTER to BACKUP until the node was rebooted or when the new button is pressed.
The second one change the role from MASTER to BACKUP in a persitent mode, for return to MASTER role need to press the new button

Attention

When you use those buttons you must login to admin page not on CARP Interface but on the single node ip address, it is also a good practice to change the theme of the administration interface of the secondary nodes to understand on the fly where you are operating.

Configure Outbound NAT

When CARP is used it’s necessary to manual adjust Outbound NAT for use of CARP IPs insted of interface IPs.
Navigate to Firewall → NAT → Outbound.

Mode:: Hybrid Outbound NAT rule generator.

This metod generate automatically NAT rule and it’s already possible to add/modify manual rules instead of automatic NAT rule generator.

Click

Now you must insert manual outbound NAT mappings, click Add.

Interface:: WAN
Address Family:: IPv4
Source:: MGMT subnet 172.18.0.0/24
Traslation->Address: WAN CARP Gateway 203.0.113.254
Description:: MGMT to WAN

Click

Now click Add.

Interface:

WAN

Address Family:

IPv4

Source:

MGMT subnet 172.18.0.0/24

Destination:

Type:: Any
Port:: 500

Traslation:

Address:: WAN CARP Gateway 203.0.113.254
Port or Range:: Check Static

Description:

ISAKMP to WAN

Click

The first rule permit to translate MGMT IPs to CARP WAN Gateway instead of single interface IP. The second rule is for connection to external IPSEC VPN.

Continue by creating the two rules for the DMZ interface, taking care to modify the Source field with the network 192.168.34.0/24.
The Automatic Rules remain in configuration only for the network than not mapped in Mappings, this section overlaps the default rules.

When complete the result look like this:

Outbound NAT Rules

Adjust DHCP Server

Modify DHCP configuration on Primary for make service in HA. This setup need to be make on all interfaces with DHCP activated.

Go to Services → DHCP Server → MGMT.

DNS Server:: MGMT CARP Gateway, 172.18.0.254
Gateway:: MGMT CARP Gateway, 172.18.0.254
Failover Peer IP: MGMT Secondary interface ip, 172.16.0.252

Click

The Secondary will automatically adjust Failover Peer IP with correct value, 172.18.0.253

Warning

Those changes need to make on all interface that have DHCP Server activated and a CARP interface.

Testing HA

Login on Primary Node by node ip address and not by CARP Ip.
On another browser tab login on Backup Node by node ip address and not by CARP Ip.

Go to Primary Node tab.
It’s time to perform a test. Navigate to Status → CARP (failover). There are two buttons; and , respectively;

CARP will be disable until reboot of the Node or when the new button is pressed.
CARP remain disabled until the button is pressed.

Tip

is necessary when perform a system upgrade of a node or when makes change on interfaces.

Test One

Make an infinite ping to google DNS 8.8.8.8 from management pc.
On Primary Node click on button , if all works as aspected, you don’t will lost any ping and the Status column will go in BACKUP.
On Secondary Node the Status column will go in MASTER.
If all works as aspected, you don’t will lost any ping.

Test Two

Make an infinite ping to google DNS 8.8.8.8 from management pc.
On Primary Node go to Diagnostics → Reboot. Select Normal Reboot and press Submit.
After reboot as finished the Primary Node will return as MASTER.
If all works as aspected, you don’t will lost any ping.

Test Three

Make an infinite ping to google DNS 8.8.8.8 from management pc.
Perform a hot shutdown of Primary Node by removing the power supply.
If all works as aspected, you will lost only one or two ping.

Tip

If all works as aspected you must perform a configuration backup of all nodes.