Equal-cost multipath routing with Ifstated, Carp and PFsync on OpenBSD

Hey guys, here I will show you how to configure equal-cost multipath routing with some another features that are pretty interesting working together.

The multipath routing is a great feature but we have a problem when one of the internet links goes down, the multipath will send the packets to both internet links the good one and the another one that went down, so with that we need another daemon to take care of it and provide the knowledge about the dead internet link, so we can keep the conection throught the good link and when the dead link come alive we can active it and bring back the multipah.

But if the firewall goes down for one reason or another we will lost the entire connection with the internet with that we will configure the carp to guarantee if one firewall server goes down we have the other one to assume all the functions of the dead firewall server and keep to provide the network the servers with the same features that was give by the dead one.

The pfsync will keep track of the all state of the firewall and will replicate to the second firewall so even the connections will be keep in the second firewall no impacting the clients.

The problem in the environment:

In this scenario we only have one ip address of each internet link, so we need to configure a private network range in each interface that will be used by the carp as internet link with it we can use the public ip address in the master carp. So for example we have 10.10.10.0/24 as the physical ip address range for the internet link and the 200.200.200.0/28 as the carp ip address range that will be visible by the internet. So the Master carp always will have the valid public ip address and the backup carp will have the same ip in standby waiting to become master and assume the new public ip address without any problem with duplicate of ip address.

  • Firewall 01
    • OS Version: OpenBSD 6.0
    • DMZ IP: 172.31.0.243/24
    • DMZ Interface: xnf0
    • CARP DMZ: 172.31.0.245/24
      • CARP DMZ INTERFACE: carp1
      • Status: Master
      • vhid: 1
      • carpdev: xnf0
      • Advskew: 0
    • PFSync: 192.168.0.1/24
      • PFSync Interface: xnf1
      • syncdev: xnf1
      • syncpeer: 192.168.0.2
    • INTERNET LINK 01: 10.10.10.1/24
    • INTERNET LINK 01 Interface: xnf2
    • CARP INTERNET LINK 01: 200.200.200.61/27
      • CARP INTERNET Interface 01: carp2
      • Status: Master
      • vhid: 2
      • carpdev: xnf2
      • Advskew: 0
    • INTERNET LINK 02: 10.20.20.1/24
    • INTERNET LINK 02 Interface: xnf3
    • CARP INTERNET LINK 02: 189.189.189.108/28
      • CARP INTERNET Interface 02: carp3
      • Status: Master
      • vhid: 3
      • carpdev: xnf3
      • Advskew: 0
  • Firewall 02
    • OS Version: OpenBSD 6.0
    • DMZ IP: 172.31.0.244/24
    • DMZ Interface: xnf0
    • CARP DMZ: 172.31.0.245/24
      • CARP DMZ INTERFACE: carp1
      • Status: Backup
      • vhid: 1
      • carpdev: xnf0
      • Advskew: 10
    • PFSync: 192.168.0.2/24
      • PFSync Interface: xnf1
      • syncdev: xnf1
      • syncpeer: 192.168.0.1
    • INTERNET LINK 01: 10.10.10.2/24
    • INTERNET LINK 01 Interface: xnf2
    • CARP INTERNET LINK 01: 200.200.200.61/27
      • CARP INTERNET Interface 01: carp2
      • Status: Backup
      • vhid: 2
      • carpdev: xnf2
      • Advskew: 10
    • INTERNET LINK 02: 10.20.20.2/24
    • INTERNET LINK 02 Interface: xnf3
    • CARP INTERNET LINK 02: 189.189.189.108/28
      • CARP INTERNET Interface 02: carp3
      • Status: Backup
      • vhid: 3
      • carpdev: xnf3
      • Advskew: 10

Here we will configure the interfaces that will be used by the master carp server.

Note: I have no ideia why and how spaces matter in the interfaces configuration, so don't put more spaces than need. Ex: param + space + another_param and so on, otherwise you will have some pass throughout some troubles that can not be explained.

Carp Master DMZ Interface

Let's configure the physical interface with the real ip address that will be used to access the server throughtout the DMZ Network.

vim /etc/hostname.xnf0
inet 172.31.0.243 255.255.255.0 172.31.0.255 description "PHYSICAL LINK DMZ"

Let's configure the carp DMZ Interface that will be used by the DMZ clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. This ip address will be used as gateway of the servers in the DMZ Network.

vim /etc/hostname.carp1
inet 172.31.0.245 255.255.255.0 172.31.0.255 vhid 1 carpdev xnf0 pass lanpasswd advskew 0 description "CARP LINK DMZ F245"

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Carp Master PFSync Interface

Let's configure the physical interface with the real ip address that will be used to change information about the pfsync between the firewalls.

vim /etc/hostname.xnf1
inet 192.168.0.1 255.255.255.0 192.168.0.255 description "PHYSICAL LINK PFSYNC"

Now let's configure the virtual pfsync interface that keeps the information about the another peer of the pfsync configuration.

Note: If you are using some kind of virtual enviroment you need to create a private interface that can be used only between the the two firewalls, because the information that will be exchange between the servers are no encrypted.

vim /etc/hostname.pfsync0
up syncdev xnf1 syncpeer 192.168.0.2

You can get more information about the pfsync configuration in pfsync(4) and ifconfig(8)

Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.

vim /etc/hostname.xnf2
inet 10.10.10.1 255.255.255.0 10.10.10.255 description "PHYSICAL LINK 01"

Let's configure the carp Internet Link 01 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem.

Note: This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.

vim /etc/hostname.carp2
inet 200.200.200.61 255.255.255.224 200.200.200.63 vhid 2 carpdev xnf2 pass lanpasswd advskew 0 description "CARP LINK 01 F63"
!route add -mpath default 200.200.200.33

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.

vim /etc/hostname.xnf2
inet 10.20.20.1 255.255.255.0 10.20.20.255 description "PHYSICAL LINK 02"

Let's configure the carp Internet Link 02 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem.

Note: This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.

vim /etc/hostname.carp3
inet 189.189.189.108 255.255.255.240 189.189.189.111 vhid 3 carpdev xnf3 pass lanpasswd advskew 0 description "CARP LINK 02 F108"
!route add -mpath default 189.189.189.97

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Now we need to enable the equal cost multipath routing into the kernel. You can get more information about the multipath routing configuration in Multipath

sysctl net.inet.ip.multipath=1
sysctl net.inet6.ip6.multipath=1

We can enable the forward routing to act as a gateway.

sysctl net.inet.ip.forwarding=1

Now we need to configure the kernel to work with the carp. You can get more information about the carp configuration in carp

sysctl net.inet.carp.allow=1
sysctl net.inet.carp.preempt=1
sysctl net.inet.carp.log=2

Now we need to keep this configuration in the boot time, so let's edit the sysctl.conf

vim /etc/sysctl.conf
[...]
net.inet.ip.multipath=1
net.inet6.ip6.multipath=1
net.inet.ip.forwarding=1
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2

Desabling the default route in the Master Carp

Now we need to remove the /etc/mygate because we have multiples routes one for each internet link and we don't need this file. The routes is in the carp2 and carp3 in the !route add -mpath default x.x.x.x directive.

So let's remove the file

rm -rf /etc/mygate

Now restart the server to get all the configuration

reboot

Checking the Master Carp Configuration

Now we need to check the master carp configuration and double check if everything is working properly.

ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
        index 6 priority 0 llprio 3
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
xnf0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 4e:00:9e:30:0a:e3
        description: PHYSICAL LINK DMZ
        index 1 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 172.31.0.243 netmask 0xffffff00 broadcast 172.31.0.255
xnf1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr c2:f2:a4:52:e0:31
        description: PHYSICAL LINK PFSYNC
        index 2 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
xnf2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 7b:c1:91:bc:3a:4e
        description: PHYSICAL LINK 01
        index 3 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255
xnf3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr f2:ba:7c:97:28:ce
        description: PHYSICAL LINK 02
        index 4 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 10.20.20.1 netmask 0xffffff00 broadcast 10.20.20.255
enc0: flags=0<>
        index 5 priority 0 llprio 3
        groups: enc
        status: active
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:01
        description: CARP LINK DMZ F245
        index 7 priority 15 llprio 3
        carp: MASTER carpdev xnf0 vhid 1 advbase 1 advskew 0
        groups: carp
        status: master
        inet 172.31.0.245 netmask 0xffffff00 broadcast 172.31.0.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:02
        description: CARP LINK 01 F63
        index 8 priority 15 llprio 3
        carp: MASTER carpdev xnf2 vhid 2 advbase 1 advskew 0
        groups: carp egress
        status: master
        inet 200.200.200.61 netmask 0xffffffe0 broadcast 200.200.200.63
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:03
        description: CARP LINK 02 F108
        index 9 priority 15 llprio 3
        carp: MASTER carpdev xnf3 vhid 3 advbase 1 advskew 0
        groups: carp egress
        status: master
        inet 189.189.189.108 netmask 0xfffffff0 broadcast 189.189.189.111
pfsync0: flags=41<UP,RUNNING> mtu 1500
        index 10 priority 0 llprio 3
        pfsync: syncdev: xnf1 syncpeer: 192.168.0.2 maxupd: 128 defer: off
        groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33144
        index 11 priority 0 llprio 3
        groups: pflog

As we can see above the carp status is master for all the carp interfaces in the Master Server, the pfsync0 is working as well.

We can check the packets are working throughtout the pfsync0 with tcpdump

tcpdump -e -n -vv -ttt -i pfsync0
tcpdump: listening on pfsync0, link-type PFSYNC
Oct 13 15:31:48.270136 PFSYNCv6 len 556
    act UPD ST REQ count 2
        id: 57ff8b7c000002b1 creatorid: 7116b56a
        id: 57ff8b7c000002b2 creatorid: 7116b56a
    act UPD ST COMP count 6
    ...
Oct 13 15:31:48.270432 PFSYNCv6 len 552
    act UPD ST count 2
    ...
Oct 13 15:31:49.280142 PFSYNCv6 len 556
    act UPD ST REQ count 2
        id: 57ff8b7c000002b1 creatorid: 7116b56a
        id: 57ff8b7c000002b2 creatorid: 7116b56a
    act UPD ST COMP count 6
    ...
Oct 13 15:31:49.280461 PFSYNCv6 len 552
    act UPD ST count 2

Another way to check if the pfsync is working is double checking the pf status

pfctl -sa
[...]
STATES:
all pfsync 192.168.0.1 -> 192.168.0.2       MULTIPLE:MULTIPLE
[...]

The states of both firewall need to be the same in the states section.

Here we will configure the interfaces that will be used by the Backup carp server.

Note: I have no ideia why and how spaces matter in the interfaces configuration, so don't put more spaces than need. Ex: param + space + another_param and so on, otherwise you will have some pass throughout some troubles that can not be explained.

Carp Backup DMZ Interface

Let's configure the physical interface with the real ip address that will be used to access the server throughtout the DMZ Network.

vim /etc/hostname.xnf0
inet 172.31.0.244 255.255.255.0 172.31.0.255 description "PHYSICAL LINK DMZ"

Let's configure the carp DMZ Interface that will be used by the DMZ clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. This ip address will be used as gateway of the servers in the DMZ Network.

This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.

vim /etc/hostname.carp1
inet 172.31.0.245 255.255.255.0 172.31.0.255 vhid 1 carpdev xnf0 pass lanpasswd advskew 10 description "CARP LINK DMZ F245"

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Carp Backup PFSync Interface

Let's configure the physical interface with the real ip address that will be used to change information about the pfsync between the firewalls.

vim /etc/hostname.xnf1
inet 192.168.0.2 255.255.255.0 192.168.0.255 description "PHYSICAL LINK PFSYNC"

Now let's configure the virtual pfsync interface that keeps the information about the another peer of the pfsync configuration.

Note: If you are using some kind of virtual enviroment you need to create a private interface that can be used only between the the two firewalls, because the information that will be exchange between the servers are no encrypted.

vim /etc/hostname.pfsync0
up syncdev xnf1 syncpeer 192.168.0.1

You can get more information about the pfsync configuration in pfsync(4) and ifconfig(8)

Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.

vim /etc/hostname.xnf2
inet 10.10.10.2 255.255.255.0 10.10.10.255 description "PHYSICAL LINK 01"

Let's configure the carp Internet Link 01 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem.

This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.

Note: This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.

vim /etc/hostname.carp2
inet 200.200.200.61 255.255.255.224 200.200.200.63 vhid 2 carpdev xnf2 pass lanpasswd advskew 10 description "CARP LINK 01 F63"
!route add -mpath default 200.200.200.33

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.

vim /etc/hostname.xnf2
inet 10.20.20.2 255.255.255.0 10.20.20.255 description "PHYSICAL LINK 02"

Let's configure the carp Internet Link 02 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem.

This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.

Note: This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.

vim /etc/hostname.carp3
inet 189.189.189.108 255.255.255.240 189.189.189.111 vhid 3 carpdev xnf3 pass lanpasswd advskew 10 description "CARP LINK 02 F108"
!route add -mpath default 189.189.189.97

You can see more options about the carp configuration in carp(4). The description will help you to get to know which link is what.

Now we need to enable the equal cost multipath routing into the kernel. You can get more information about the multipath routing configuration in Multipath

sysctl net.inet.ip.multipath=1
sysctl net.inet6.ip6.multipath=1

We can enable the forward routing to act as a gateway.

sysctl net.inet.ip.forwarding=1

Now we need to configure the kernel to work with the carp. You can get more information about the carp configuration in carp

sysctl net.inet.carp.allow=1
sysctl net.inet.carp.preempt=1
sysctl net.inet.carp.log=2

Now we need to keep this configuration in the boot time, so let's edit the sysctl.conf

vim /etc/sysctl.conf
[...]
net.inet.ip.multipath=1
net.inet6.ip6.multipath=1
net.inet.ip.forwarding=1
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2

Desabling the default route in the Backup Carp

Now we need to remove the /etc/mygate because we have multiples routes one for each internet link and we don't need this file. The routes is in the carp2 and carp3 in the !route add -mpath default x.x.x.x directive.

So let's remove the file

rm -rf /etc/mygate

Now restart the server to get all the configuration

reboot

Checking the Backup Carp Configuration

Now we need to check the backup carp configuration and double check if everything is working properly.

ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
        index 6 priority 0 llprio 3
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
xnf0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 5b:48:fe:25:3b:fe
        description: PHYSICAL LINK DMZ
        index 1 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 172.31.0.244 netmask 0xffffff00 broadcast 172.31.0.255
xnf1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 62:c2:35:79:13:e8
        description: PHYSICAL LINK PFSYNC
        index 2 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
xnf2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr ae:b0:4c:78:15:84
        description: PHYSICAL LINK COPEL
        index 3 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 10.10.10.2 netmask 0xffffff00 broadcast 10.10.10.255
xnf3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 78:c0:7d:76:af:10
        description: PHYSICAL LINK EMBRATEL
        index 4 priority 0 llprio 3
        media: Ethernet manual
        status: active
        inet 10.20.20.2 netmask 0xffffff00 broadcast 10.20.20.255
enc0: flags=0<>
        index 5 priority 0 llprio 3
        groups: enc
        status: active
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:01
        description: CARP LINK DMZ F245
        index 7 priority 15 llprio 3
        carp: BACKUP carpdev xnf0 vhid 1 advbase 1 advskew 10
        groups: carp
        status: backup
        inet 172.31.0.245 netmask 0xffffff00 broadcast 172.31.0.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:02
        description: CARP LINK COPEL F63
        index 8 priority 15 llprio 3
        carp: BACKUP carpdev xnf2 vhid 2 advbase 1 advskew 10
        groups: carp egress
        status: backup
        inet 200.195.146.61 netmask 0xffffffe0 broadcast 200.195.146.63
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:03
        description: CARP LINK EMBRATEL F108
        index 9 priority 15 llprio 3
        carp: BACKUP carpdev xnf3 vhid 3 advbase 1 advskew 10
        groups: carp egress
        status: backup
        inet 189.2.39.108 netmask 0xfffffff0 broadcast 189.2.39.111
pfsync0: flags=41<UP,RUNNING> mtu 1500
        index 10 priority 0 llprio 3
        pfsync: syncdev: xnf1 syncpeer: 192.168.0.1 maxupd: 128 defer: off
        groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33144
        index 11 priority 0 llprio 3
        groups: pflog

As we can see above the carp status is backup for all the carp interfaces in the Backup Server, the pfsync0 is working as well.

We can check the packets are working throughtout the pfsync0 with tcpdump

tcpdump -e -n -vv -ttt -i pfsync0
tcpdump: listening on pfsync0, link-type PFSYNC
Oct 13 16:03:27.785419 PFSYNCv6 len 360
    act UPD ST COMP count 4
        id: 57ff8b7c000002b0 creatorid: 7116b56a
    ...
Oct 13 16:03:27.785783 PFSYNCv6 len 580
    act UPD ST REQ count 2
        id: 57ff8bea000000aa creatorid: 95a434c3
        id: 57ff8bea000000a9 creatorid: 95a434c3
    act UPD ST count 2
    ...
Oct 13 16:03:28.795391 PFSYNCv6 len 360
    act UPD ST COMP count 4
        id: 57ff8b7c000002b0 creatorid: 7116b56a
    ...
Oct 13 16:03:28.795781 PFSYNCv6 len 580
    act UPD ST REQ count 2
        id: 57ff8bea000000aa creatorid: 95a434c3
        id: 57ff8bea000000a9 creatorid: 95a434c3
    act UPD ST count 2

Another way to check if the pfsync is working is double checking the pf status

pfctl -sa
[...]
STATES:
all pfsync 192.168.0.2 -> 192.168.0.1       MULTIPLE:MULTIPLE
[...]

The states of both firewall need to be the same in the states section.

Here as the firewall are sharing the information about the connections and the ip address we need to keep the pf.conf the same on both sides, here so far I cannot figure out a good daemon to do that, so let's create the file in one side and copy to the other side.

Here I will share with you guys an pf.conf with some configuration for the environment that we are planning to work, in other words with multipath routing.

You can get more information about the pf syntax in PF: User's Guide

Let's create the pf.conf on the Firewall 01

vim /etc/pf.conf
#/etc/pf.conf

### INTERFACES
#########################
# LC = LINK 01
# LE = LINK 02
# Fx = FINAL x EX: 200.200.200.x
#########################
lc_if     = "{ xnf2 }"                          # INTERNET 01 PHYSICAL INTEFACE
le_if     = "{ xnf3 }"                          # INTERNET 02 PHYSICAL INTERFACE
lc_f61    = "{ carp2 }"                         # CARP 2 -> 200.200.200.61
le_f108   = "{ carp3 }"                         # CARP 3 -> 189.189.189.108
dmz_if    = "{ xnf0 }"                          # DMZ PHYSICAL INTERFACE
ldmz_f245 = "{ carp1 }"                         # CARP 1 -> 172.31.0.245
all_if    = "{ xnf0 xnf1 xnf2 xnf3 }"           # ALL INTERFACES
sync_if   = "{ xnf1 }"                          # PFSYNC INTERFACE
carp_if   = "{ xnf0 xnf2 xnf3 }"                # CARP INTERFACES

### NETWORKS
dmz_net = "172.31.0.0/24"                               # RANGE: 172.31.0.1-172.31.0.254          GW: 172.31.0.254    BRD: 172.31.0.255      NETMASK: 255.255.255.0
lc_net  = "200.200.200.32/27"                           # RANGE: 200.200.200.33-200.200.200.62    GW: 200.200.200.33  BRD: 200.200.200.63    NETMASK: 255.255.255.224
le_net  = "189.189.189.96/28"                           # RANGE: 189.189.189.97-189.189.189.110   GW: 189.189.189.97  BRD: 189.189.189.111   NETMASK: 255.255.255.240
lcc_net = "10.10.10.0/24"                               # RANGE: 10.10.10.1-10.10.10.254          GW:                 BRD: 10.10.10.255      NETMASK: 255.255.255.0
lce_net = "10.20.20.0/24"                               # RANGE: 10.20.20.1-10.20.20.254          GW:                 BRD: 10.20.20.255      NETMASK: 255.255.255.0

### SERVERS  ###
web_srvs     = "{ 172.31.0.131 172.31.0.132 }"                  # labpfcli01, labpfcli02
mysql_srv    = "{ 172.31.0.134 }"                               # labpfcli03
pgsql_srv    = "{ 172.31.0.135 }"                               # labpfcli04
dmz_dns_srv  = "{ 172.31.0.200 172.31.0.201 172.31.0.202 }"     # labpfdns01, labpfdns02, labpfdns03
#ext_dns_srv  = "{ 8.8.8.8 8.8.4.4 }"                           # googledns1, googledns2
all_dns_srv  = "{ 172.31.0.200 172.31.0.201 8.8.8.8 8.8.4.4 }"  # labpfdns01, labpfdns02, googledns1, googledns2
ftp_srv      = "{ 172.31.0.135 }"                               # labpfcli04

### TABLES
## REPOSITORIES
table <debian_repo>     file "/etc/tables/debianrepositories"   # DEBIAN REPOSITORIES
table <centos_repo>     file "/etc/tables/centosrepositories"   # CENTOS REPOSITORIES

## NTP SERVERS
table <ntp_srv>         file "/etc/tables/ntpservers"           # EXTERNAL NTP SERVERS
table <ntp_srv_dmz>     file "/etc/tables/ntpservers_dmz"       # DMZ NTP SERVERS

### SERVICES
icmp_types = "{ echoreq unreach }"              # ICMP TYPES ALLOWED
dns_port   = "53"                               # DNS PORT
ssh_port   = "2222"                             # SSH PORT
mysql_port = "3306"                             # MYSQL PORT
pgsql_port = "5432"                             # PSGQL PORT
http_port  = "80"                               # HTTP PORT
ftp_port   = "21"                               # FTP PORT FTPPROXY
ntp_port   = "123"                              # NTP PORT
repo_ports = "{ 80 443 }"			# REPOSITORY PORTS
ftp_p_ports = "49151"				# FTP PASSIVE PORTS >= 49151
ftp_ports = "{ 20 21 }"				# FTP PORTS

### SETTING UP THE RUNTIME OPTIONS BEGIN ###
set skip on lo
set block-policy drop
set loginterface egress
set state-policy floating
### SETTING UP THE RUNTIME OPTIONS END   ###

### NORMALIZING INCOMING PACKETS     ###
match in all scrub (no-df random-id max-mss 1440)
### NORMALIZING INCOMING PACKETS END ###

### DEFAULT POLICY BEGIN ###
block log all
### DEFAULT POLICY END   ###

### ANTISPOOF BEGIN ###
antispoof for $all_if inet
### ANTISPOOF END   ###

### PFSYNC BEGIN ###
pass log quick on $sync_if proto pfsync keep state
### PFSYNC END   ###

### CARP BEGIN ###
pass log on $carp_if proto carp keep state
### CARP END ###

### SSH BEGIN ###
## DMZ TO FW
pass in log quick on $dmz_if inet proto tcp from $dmz_net to $ldmz_f245 port $ssh_port
pass in log quick on $dmz_if inet proto tcp from $dmz_net to $dmz_if    port $ssh_port
### SSH END  ###

### TRACEROUTE BEGIN ###
## FW TO ANY
pass out on $lc_if  inet proto udp from $lc_net  to any port 33433 >< 33626 keep state
pass out on $le_if  inet proto udp from $le_net  to any port 33433 >< 33626 keep state

## DMZ TO ANY
pass out log on $dmz_if inet proto udp from $dmz_net to any port 33433 >< 33626 keep state
pass in  log on $dmz_if inet proto udp from $dmz_net to any port 33433 >< 33626 keep state
### TRACEROUTE END ###

### ICMP BEGIN ###
## FW TO ANY
pass out log inet proto icmp from $lc_net to any  icmp-type $icmp_types keep state
pass in  log inet proto icmp from any to $lc_net  icmp-type $icmp_types keep state
pass out log inet proto icmp from $le_net to any  icmp-type $icmp_types keep state
pass in  log inet proto icmp from any to $le_net  icmp-type $icmp_types keep state
pass out log inet proto icmp from $lcc_net to any icmp-type $icmp_types keep state
pass in  log inet proto icmp from any to $lcc_net icmp-type $icmp_types keep state
pass out log inet proto icmp from $lce_net to any icmp-type $icmp_types keep state
pass in  log inet proto icmp from any to $lce_net icmp-type $icmp_types keep state

## DMZ TO ANY
pass in log on $dmz_if inet proto icmp from $dmz_net to any icmp-type $icmp_types keep state
pass in log on $dmz_if inet proto icmp from $dmz_net to any icmp-type $icmp_types keep state
### ICMP END   ###

### DNS BEGIN ###
## FW TO EXTERNAL AND DMZ DNS SERVERS
pass out log on $lc_if inet proto udp from $lc_net to $all_dns_srv port $dns_port keep state
pass out log on $le_if inet proto udp from $le_net to $all_dns_srv port $dns_port keep state

### DNS SERVER TO EXTERNAL
pass out log on $dmz_if inet proto udp from $dmz_dns_srv to any port $dns_port keep state
pass in  log on $dmz_if inet proto udp from $dmz_dns_srv to any port $dns_port keep state
pass out log on $dmz_if inet proto tcp from $dmz_dns_srv to any port $dns_port keep state
pass in  log on $dmz_if inet proto tcp from $dmz_dns_srv to any port $dns_port keep state

## DMZ TO EXTERNAL AND DMZ SERVERS
pass out log on $dmz_if inet proto udp from $dmz_net to $all_dns_srv port $dns_port keep state
pass in  log on $dmz_if inet proto udp from $dmz_net to $all_dns_srv port $dns_port keep state
### DNS END   ###

### NTP BEGIN ###
## FW TO EXTERNAL NTP SERVERS
pass out log on $lc_if inet proto udp from $lc_net to { <ntp_srv>, <ntp_srv_dmz> } port $ntp_port keep state
pass out log on $le_if inet proto udp from $le_net to { <ntp_srv>, <ntp_srv_dmz> } port $ntp_port keep state

## DMZ TO NTP SERVERS
pass out log on $dmz_if inet proto udp from $dmz_net to <ntp_srv_dmz> port $ntp_port keep state
pass in  log on $dmz_if inet proto udp from $dmz_net to <ntp_srv_dmz> port $ntp_port keep state
### NTP END   ###

### REPOSITORIES BEGIN ###
pass out log on $dmz_if inet proto tcp from $dmz_net to <debian_repo> port $http_port keep state
pass in  log on $dmz_if inet proto tcp from $dmz_net to <debian_repo> port $http_port keep state
pass out log on $dmz_if inet proto tcp from $dmz_net to <centos_repo> port $repo_ports keep state
pass in  log on $dmz_if inet proto tcp from $dmz_net to <centos_repo> port $repo_ports keep state
anchor "ftp-proxy/*"
pass in quick on $dmz_if inet proto tcp to port $ftp_port divert-to 127.0.0.1 port 8021
pass out inet proto tcp from (self) to <debian_repo> port $ftp_port
pass out inet proto tcp from (self) to <centos_repo> port $ftp_port
### REPOSITORIES END   ###

### MYSQL BEGIN ###
## REDIRECT TO MSSQL SERVER labpfcli03
pass in  log on $lc_if  proto tcp to $lc_f61 port 33060 rdr-to $mysql_srv port $mysql_port
pass out log on $dmz_if proto tcp to $mysql_srv port $mysql_port
### MYSQL END   ###

### POSTGRESQL BEGIN ###
## REDIRECT TO PGSQL SERVER labpfcli04
pass in  log on $le_if  proto tcp to $le_f108 port 54320 rdr-to $pgsql_srv port $pgsql_port
pass out log on $dmz_if proto tcp to $pgsql_srv port $pgsql_port
### POSTGRESQL END   ###

### FTP SERVER BEGIN ###
## REDIRECT TO FTP SERVER labpfcli04
pass in  log on $le_if  proto tcp to   $le_f108 port   $ftp_ports   rdr-to $ftp_srv
pass in  log on $le_if  proto tcp to   $le_f108 port > $ftp_p_ports rdr-to $ftp_srv
pass out log on $dmz_if proto tcp to   $ftp_srv port   $ftp_ports
pass out log on $dmz_if proto tcp from $ftp_srv port   $ftp_ports to any
pass in  log on $dmz_if proto tcp from $ftp_srv port   $ftp_ports to any
pass out log on $dmz_if proto tcp to   $ftp_srv port > $ftp_p_ports

### FTP SERVER END   ###

### POOL WEB SERVERS BEGIN ###
## REDIRECT TO POOL WEB SERVERS labpfcli01, labpfcli02
pass in  log on $lc_if  proto tcp to port $http_port rdr-to $web_srvs round-robin sticky-address
pass in  log on $le_if  proto tcp to port $http_port rdr-to $web_srvs round-robin sticky-address
pass out log on $dmz_if proto tcp to $web_srvs port $http_port
### POOL WEB SERVERS END   ###

### NAT ALL THE DMZ NET BEGIN ###
pass out log inet from { $dmz_net } to any nat-to (egress)
### NAT ALL THE DMZ NET END   ###

Now we need to enable the ftpproxy that will help the pf to create dynamic rules to pass the ftp clients connections, you can get more information about the ftpproxy in Issues with FTP

Now we need to add it into /etc/rc.conf.local to set up the flags to initialize the daemon of ftpproxy.

vim /etc/rc.conf.local
ftpproxy_flags="-D7 -v"

Now we need to enable the ftpproxy and start it.

rcctl enable ftpproxy
rcctl start  ftpproxy

Now we need to create the directory to store the files with the tables

mkdir -p /etc/tables

Now let's change the permission of the directory

chmod 750 /etc/tables

Note: Here I will use the ip address rather than the dns name because will boost the speed in the processing the tables without resolving the dns name.

Now we need to create the files with the content, the first file will be the CentOS repositories file

vim /etc/tables/centosrepositories
# http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
# http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras
# http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
# http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=centosplus
# http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=fasttrack
# https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64
# mirrorlist.centos.org
85.236.43.108
212.69.166.138
216.176.179.218
67.219.148.138
# centos.brnet.net.br
177.124.188.233
# centos.brisanet.com.br
177.37.220.74
# mirror.facom.ufms.br
200.129.206.120
# mirror.globo.com
131.0.25.51
# centos.xpg.com.br
187.17.123.242
# mirror.nbtelecom.com.br
189.45.5.90
# centos.ufes.br
200.137.64.198
# mirrors.uprm.edu
136.145.216.245
# mirror.orbyta.com
190.196.215.59
# mirror.gtdinternet.com
190.196.123.25
# linorg.usp.br
200.144.183.235
# mirror.bytemark.co.uk
212.110.161.69
# mirrors.coreix.net
85.13.241.50
# ftp.free.fr
212.27.60.27
# ftp.hosteurope.de
80.237.136.138
# mirror.as24220.net
116.66.162.254
# centos.mirror.iweb.ca
192.175.120.169
# mirror.steadfast.net
208.100.4.53
# mirror.centos.org
187.45.181.183
# ftp.jaist.ac.jp
150.65.7.130
# mirror.us.leaseweb.net
108.59.10.97
# mirror.vtti.vt.edu
198.82.152.116
# mirror.us.oneandone.net
74.208.4.167
# mirror.steadfast.net
208.100.4.53
# vault.centos.org -> CentOS Sources
160.10.26.25
# mirror.uta.edu.ec
200.93.227.165
# epel.gtdinternet.com
190.196.123.25
# apps.fedoraproject.org
209.132.181.16
152.19.134.142
140.211.169.206
67.219.144.68
174.141.234.172
152.19.134.198
209.132.181.15
140.211.169.196
185.141.165.254
# pkgs.repoforge.org
78.46.17.228

Now we need to create the file that will store the information about the Debian repositories

vim /etc/tables/debianrepositories
# ftp.br.debian.org
200.236.31.3
# security.debian.org
200.17.202.197
# www.debian-multimedia.org
169.47.15.77
# packages.dotdeb.org
195.154.242.153
# nginx.org
206.251.255.63
95.211.80.227
# apt.dockerproject.org
52.84.170.153

Now we need to create the file that will store the information about the public NTP servers

vim /etc/tables/ntpservers
# http://www.pool.ntp.org/zone/br
# a.ntp.br
200.160.0.8
# 0.br.pool.ntp.org
200.186.125.195
200.229.193.194
# 1.br.pool.ntp.org
192.155.90.13
200.160.0.8
# 2.br.pool.ntp.org
200.160.7.193
200.189.40.8
# 3.br.pool.ntp.org
200.192.232.8
192.99.2.8

Now we need to create the file that will store the information about the DMZ NTP servers

vim /etc/tables/ntpservers_dmz
# labpfntp01.dqs.local
172.31.0.210
# labpfntp02.dqs.local
172.31.0.211

Note: Don't forget to change the ip address and the port numbers to fit your needs.

Now we can start the pf but before it we need to test the file configuration

pfctl -nvvf /etc/pf.conf
Loaded 710 passive OS fingerprints
lc_if = "{ xnf2 }"
le_if = "{ xnf3 }"
lc_f61 = "{ carp2 }"
le_f108 = "{ carp3 }"
dmz_if = "{ xnf0 }"
ldmz_f245 = "{ carp1 }"
all_if = "{ xnf0 xnf1 xnf2 xnf3 }"
sync_if = "{ xnf1 }"
carp_if = "{ xnf0 xnf2 xnf3 }"
dmz_net = "172.31.0.0/24"
lc_net = "200.200.200.32/27"
le_net = "189.189.189.96/28"
lcc_net = "10.10.10.0/24"
lce_net = "10.20.20.0/24"
web_srvs = "{ 172.31.0.131 172.31.0.132 }"
mysql_srv = "{ 172.31.0.202 }"
pgsql_srv = "{ 172.31.0.135 }"
dmz_dns_srv = "{ 172.31.0.200 172.31.0.201 172.31.0.202 }"
ext_dns_srv = "{ 8.8.8.8 8.8.4.4 }"
all_dns_srv = "{ 172.31.0.200 172.31.0.201 8.8.8.8 8.8.4.4 }"
ftp_srv = "{ 172.31.0.135 }"
table <debian_repo> file "/etc/tables/debianrepositories"
table <centos_repo> file "/etc/tables/centosrepositories"
table <ntp_srv> file "/etc/tables/ntpservers"
table <ntp_srv_dmz> file "/etc/tables/ntpservers_dmz"
icmp_types = "{ echoreq unreach }"
dns_port = "53"
ssh_port = "22022"
mysql_port = "3306"
pgsql_port = "5432"
http_port = "80"
ftp_port = "21"
ntp_port = "123"
repo_ports = "{ 80 443 }"
ftp_p_ports = "49151"
ftp_ports = "{ 20 21 }"
set skip on { lo }
set block-policy drop
set loginterface egress
set state-policy floating
table <__automatic_0> const { 172.31.0.131 172.31.0.132 }
table <__automatic_1> const { 172.31.0.131 172.31.0.132 }
@0 match in all scrub (no-df random-id max-mss 1440)
@1 block drop log all
@2 block drop in on ! xnf0 inet from 172.31.0.0/24 to any
@3 block drop in inet from 172.31.0.243 to any
@4 block drop in on ! xnf1 inet from 192.168.0.0/24 to any
@5 block drop in inet from 192.168.0.1 to any
@6 block drop in on ! xnf2 inet from 10.10.10.0/24 to any
@7 block drop in inet from 10.10.10.1 to any
@8 block drop in on ! xnf3 inet from 10.20.20.0/24 to any
@9 block drop in inet from 10.20.20.1 to any
@10 pass log quick on xnf1 proto pfsync all
@11 pass log on xnf0 proto carp all
@12 pass log on xnf2 proto carp all
@13 pass log on xnf3 proto carp all
@14 pass in log quick on xnf0 inet proto tcp from 172.31.0.0/24 to 172.31.0.245 port = 22022 flags S/SA
@15 pass in log quick on xnf0 inet proto tcp from 172.31.0.0/24 to 172.31.0.243 port = 22022 flags S/SA
@16 pass out on xnf2 inet proto udp from 200.200.200.32/27 to any port 33433 >< 33626
@17 pass out on xnf3 inet proto udp from 189.189.189.96/28 to any port 33433 >< 33626
@18 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to 172.31.0.200 port = 53
@19 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to 172.31.0.201 port = 53
@20 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to 8.8.8.8 port = 53
@21 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to 8.8.4.4 port = 53
@22 pass out log on xnf0 inet proto udp from 172.31.0.200 to any port = 53
@23 pass out log on xnf0 inet proto udp from 172.31.0.201 to any port = 53
@24 pass out log on xnf0 inet proto udp from 172.31.0.202 to any port = 53
@25 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to 172.31.0.200 port = 53
@26 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to 172.31.0.201 port = 53
@27 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to 8.8.8.8 port = 53
@28 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to 8.8.4.4 port = 53
@29 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to 172.31.0.200 port = 53
@30 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to 172.31.0.201 port = 53
@31 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to 8.8.8.8 port = 53
@32 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to 8.8.4.4 port = 53
@33 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to <ntp_srv_dmz:0> port = 123
@34 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to <ntp_srv_dmz:0> port = 123
@35 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to <ntp_srv_dmz:0> port = 123
@36 pass out log on xnf2 inet proto udp from 200.200.200.32/27 to <ntp_srv:0> port = 123
@37 pass out log on xnf3 inet proto udp from 189.189.189.96/28 to <ntp_srv:0> port = 123
@38 pass out log on xnf0 inet proto udp from 172.31.0.0/24 to any port 33433 >< 33626
@39 pass out log inet proto icmp from 200.200.200.32/27 to any icmp-type echoreq
@40 pass out log inet proto icmp from 200.200.200.32/27 to any icmp-type unreach
@41 pass out log inet proto icmp from 189.189.189.96/28 to any icmp-type echoreq
@42 pass out log inet proto icmp from 189.189.189.96/28 to any icmp-type unreach
@43 pass out log inet proto icmp from 10.10.10.0/24 to any icmp-type echoreq
@44 pass out log inet proto icmp from 10.10.10.0/24 to any icmp-type unreach
@45 pass out log inet proto icmp from 10.20.20.0/24 to any icmp-type echoreq
@46 pass out log inet proto icmp from 10.20.20.0/24 to any icmp-type unreach
@47 pass out log on xnf0 inet proto tcp from 172.31.0.0/24 to <centos_repo:0> port = 80 flags S/SA
@48 pass out log on xnf0 inet proto tcp from 172.31.0.0/24 to <centos_repo:0> port = 443 flags S/SA
@49 pass out log on xnf0 inet proto tcp from 172.31.0.0/24 to <debian_repo:0> port = 80 flags S/SA
@50 pass out log on xnf0 inet proto tcp from 172.31.0.200 to any port = 53 flags S/SA
@51 pass out log on xnf0 inet proto tcp from 172.31.0.201 to any port = 53 flags S/SA
@52 pass out log on xnf0 inet proto tcp from 172.31.0.202 to any port = 53 flags S/SA
@53 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to 172.31.0.200 port = 53
@54 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to 172.31.0.201 port = 53
@55 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to 8.8.8.8 port = 53
@56 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to 8.8.4.4 port = 53
@57 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to any port 33433 >< 33626
@58 pass in log on xnf0 inet proto udp from 172.31.0.0/24 to <ntp_srv_dmz:0> port = 123
@59 pass in log on xnf0 inet proto tcp from 172.31.0.0/24 to <centos_repo:0> port = 80 flags S/SA
@60 pass in log on xnf0 inet proto tcp from 172.31.0.0/24 to <centos_repo:0> port = 443 flags S/SA
@61 pass in log on xnf0 inet proto tcp from 172.31.0.0/24 to <debian_repo:0> port = 80 flags S/SA
@62 pass in log on xnf0 inet proto icmp from 172.31.0.0/24 to any icmp-type echoreq
@63 pass in log on xnf0 inet proto icmp from 172.31.0.0/24 to any icmp-type unreach
@64 pass in log on xnf0 inet proto udp from 172.31.0.200 to any port = 53
@65 pass in log on xnf0 inet proto udp from 172.31.0.201 to any port = 53
@66 pass in log on xnf0 inet proto udp from 172.31.0.202 to any port = 53
@67 pass in log on xnf0 inet proto tcp from 172.31.0.200 to any port = 53 flags S/SA
@68 pass in log on xnf0 inet proto tcp from 172.31.0.201 to any port = 53 flags S/SA
@69 pass in log on xnf0 inet proto tcp from 172.31.0.202 to any port = 53 flags S/SA
@70 pass in log inet proto icmp from any to 200.200.200.32/27 icmp-type echoreq
@71 pass in log inet proto icmp from any to 200.200.200.32/27 icmp-type unreach
@72 pass in log inet proto icmp from any to 189.189.189.96/28 icmp-type echoreq
@73 pass in log inet proto icmp from any to 189.189.189.96/28 icmp-type unreach
@74 pass in log inet proto icmp from any to 10.10.10.0/24 icmp-type echoreq
@75 pass in log inet proto icmp from any to 10.10.10.0/24 icmp-type unreach
@76 pass in log inet proto icmp from any to 10.20.20.0/24 icmp-type echoreq
@77 pass in log inet proto icmp from any to 10.20.20.0/24 icmp-type unreach
@78 anchor "ftp-proxy/*" all
@79 pass in quick on xnf0 inet proto tcp from any to any port = 21 flags S/SA divert-to 127.0.0.1 port 8021
@80 pass out inet proto tcp from (self:*) to <debian_repo:0> port = 21 flags S/SA
@81 pass out inet proto tcp from (self:*) to <centos_repo:0> port = 21 flags S/SA
@82 pass in log on xnf2 inet proto tcp from any to 200.200.200.61 port = 33060 flags S/SA rdr-to 172.31.0.202 port 3306
@83 pass out log on xnf0 inet proto tcp from any to 172.31.0.202 port = 3306 flags S/SA
@84 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 54320 flags S/SA rdr-to 172.31.0.135 port 5432
@85 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 5432 flags S/SA
@86 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 20 flags S/SA rdr-to 172.31.0.135
@87 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 21 flags S/SA rdr-to 172.31.0.135
@88 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port > 49151 flags S/SA rdr-to 172.31.0.135
@89 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 20 flags S/SA
@90 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 21 flags S/SA
@91 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port > 49151 flags S/SA
@92 pass out log on xnf0 inet proto tcp from 172.31.0.135 port = 20 to any flags S/SA
@93 pass out log on xnf0 inet proto tcp from 172.31.0.135 port = 21 to any flags S/SA
@94 pass in log on xnf0 inet proto tcp from 172.31.0.135 port = 20 to any flags S/SA
@95 pass in log on xnf0 inet proto tcp from 172.31.0.135 port = 21 to any flags S/SA
@96 pass in log on xnf2 inet proto tcp from any to any port = 80 flags S/SA rdr-to <__automatic_0:0> round-robin sticky-address
@97 pass in log on xnf3 inet proto tcp from any to any port = 80 flags S/SA rdr-to <__automatic_1:0> round-robin sticky-address
@98 pass out log on xnf0 inet proto tcp from any to 172.31.0.131 port = 80 flags S/SA
@99 pass out log on xnf0 inet proto tcp from any to 172.31.0.132 port = 80 flags S/SA
@100 pass out log inet from 172.31.0.0/24 to any flags S/SA nat-to (egress:*) round-robin

Now as we don't get any problems with our configuration we can load it.

pfctl -f /etc/pf.conf

If you don't remember that is inside a table you can check it out with the following command

pfctl -t debian_repo -T show
   52.84.170.153
   95.211.80.227
   169.47.15.77
   195.154.242.153
   200.17.202.197
   200.236.31.3
   206.251.255.63

If you want to list all the rules

pfctl -vv -sr
@0 match in all scrub (no-df random-id max-mss 1440)
  [ Evaluations: 166       Packets: 3         Bytes: 124         States: 1     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@1 block drop log all
  [ Evaluations: 166       Packets: 141       Bytes: 16396       States: 0     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@2 block drop in on ! xnf0 inet from 172.31.0.0/24 to any
  [ Evaluations: 166       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@3 block drop in inet from 172.31.0.243 to any
  [ Evaluations: 143       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@4 block drop in on ! xnf1 inet from 192.168.0.0/24 to any
  [ Evaluations: 142       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@5 block drop in inet from 192.168.0.1 to any
  [ Evaluations: 142       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
@6 block drop in on ! xnf2 inet from 10.10.10.0/24 to any
  [ Evaluations: 142       Packets: 0         Bytes: 0           States: 0     ]
  [...]

If you want to check the states of the connections

pfctl -vv -s state
all pfsync 192.168.0.1 -> 192.168.0.2       MULTIPLE:MULTIPLE
   age 23:51:44, expires in 00:01:00, 173059:172953 pkts, 96057592:79725872 bytes
   id: 57ff8bea00000004 creatorid: 95a434c3
all tcp 172.31.0.244:22022 <- 172.31.0.250:51356       ESTABLISHED:ESTABLISHED
   [2698758666 + 130976] wscale 3  [202785879 + 16952] wscale 5
   age 23:52:50, expires in 22:51:53, 0:0 pkts, 0:0 bytes
   id: 57ff8b7c000001eb creatorid: 7116b56a
all carp 200.195.146.61 (172.31.0.243) -> 224.0.0.18       SINGLE:NO_TRAFFIC
   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
   id: 57ff8bea000000a6 creatorid: 95a434c3
all carp 10.10.10.1 -> 224.0.0.18       SINGLE:NO_TRAFFIC
   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
   id: 57ff8bea000000a7 creatorid: 95a434c3
all carp 10.20.20.1 -> 224.0.0.18       SINGLE:NO_TRAFFIC
   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
   id: 57ff8bea000000a8 creatorid: 95a434c3
   [...]

Now we can copy the pf.conf and the /etc/tables with its content to the Firewall 02 and launch the pfctl and start the script.

Checking the PF Log

Sometimes we have a hard time trying to figure out what is happining with a bunch of rules or a client that does not have the properly access, so we need a way to get information about the rules and we can check it out with the pflog0 interface or with /var/log/pflog

We can get more information about it in: PF: Logging

Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:

tcpdump -n -e -ttt -r /var/log/pflog port 80

This can be further refined by limiting the display of packets to a certain host and port combination:

tcpdump -n -e -ttt -r /var/log/pflog port 80 and host 172.31.0.220

The same idea can be applied when reading from the pflog0 interface:

tcpdump -n -e -ttt -i pflog0 host 172.31.0.221

Note that this has no impact on which packets are logged to the pflogd log file; the above commands only display packets as they are being logged.

In addition to using the standard tcpdump(8) filter rules, the tcpdump filter language has been extended for reading pflogd output:

  • ip - address family is IPv4.
  • ip6 - address family is IPv6.
  • on int - packet passed through the interface int.
  • ifname int - same as on int.
  • ruleset name - the ruleset/anchor that the packet was matched in.
  • rulenum num - the filter rule that the packet matched was rule number num.
  • action act - the action taken on the packet. Possible actions are pass and block.
  • reason res - the reason that action was taken. Possible reasons are match, bad-offset, fragment, short, normalize, memory, bad-timestamp, congestion, ip-option, proto-cksum, state-mismatch, state-insert, state-limit, src-limit and synproxy.
  • inbound - packet was inbound.
  • outbound - packet was outbound.

Example:

tcpdump -n -e -ttt -i pflog0 inbound and action block and on xnf0

This display the log, in real-time, of inbound packets that were blocked on the xnf0 interface.

Now we need to configure the ifstated to check if the internet link is working and if we have some issue with the link such as stop working for some reason or another we need to disable the route and keep routing the the good internet link until the link come back.

The file does not exists so we need to create it.

vim /etc/ifstated.conf
#/etc/ifstated.conf
# net.inet.carp.preempt must be enabled (set to 1) for this to work correctly.

# LCC -> INTERNET LINK 01
lcc_up = "carp2.link.up"

# LCE -> INTERNET LINK 02
lce_up = "carp3.link.up"

### SETTING UP THE INICIAL STATE
init-state all_ok

# CHECKING IF THE INTERNET LINK 01 IS WORKING AND REACHING THE GATEWAY
xnf2_ok = '( "ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null" every 10 )'

# CHECKING IF THE INTERNET LINK 02 IS WORKING AND REACHING THE GATEWAY
xnf3_ok = '( "ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null" every 10 )'

### RULES OF LINK STATE
state all_ok {
    # CHECK IF THE LINK 01 IS NOT WORKING AND THE CARP IS BACKUP
    if ! $xnf2_ok && ! $lcc_up {
        # CALL THE xnf2_down FUNCTION AND CLEAN UP THE ROUTES
        set-state xnf2_down
    }

    # CHECK IF THE LINK 01 IS NOT WORKING AND THE CARP IS BACKUP
    if ! $xnf3_ok && ! $lce_up {
        # CALL THE xnf3_down FUNCTION AND CLEAN UP THE ROUTES
        set-state xnf3_down
    }
}


# STATE TO SET THE LINK 01 AS DOWN
state xnf2_down {
    # INITIAL CALL
    init {
      # CALL THE COMMAND TO START THE FUNCTION. SO CLEANING UP THE ROUTES
      run "route flush -iface xnf2; pfctl -k label -k out_xnf2"
    }
    # CHECK IF THE CARP IS MASTER AND THE LINK IS OK AGAIN
    if $lcc_up && $xnf2_ok {
        # IF THE LINK IS OK AGAIN RESTORE THE ROUTES
        run "sh /etc/netstart xnf2"
        # CALL THE CHECK STATE AGAIN
        set-state all_ok
    }
}

# STATE TO SET THE LINK 02 AS DOWN
state xnf3_down {
    # INITIAL CALL
    init {
      # CALL THE COMMAND TO START THE FUNCTION. SO CLEANING UP THE ROUTES
      run "route flush -iface xnf3; pfctl -k label -k out_xnf3"
    }
    # CHECK IF THE CARP IS MASTER AND THE LINK IS OK AGAIN
    if $lce_up && $xnf3_ok {
        # IF THE LINK IS OK AGAIN RESTORE THE ROUTES
        run "sh /etc/netstart xnf3"
        # CALL THE CHECK STATE AGAIN
        set-state all_ok
    }
}

Now we can star the check and make sure if everything is working.

Here I will start the daemon manually to check if everything is working as we need.

ifstated -dvv
lcc_up = "carp2.link.up"
lce_up = "carp3.link.up"
xnf2_ok = "( "ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null" every 10 )"
xnf3_ok = "( "ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null" every 10 )"
initial state: all_ok
changing state to all_ok
running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
started
running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null

Now we need to unplug the cable of the link 01 and check the logs

running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
ping: sendto: Network is unreachable
changing state to xnf2_down
running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
ping: sendto: Network is unreachable
running route flush -iface xnf2; pfctl -k label -k out_xnf2
killed 0 states

As we can see the link 01 is not working so the route was cleaning up and the ifstated will keep checking the status.

Now plug the cable again and check the logs

running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
running sh /etc/netstart xnf2
changing state to all_ok
running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null

As we can see when the ifstated figure out the link up again it will bring back the link and re-establish the multipath again.

Now we can press ctrl + c and stop the ifstated now we need to add it to the rc.conf.local

vim /etc/rc.conf.local
ftpproxy_flags="-D7 -v"
ifstated_flags=""

Now we need to put the ifstated in the boot time and start it

rcctl enable ifstated
rcctl start ifstated

Now you can do the same process in the Firewall 02.

References