Differences

This shows you the differences between two versions of the page.

Link to this comparison view

equal_cost_multipath_routing_with_ifstated_carp_and_pfsync_on_openbsd_en [2017/09/05 12:18] (current)
Line 1: Line 1:
 +====== Equal-cost multipath routing with Ifstated, Carp and PFsync on OpenBSD ======
  
 +Hey guys, here I will show you how to configure equal-cost multipath routing with some another features that are pretty interesting working together.
 +
 +The multipath routing is a great feature but we have a problem when one of the internet links goes down, the multipath will send the packets to both internet links the good one and the another one that went down, so with that we need another daemon to take care of it and provide the knowledge about the dead internet link, so we can keep the conection throught the good link and when the dead link come alive we can active it and bring back the multipah.
 +
 +But if the firewall goes down for one reason or another we will lost the entire connection with the internet with that we will configure the carp to guarantee if one firewall server goes down we have the other one to assume all the functions of the dead firewall server and keep to provide the network the servers with the same features that was give by the dead one.
 +
 +The pfsync will keep track of the all state of the firewall and will replicate to the second firewall so even the connections will be keep in the second firewall no impacting the clients.
 +
 +**The problem in the environment:​**
 +
 +In this scenario we only have one ip address of each internet link, so we need to configure a private network range in each interface that will be used by the carp as internet link with it we can use the public ip address in the master carp. So for example we have 10.10.10.0/​24 as the physical ip address range for the internet link and the 200.200.200.0/​28 as the carp ip address range that will be visible by the internet. So the Master carp always will have the valid public ip address and the backup carp will have the same ip in standby waiting to become master and assume the new public ip address without any problem with duplicate of ip address.
 +
 +
 +
 +===== Enviroment used here =====
 +
 +  * **Firewall 01**
 +    * **OS Version:** OpenBSD 6.0
 +    * **DMZ IP:** 172.31.0.243/​24
 +    * **DMZ Interface:​** xnf0
 +    * **CARP DMZ:** 172.31.0.245/​24
 +      * **CARP DMZ INTERFACE:​** carp1
 +      * **Status:** Master
 +      * **vhid:** 1
 +      * **carpdev:​** xnf0
 +      * **Advskew:​** 0
 +    * **PFSync:** 192.168.0.1/​24
 +      * **PFSync Interface:​** xnf1
 +      * **syncdev:​** xnf1
 +      * **syncpeer:​** 192.168.0.2
 +    * **INTERNET LINK 01:** 10.10.10.1/​24
 +    * **INTERNET LINK 01 Interface:​** xnf2
 +    * **CARP INTERNET LINK 01:** 200.200.200.61/​27
 +      * **CARP INTERNET Interface 01:** carp2
 +      * **Status:** Master
 +      * **vhid:** 2
 +      * **carpdev:​** xnf2
 +      * **Advskew:​** 0
 +    * **INTERNET LINK 02:** 10.20.20.1/​24
 +    * **INTERNET LINK 02 Interface:​** xnf3
 +    * **CARP INTERNET LINK 02:** 189.189.189.108/​28
 +      * **CARP INTERNET Interface 02:** carp3
 +      * **Status:** Master
 +      * **vhid:** 3
 +      * **carpdev:​** xnf3
 +      * **Advskew:​** 0
 +
 +  * **Firewall 02**
 +    * **OS Version:** OpenBSD 6.0
 +    * **DMZ IP:** 172.31.0.244/​24
 +    * **DMZ Interface:​** xnf0
 +    * **CARP DMZ:** 172.31.0.245/​24
 +      * **CARP DMZ INTERFACE:​** carp1
 +      * **Status:** Backup
 +      * **vhid:** 1
 +      * **carpdev:​** xnf0
 +      * **Advskew:​** 10
 +    * **PFSync:** 192.168.0.2/​24
 +      * **PFSync Interface:​** xnf1
 +      * **syncdev:​** xnf1
 +      * **syncpeer:​** 192.168.0.1
 +    * **INTERNET LINK 01:** 10.10.10.2/​24
 +    * **INTERNET LINK 01 Interface:​** xnf2
 +    * **CARP INTERNET LINK 01:** 200.200.200.61/​27
 +      * **CARP INTERNET Interface 01:** carp2
 +      * **Status:** Backup
 +      * **vhid:** 2
 +      * **carpdev:​** xnf2
 +      * **Advskew:​** 10
 +    * **INTERNET LINK 02:** 10.20.20.2/​24
 +    * **INTERNET LINK 02 Interface:​** xnf3
 +    * **CARP INTERNET LINK 02:** 189.189.189.108/​28
 +      * **CARP INTERNET Interface 02:** carp3
 +      * **Status:** Backup
 +      * **vhid:** 3
 +      * **carpdev:​** xnf3
 +      * **Advskew:​** 10
 +
 +===== Configuring the Carp Master Network Interfaces =====
 +
 +Here we will configure the interfaces that will be used by the master carp server.
 +
 +Note: I have no ideia why and how spaces matter in the interfaces configuration,​ so don't put more spaces than need. Ex: param + space + another_param and so on, otherwise you will have some pass throughout some troubles that can not be explained.
 +
 +==== Carp Master DMZ Interface ====
 +
 +Let's configure the physical interface with the real ip address that will be used to access the server throughtout the DMZ Network.
 +<sxh bash>
 +vim /​etc/​hostname.xnf0
 +inet 172.31.0.243 255.255.255.0 172.31.0.255 description "​PHYSICAL LINK DMZ"
 +</​sxh>​
 +
 +
 +Let's configure the carp DMZ Interface that will be used by the DMZ clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. This ip address will be used as gateway of the servers in the DMZ Network.
 +<sxh bash>
 +vim /​etc/​hostname.carp1
 +inet 172.31.0.245 255.255.255.0 172.31.0.255 vhid 1 carpdev xnf0 pass lanpasswd advskew 0 description "CARP LINK DMZ F245"
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +==== Carp Master PFSync Interface ====
 +
 +Let's configure the physical interface with the real ip address that will be used to change information about the pfsync between the firewalls.
 +<sxh bash>
 +vim /​etc/​hostname.xnf1
 +inet 192.168.0.1 255.255.255.0 192.168.0.255 description "​PHYSICAL LINK PFSYNC"​
 +</​sxh>​
 +
 +Now let's configure the virtual pfsync interface that keeps the information about the another peer of the pfsync configuration.
 +
 +**Note:** If you are using some kind of virtual enviroment you need to create a private interface that can be used only between the the two firewalls, because the information that will be exchange between the servers are no encrypted.
 +<sxh bash>
 +vim /​etc/​hostname.pfsync0
 +up syncdev xnf1 syncpeer 192.168.0.2
 +</​sxh>​
 +
 +You can get more information about the pfsync configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​pfsync.4|pfsync(4)]] and [[http://​man.openbsd.org/​ifconfig.8|ifconfig(8)]]
 +==== Carp Master Internet Link 01 Interface ====
 +
 +Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.
 +<sxh bash>
 +vim /​etc/​hostname.xnf2
 +inet 10.10.10.1 255.255.255.0 10.10.10.255 description "​PHYSICAL LINK 01"
 +</​sxh>​
 +
 +Let's configure the carp Internet Link 01 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. ​
 +
 +**Note:** This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.
 +
 +<sxh bash>
 +vim /​etc/​hostname.carp2
 +inet 200.200.200.61 255.255.255.224 200.200.200.63 vhid 2 carpdev xnf2 pass lanpasswd advskew 0 description "CARP LINK 01 F63"
 +!route add -mpath default 200.200.200.33
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +==== Carp Master Internet Link 02 Interface ====
 +
 +Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.
 +<sxh bash>
 +vim /​etc/​hostname.xnf2
 +inet 10.20.20.1 255.255.255.0 10.20.20.255 description "​PHYSICAL LINK 02"
 +</​sxh>​
 +
 +
 +Let's configure the carp Internet Link 02 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. ​
 +
 +**Note:** This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.
 +
 +<sxh bash>
 +vim /​etc/​hostname.carp3
 +inet 189.189.189.108 255.255.255.240 189.189.189.111 vhid 3 carpdev xnf3 pass lanpasswd advskew 0 description "CARP LINK 02 F108"
 +!route add -mpath default 189.189.189.97
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +
 +===== Carp Master Kernel Configuration =====
 +
 +Now we need to enable the equal cost multipath routing into the kernel. You can get more information about the multipath routing configuration in [[http://​www.openbsd.org/​faq/​faq6.html#​Multipath|Multipath]]
 +<sxh bash>
 +sysctl net.inet.ip.multipath=1
 +sysctl net.inet6.ip6.multipath=1
 +</​sxh>​
 +
 +We can enable the forward routing to act as a gateway. ​
 +<sxh bash>
 +sysctl net.inet.ip.forwarding=1
 +</​sxh>​
 +
 +Now we need to configure the kernel to work with the carp.  You can get more information about the carp configuration in [[http://​www.openbsd.org/​faq/​pf/​carp.html|carp]]
 +<sxh bash>
 +sysctl net.inet.carp.allow=1
 +sysctl net.inet.carp.preempt=1
 +sysctl net.inet.carp.log=2
 +</​sxh>​
 +
 +Now we need to keep this configuration in the boot time, so let's edit the sysctl.conf
 +<sxh bash>
 +vim /​etc/​sysctl.conf
 +[...]
 +net.inet.ip.multipath=1
 +net.inet6.ip6.multipath=1
 +net.inet.ip.forwarding=1
 +net.inet.carp.allow=1
 +net.inet.carp.preempt=1
 +net.inet.carp.log=2
 +</​sxh>​
 +
 +==== Desabling the default route in the Master Carp ====
 +
 +Now we need to remove the **/​etc/​mygate** because we have multiples routes one for each internet link and we don't need this file. The routes is in the carp2 and carp3 in the **!route add -mpath default x.x.x.x** directive.
 +
 +So let's remove the file
 +<sxh bash>
 +rm -rf /etc/mygate
 +</​sxh>​
 +
 +Now restart the server to get all the configuration
 +<sxh bash>
 +reboot
 +</​sxh>​
 +
 +==== Checking the Master Carp Configuration ====
 +
 +Now we need to check the master carp configuration and double check if everything is working properly.
 +<sxh bash>
 +ifconfig
 +lo0: flags=8049<​UP,​LOOPBACK,​RUNNING,​MULTICAST>​ mtu 32768
 +        index 6 priority 0 llprio 3
 +        groups: lo
 +        inet6 ::1 prefixlen 128
 +        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
 +        inet 127.0.0.1 netmask 0xff000000
 +xnf0: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 4e:​00:​9e:​30:​0a:​e3
 +        description:​ PHYSICAL LINK DMZ
 +        index 1 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 172.31.0.243 netmask 0xffffff00 broadcast 172.31.0.255
 +xnf1: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr c2:​f2:​a4:​52:​e0:​31
 +        description:​ PHYSICAL LINK PFSYNC
 +        index 2 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
 +xnf2: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 7b:​c1:​91:​bc:​3a:​4e
 +        description:​ PHYSICAL LINK 01
 +        index 3 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255
 +xnf3: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr f2:​ba:​7c:​97:​28:​ce
 +        description:​ PHYSICAL LINK 02
 +        index 4 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 10.20.20.1 netmask 0xffffff00 broadcast 10.20.20.255
 +enc0: flags=0<>​
 +        index 5 priority 0 llprio 3
 +        groups: enc
 +        status: active
 +carp1: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​01
 +        description:​ CARP LINK DMZ F245
 +        index 7 priority 15 llprio 3
 +        carp: MASTER carpdev xnf0 vhid 1 advbase 1 advskew 0
 +        groups: carp
 +        status: master
 +        inet 172.31.0.245 netmask 0xffffff00 broadcast 172.31.0.255
 +carp2: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​02
 +        description:​ CARP LINK 01 F63
 +        index 8 priority 15 llprio 3
 +        carp: MASTER carpdev xnf2 vhid 2 advbase 1 advskew 0
 +        groups: carp egress
 +        status: master
 +        inet 200.200.200.61 netmask 0xffffffe0 broadcast 200.200.200.63
 +carp3: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​03
 +        description:​ CARP LINK 02 F108
 +        index 9 priority 15 llprio 3
 +        carp: MASTER carpdev xnf3 vhid 3 advbase 1 advskew 0
 +        groups: carp egress
 +        status: master
 +        inet 189.189.189.108 netmask 0xfffffff0 broadcast 189.189.189.111
 +pfsync0: flags=41<​UP,​RUNNING>​ mtu 1500
 +        index 10 priority 0 llprio 3
 +        pfsync: syncdev: xnf1 syncpeer: 192.168.0.2 maxupd: 128 defer: off
 +        groups: carp pfsync
 +pflog0: flags=141<​UP,​RUNNING,​PROMISC>​ mtu 33144
 +        index 11 priority 0 llprio 3
 +        groups: pflog
 +</​sxh>​
 +
 +
 +As we can see above the carp status is master for all the carp interfaces in the Master Server, the pfsync0 is working as well.
 +
 +We can check the packets are working throughtout the pfsync0 with tcpdump
 +<sxh bash>
 +tcpdump -e -n -vv -ttt -i pfsync0
 +tcpdump: listening on pfsync0, link-type PFSYNC
 +Oct 13 15:​31:​48.270136 PFSYNCv6 len 556
 +    act UPD ST REQ count 2
 +        id: 57ff8b7c000002b1 creatorid: 7116b56a
 +        id: 57ff8b7c000002b2 creatorid: 7116b56a
 +    act UPD ST COMP count 6
 +    ...
 +Oct 13 15:​31:​48.270432 PFSYNCv6 len 552
 +    act UPD ST count 2
 +    ...
 +Oct 13 15:​31:​49.280142 PFSYNCv6 len 556
 +    act UPD ST REQ count 2
 +        id: 57ff8b7c000002b1 creatorid: 7116b56a
 +        id: 57ff8b7c000002b2 creatorid: 7116b56a
 +    act UPD ST COMP count 6
 +    ...
 +Oct 13 15:​31:​49.280461 PFSYNCv6 len 552
 +    act UPD ST count 2
 +</​sxh>​
 +
 +Another way to check if the pfsync is working is double checking the pf status
 +<sxh bash>
 +pfctl -sa
 +[...]
 +STATES:
 +all pfsync 192.168.0.1 -> 192.168.0.2 ​      ​MULTIPLE:​MULTIPLE
 +[...]
 +</​sxh>​
 +
 +The states of both firewall need to be the same in the states section.
 +===== Configuring the Carp Backup Network Interfaces =====
 +
 +Here we will configure the interfaces that will be used by the Backup carp server.
 +
 +Note: I have no ideia why and how spaces matter in the interfaces configuration,​ so don't put more spaces than need. Ex: param + space + another_param and so on, otherwise you will have some pass throughout some troubles that can not be explained.
 +
 +==== Carp Backup DMZ Interface ====
 +
 +Let's configure the physical interface with the real ip address that will be used to access the server throughtout the DMZ Network.
 +<sxh bash>
 +vim /​etc/​hostname.xnf0
 +inet 172.31.0.244 255.255.255.0 172.31.0.255 description "​PHYSICAL LINK DMZ"
 +</​sxh>​
 +
 +
 +Let's configure the carp DMZ Interface that will be used by the DMZ clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. This ip address will be used as gateway of the servers in the DMZ Network.
 +
 +This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.
 +
 +<sxh bash>
 +vim /​etc/​hostname.carp1
 +inet 172.31.0.245 255.255.255.0 172.31.0.255 vhid 1 carpdev xnf0 pass lanpasswd advskew 10 description "CARP LINK DMZ F245"
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +==== Carp Backup PFSync Interface ====
 +
 +Let's configure the physical interface with the real ip address that will be used to change information about the pfsync between the firewalls.
 +<sxh bash>
 +vim /​etc/​hostname.xnf1
 +inet 192.168.0.2 255.255.255.0 192.168.0.255 description "​PHYSICAL LINK PFSYNC"​
 +</​sxh>​
 +
 +Now let's configure the virtual pfsync interface that keeps the information about the another peer of the pfsync configuration.
 +
 +**Note:** If you are using some kind of virtual enviroment you need to create a private interface that can be used only between the the two firewalls, because the information that will be exchange between the servers are no encrypted.
 +
 +<sxh bash>
 +vim /​etc/​hostname.pfsync0
 +up syncdev xnf1 syncpeer 192.168.0.1
 +</​sxh>​
 +
 +You can get more information about the pfsync configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​pfsync.4|pfsync(4)]] and [[http://​man.openbsd.org/​ifconfig.8|ifconfig(8)]]
 +==== Carp Backup Internet Link 01 Interface ====
 +
 +Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.
 +<sxh bash>
 +vim /​etc/​hostname.xnf2
 +inet 10.10.10.2 255.255.255.0 10.10.10.255 description "​PHYSICAL LINK 01"
 +</​sxh>​
 +
 +
 +Let's configure the carp Internet Link 01 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. ​
 +
 +This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.
 +
 +**Note:** This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.
 +
 +<sxh bash>
 +vim /​etc/​hostname.carp2
 +inet 200.200.200.61 255.255.255.224 200.200.200.63 vhid 2 carpdev xnf2 pass lanpasswd advskew 10 description "CARP LINK 01 F63"
 +!route add -mpath default 200.200.200.33
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +==== Carp Backup Internet Link 02 Interface ====
 +
 +Let's configure the physical interface with the private ip address that will be used only between the firewall to talk with each other.
 +<sxh bash>
 +vim /​etc/​hostname.xnf2
 +inet 10.20.20.2 255.255.255.0 10.20.20.255 description "​PHYSICAL LINK 02"
 +</​sxh>​
 +
 +
 +Let's configure the carp Internet Link 02 Interface that will be used by Internet clients, with it if one of the server goes down the another one will assume this ip address and the clients will continue working without any problem. ​
 +
 +This Backup server will only assume this ip address if the master carp server goes down or the server has some issue the network interface, otherwise this server will be in standby.
 +
 +**Note:** This is the real public ip address that will be used by the internet, as we only have one ip address we need to use an extra network ip address in the physical interface and the real one in the carp interface.
 +
 +<sxh bash>
 +vim /​etc/​hostname.carp3
 +inet 189.189.189.108 255.255.255.240 189.189.189.111 vhid 3 carpdev xnf3 pass lanpasswd advskew 10 description "CARP LINK 02 F108"
 +!route add -mpath default 189.189.189.97
 +</​sxh>​
 +
 +You can see more options about the carp configuration in [[http://​man.openbsd.org/​OpenBSD-current/​man4/​carp.4|carp(4)]]. The description will help you to get to know which link is what.
 +
 +
 +===== Carp Backup Kernel Configuration =====
 +
 +Now we need to enable the equal cost multipath routing into the kernel. You can get more information about the multipath routing configuration in [[http://​www.openbsd.org/​faq/​faq6.html#​Multipath|Multipath]]
 +<sxh bash>
 +sysctl net.inet.ip.multipath=1
 +sysctl net.inet6.ip6.multipath=1
 +</​sxh>​
 +
 +We can enable the forward routing to act as a gateway. ​
 +<sxh bash>
 +sysctl net.inet.ip.forwarding=1
 +</​sxh>​
 +
 +Now we need to configure the kernel to work with the carp.  You can get more information about the carp configuration in [[http://​www.openbsd.org/​faq/​pf/​carp.html|carp]]
 +<sxh bash>
 +sysctl net.inet.carp.allow=1
 +sysctl net.inet.carp.preempt=1
 +sysctl net.inet.carp.log=2
 +</​sxh>​
 +
 +Now we need to keep this configuration in the boot time, so let's edit the sysctl.conf
 +<sxh bash>
 +vim /​etc/​sysctl.conf
 +[...]
 +net.inet.ip.multipath=1
 +net.inet6.ip6.multipath=1
 +net.inet.ip.forwarding=1
 +net.inet.carp.allow=1
 +net.inet.carp.preempt=1
 +net.inet.carp.log=2
 +</​sxh>​
 +
 +==== Desabling the default route in the Backup Carp ====
 +
 +Now we need to remove the **/​etc/​mygate** because we have multiples routes one for each internet link and we don't need this file. The routes is in the carp2 and carp3 in the **!route add -mpath default x.x.x.x** directive.
 +
 +So let's remove the file
 +<sxh bash>
 +rm -rf /etc/mygate
 +</​sxh>​
 +
 +Now restart the server to get all the configuration
 +<sxh bash>
 +reboot
 +</​sxh>​
 +
 +==== Checking the Backup Carp Configuration ====
 +
 +Now we need to check the backup carp configuration and double check if everything is working properly.
 +<sxh bash>
 +ifconfig
 +lo0: flags=8049<​UP,​LOOPBACK,​RUNNING,​MULTICAST>​ mtu 32768
 +        index 6 priority 0 llprio 3
 +        groups: lo
 +        inet6 ::1 prefixlen 128
 +        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
 +        inet 127.0.0.1 netmask 0xff000000
 +xnf0: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 5b:​48:​fe:​25:​3b:​fe
 +        description:​ PHYSICAL LINK DMZ
 +        index 1 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 172.31.0.244 netmask 0xffffff00 broadcast 172.31.0.255
 +xnf1: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 62:​c2:​35:​79:​13:​e8
 +        description:​ PHYSICAL LINK PFSYNC
 +        index 2 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
 +xnf2: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr ae:​b0:​4c:​78:​15:​84
 +        description:​ PHYSICAL LINK COPEL
 +        index 3 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 10.10.10.2 netmask 0xffffff00 broadcast 10.10.10.255
 +xnf3: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 78:​c0:​7d:​76:​af:​10
 +        description:​ PHYSICAL LINK EMBRATEL
 +        index 4 priority 0 llprio 3
 +        media: Ethernet manual
 +        status: active
 +        inet 10.20.20.2 netmask 0xffffff00 broadcast 10.20.20.255
 +enc0: flags=0<>​
 +        index 5 priority 0 llprio 3
 +        groups: enc
 +        status: active
 +carp1: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​01
 +        description:​ CARP LINK DMZ F245
 +        index 7 priority 15 llprio 3
 +        carp: BACKUP carpdev xnf0 vhid 1 advbase 1 advskew 10
 +        groups: carp
 +        status: backup
 +        inet 172.31.0.245 netmask 0xffffff00 broadcast 172.31.0.255
 +carp2: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​02
 +        description:​ CARP LINK COPEL F63
 +        index 8 priority 15 llprio 3
 +        carp: BACKUP carpdev xnf2 vhid 2 advbase 1 advskew 10
 +        groups: carp egress
 +        status: backup
 +        inet 200.195.146.61 netmask 0xffffffe0 broadcast 200.195.146.63
 +carp3: flags=8843<​UP,​BROADCAST,​RUNNING,​SIMPLEX,​MULTICAST>​ mtu 1500
 +        lladdr 00:​00:​5e:​00:​01:​03
 +        description:​ CARP LINK EMBRATEL F108
 +        index 9 priority 15 llprio 3
 +        carp: BACKUP carpdev xnf3 vhid 3 advbase 1 advskew 10
 +        groups: carp egress
 +        status: backup
 +        inet 189.2.39.108 netmask 0xfffffff0 broadcast 189.2.39.111
 +pfsync0: flags=41<​UP,​RUNNING>​ mtu 1500
 +        index 10 priority 0 llprio 3
 +        pfsync: syncdev: xnf1 syncpeer: 192.168.0.1 maxupd: 128 defer: off
 +        groups: carp pfsync
 +pflog0: flags=141<​UP,​RUNNING,​PROMISC>​ mtu 33144
 +        index 11 priority 0 llprio 3
 +        groups: pflog
 +</​sxh>​
 +
 +
 +As we can see above the carp status is backup for all the carp interfaces in the Backup Server, the pfsync0 is working as well.
 +
 +We can check the packets are working throughtout the pfsync0 with tcpdump
 +<sxh bash>
 +tcpdump -e -n -vv -ttt -i pfsync0
 +tcpdump: listening on pfsync0, link-type PFSYNC
 +Oct 13 16:​03:​27.785419 PFSYNCv6 len 360
 +    act UPD ST COMP count 4
 +        id: 57ff8b7c000002b0 creatorid: 7116b56a
 +    ...
 +Oct 13 16:​03:​27.785783 PFSYNCv6 len 580
 +    act UPD ST REQ count 2
 +        id: 57ff8bea000000aa creatorid: 95a434c3
 +        id: 57ff8bea000000a9 creatorid: 95a434c3
 +    act UPD ST count 2
 +    ...
 +Oct 13 16:​03:​28.795391 PFSYNCv6 len 360
 +    act UPD ST COMP count 4
 +        id: 57ff8b7c000002b0 creatorid: 7116b56a
 +    ...
 +Oct 13 16:​03:​28.795781 PFSYNCv6 len 580
 +    act UPD ST REQ count 2
 +        id: 57ff8bea000000aa creatorid: 95a434c3
 +        id: 57ff8bea000000a9 creatorid: 95a434c3
 +    act UPD ST count 2
 +</​sxh>​
 +
 +Another way to check if the pfsync is working is double checking the pf status
 +<sxh bash>
 +pfctl -sa
 +[...]
 +STATES:
 +all pfsync 192.168.0.2 -> 192.168.0.1 ​      ​MULTIPLE:​MULTIPLE
 +[...]
 +</​sxh>​
 +
 +The states of both firewall need to be the same in the states section.
 +
 +===== PF Configuration for both Firewalls =====
 +
 +Here as the firewall are sharing the information about the connections and the ip address we need to keep the pf.conf the same on both sides, here so far I cannot figure out a good daemon to do that, so let's create the file in one side and copy to the other side.
 +
 +Here I will share with you guys an pf.conf with some configuration for the environment that we are planning to work, in other words with multipath routing.
 +
 +You can get more information about the pf syntax in [[http://​www.openbsd.org/​faq/​pf/​index.html|PF:​ User's Guide]]
 +
 +Let's create the pf.conf on the Firewall 01
 +<sxh bash>
 +vim /​etc/​pf.conf
 +#/​etc/​pf.conf
 +
 +### INTERFACES
 +#########################​
 +# LC = LINK 01
 +# LE = LINK 02
 +# Fx = FINAL x EX: 200.200.200.x
 +#########################​
 +lc_if     = "{ xnf2 }" ​                         # INTERNET 01 PHYSICAL INTEFACE
 +le_if     = "{ xnf3 }" ​                         # INTERNET 02 PHYSICAL INTERFACE
 +lc_f61 ​   = "{ carp2 }" ​                        # CARP 2 -> 200.200.200.61
 +le_f108 ​  = "{ carp3 }" ​                        # CARP 3 -> 189.189.189.108
 +dmz_if ​   = "{ xnf0 }" ​                         # DMZ PHYSICAL INTERFACE
 +ldmz_f245 = "{ carp1 }" ​                        # CARP 1 -> 172.31.0.245
 +all_if ​   = "{ xnf0 xnf1 xnf2 xnf3 }" ​          # ALL INTERFACES
 +sync_if ​  = "{ xnf1 }" ​                         # PFSYNC INTERFACE
 +carp_if ​  = "{ xnf0 xnf2 xnf3 }" ​               # CARP INTERFACES
 +
 +### NETWORKS
 +dmz_net = "​172.31.0.0/​24" ​                              # RANGE: 172.31.0.1-172.31.0.254 ​         GW: 172.31.0.254 ​   BRD: 172.31.0.255 ​     NETMASK: 255.255.255.0
 +lc_net ​ = "​200.200.200.32/​27" ​                          # RANGE: 200.200.200.33-200.200.200.62 ​   GW: 200.200.200.33 ​ BRD: 200.200.200.63 ​   NETMASK: 255.255.255.224
 +le_net ​ = "​189.189.189.96/​28" ​                          # RANGE: 189.189.189.97-189.189.189.110 ​  GW: 189.189.189.97 ​ BRD: 189.189.189.111 ​  ​NETMASK:​ 255.255.255.240
 +lcc_net = "​10.10.10.0/​24" ​                              # RANGE: 10.10.10.1-10.10.10.254 ​         GW:                 BRD: 10.10.10.255 ​     NETMASK: 255.255.255.0
 +lce_net = "​10.20.20.0/​24" ​                              # RANGE: 10.20.20.1-10.20.20.254 ​         GW:                 BRD: 10.20.20.255 ​     NETMASK: 255.255.255.0
 +
 +### SERVERS ​ ###
 +web_srvs ​    = "{ 172.31.0.131 172.31.0.132 }" ​                 # labpfcli01, labpfcli02
 +mysql_srv ​   = "{ 172.31.0.134 }" ​                              # labpfcli03
 +pgsql_srv ​   = "{ 172.31.0.135 }" ​                              # labpfcli04
 +dmz_dns_srv ​ = "{ 172.31.0.200 172.31.0.201 172.31.0.202 }" ​    # labpfdns01, labpfdns02, labpfdns03
 +#​ext_dns_srv ​ = "{ 8.8.8.8 8.8.4.4 }" ​                          # googledns1, googledns2
 +all_dns_srv ​ = "{ 172.31.0.200 172.31.0.201 8.8.8.8 8.8.4.4 }" ​ # labpfdns01, labpfdns02, googledns1, googledns2
 +ftp_srv ​     = "{ 172.31.0.135 }" ​                              # labpfcli04
 +
 +### TABLES
 +## REPOSITORIES
 +table <​debian_repo> ​    file "/​etc/​tables/​debianrepositories" ​  # DEBIAN REPOSITORIES
 +table <​centos_repo> ​    file "/​etc/​tables/​centosrepositories" ​  # CENTOS REPOSITORIES
 +
 +## NTP SERVERS
 +table <​ntp_srv> ​        file "/​etc/​tables/​ntpservers" ​          # EXTERNAL NTP SERVERS
 +table <​ntp_srv_dmz> ​    file "/​etc/​tables/​ntpservers_dmz" ​      # DMZ NTP SERVERS
 +
 +### SERVICES
 +icmp_types = "{ echoreq unreach }" ​             # ICMP TYPES ALLOWED
 +dns_port ​  = "​53" ​                              # DNS PORT
 +ssh_port ​  = "​2222" ​                            # SSH PORT
 +mysql_port = "​3306" ​                            # MYSQL PORT
 +pgsql_port = "​5432" ​                            # PSGQL PORT
 +http_port ​ = "​80" ​                              # HTTP PORT
 +ftp_port ​  = "​21" ​                              # FTP PORT FTPPROXY
 +ntp_port ​  = "​123" ​                             # NTP PORT
 +repo_ports = "{ 80 443 }" # REPOSITORY PORTS
 +ftp_p_ports = "​49151"​ #​ FTP PASSIVE PORTS >= 49151
 +ftp_ports = "{ 20 21 }"​ #​ FTP PORTS
 +
 +### SETTING UP THE RUNTIME OPTIONS BEGIN ###
 +set skip on lo
 +set block-policy drop
 +set loginterface egress
 +set state-policy floating
 +### SETTING UP THE RUNTIME OPTIONS END   ###
 +
 +### NORMALIZING INCOMING PACKETS ​    ###
 +match in all scrub (no-df random-id max-mss 1440)
 +### NORMALIZING INCOMING PACKETS END ###
 +
 +### DEFAULT POLICY BEGIN ###
 +block log all
 +### DEFAULT POLICY END   ###
 +
 +### ANTISPOOF BEGIN ###
 +antispoof for $all_if inet
 +### ANTISPOOF END   ###
 +
 +### PFSYNC BEGIN ###
 +pass log quick on $sync_if proto pfsync keep state
 +### PFSYNC END   ###
 +
 +### CARP BEGIN ###
 +pass log on $carp_if proto carp keep state
 +### CARP END ###
 +
 +### SSH BEGIN ###
 +## DMZ TO FW
 +pass in log quick on $dmz_if inet proto tcp from $dmz_net to $ldmz_f245 port $ssh_port
 +pass in log quick on $dmz_if inet proto tcp from $dmz_net to $dmz_if ​   port $ssh_port
 +### SSH END  ###
 +
 +### TRACEROUTE BEGIN ###
 +## FW TO ANY
 +pass out on $lc_if ​ inet proto udp from $lc_net ​ to any port 33433 >< 33626 keep state
 +pass out on $le_if ​ inet proto udp from $le_net ​ to any port 33433 >< 33626 keep state
 +
 +## DMZ TO ANY
 +pass out log on $dmz_if inet proto udp from $dmz_net to any port 33433 >< 33626 keep state
 +pass in  log on $dmz_if inet proto udp from $dmz_net to any port 33433 >< 33626 keep state
 +### TRACEROUTE END ###
 +
 +### ICMP BEGIN ###
 +## FW TO ANY
 +pass out log inet proto icmp from $lc_net to any  icmp-type $icmp_types keep state
 +pass in  log inet proto icmp from any to $lc_net ​ icmp-type $icmp_types keep state
 +pass out log inet proto icmp from $le_net to any  icmp-type $icmp_types keep state
 +pass in  log inet proto icmp from any to $le_net ​ icmp-type $icmp_types keep state
 +pass out log inet proto icmp from $lcc_net to any icmp-type $icmp_types keep state
 +pass in  log inet proto icmp from any to $lcc_net icmp-type $icmp_types keep state
 +pass out log inet proto icmp from $lce_net to any icmp-type $icmp_types keep state
 +pass in  log inet proto icmp from any to $lce_net icmp-type $icmp_types keep state
 +
 +## DMZ TO ANY
 +pass in log on $dmz_if inet proto icmp from $dmz_net to any icmp-type $icmp_types keep state
 +pass in log on $dmz_if inet proto icmp from $dmz_net to any icmp-type $icmp_types keep state
 +### ICMP END   ###
 +
 +### DNS BEGIN ###
 +## FW TO EXTERNAL AND DMZ DNS SERVERS
 +pass out log on $lc_if inet proto udp from $lc_net to $all_dns_srv port $dns_port keep state
 +pass out log on $le_if inet proto udp from $le_net to $all_dns_srv port $dns_port keep state
 +
 +### DNS SERVER TO EXTERNAL
 +pass out log on $dmz_if inet proto udp from $dmz_dns_srv to any port $dns_port keep state
 +pass in  log on $dmz_if inet proto udp from $dmz_dns_srv to any port $dns_port keep state
 +pass out log on $dmz_if inet proto tcp from $dmz_dns_srv to any port $dns_port keep state
 +pass in  log on $dmz_if inet proto tcp from $dmz_dns_srv to any port $dns_port keep state
 +
 +## DMZ TO EXTERNAL AND DMZ SERVERS
 +pass out log on $dmz_if inet proto udp from $dmz_net to $all_dns_srv port $dns_port keep state
 +pass in  log on $dmz_if inet proto udp from $dmz_net to $all_dns_srv port $dns_port keep state
 +### DNS END   ###
 +
 +### NTP BEGIN ###
 +## FW TO EXTERNAL NTP SERVERS
 +pass out log on $lc_if inet proto udp from $lc_net to { <​ntp_srv>,​ <​ntp_srv_dmz>​ } port $ntp_port keep state
 +pass out log on $le_if inet proto udp from $le_net to { <​ntp_srv>,​ <​ntp_srv_dmz>​ } port $ntp_port keep state
 +
 +## DMZ TO NTP SERVERS
 +pass out log on $dmz_if inet proto udp from $dmz_net to <​ntp_srv_dmz>​ port $ntp_port keep state
 +pass in  log on $dmz_if inet proto udp from $dmz_net to <​ntp_srv_dmz>​ port $ntp_port keep state
 +### NTP END   ###
 +
 +### REPOSITORIES BEGIN ###
 +pass out log on $dmz_if inet proto tcp from $dmz_net to <​debian_repo>​ port $http_port keep state
 +pass in  log on $dmz_if inet proto tcp from $dmz_net to <​debian_repo>​ port $http_port keep state
 +pass out log on $dmz_if inet proto tcp from $dmz_net to <​centos_repo>​ port $repo_ports keep state
 +pass in  log on $dmz_if inet proto tcp from $dmz_net to <​centos_repo>​ port $repo_ports keep state
 +anchor "​ftp-proxy/​*"​
 +pass in quick on $dmz_if inet proto tcp to port $ftp_port divert-to 127.0.0.1 port 8021
 +pass out inet proto tcp from (self) to <​debian_repo>​ port $ftp_port
 +pass out inet proto tcp from (self) to <​centos_repo>​ port $ftp_port
 +### REPOSITORIES END   ###
 +
 +### MYSQL BEGIN ###
 +## REDIRECT TO MSSQL SERVER labpfcli03
 +pass in  log on $lc_if ​ proto tcp to $lc_f61 port 33060 rdr-to $mysql_srv port $mysql_port
 +pass out log on $dmz_if proto tcp to $mysql_srv port $mysql_port
 +### MYSQL END   ###
 +
 +### POSTGRESQL BEGIN ###
 +## REDIRECT TO PGSQL SERVER labpfcli04
 +pass in  log on $le_if ​ proto tcp to $le_f108 port 54320 rdr-to $pgsql_srv port $pgsql_port
 +pass out log on $dmz_if proto tcp to $pgsql_srv port $pgsql_port
 +### POSTGRESQL END   ###
 +
 +### FTP SERVER BEGIN ###
 +## REDIRECT TO FTP SERVER labpfcli04
 +pass in  log on $le_if ​ proto tcp to   ​$le_f108 port   ​$ftp_ports ​  ​rdr-to $ftp_srv
 +pass in  log on $le_if ​ proto tcp to   ​$le_f108 port > $ftp_p_ports rdr-to $ftp_srv
 +pass out log on $dmz_if proto tcp to   ​$ftp_srv port   ​$ftp_ports
 +pass out log on $dmz_if proto tcp from $ftp_srv port   ​$ftp_ports to any
 +pass in  log on $dmz_if proto tcp from $ftp_srv port   ​$ftp_ports to any
 +pass out log on $dmz_if proto tcp to   ​$ftp_srv port > $ftp_p_ports
 +
 +### FTP SERVER END   ###
 +
 +### POOL WEB SERVERS BEGIN ###
 +## REDIRECT TO POOL WEB SERVERS labpfcli01, labpfcli02
 +pass in  log on $lc_if ​ proto tcp to port $http_port rdr-to $web_srvs round-robin sticky-address
 +pass in  log on $le_if ​ proto tcp to port $http_port rdr-to $web_srvs round-robin sticky-address
 +pass out log on $dmz_if proto tcp to $web_srvs port $http_port
 +### POOL WEB SERVERS END   ###
 +
 +### NAT ALL THE DMZ NET BEGIN ###
 +pass out log inet from { $dmz_net } to any nat-to (egress)
 +### NAT ALL THE DMZ NET END   ###
 +</​sxh>​
 +
 +Now we need to enable the ftpproxy that will help the pf to create dynamic rules to pass the ftp clients connections,​ you can get more information about the ftpproxy in [[http://​www.openbsd.org/​faq/​pf/​ftp.html|Issues with FTP]]
 +
 +Now we need to add it into **/​etc/​rc.conf.local** to set up the flags to initialize the daemon of ftpproxy.
 +<sxh bash>
 +vim /​etc/​rc.conf.local
 +ftpproxy_flags="​-D7 -v"
 +</​sxh>​
 +
 +Now we need to enable the ftpproxy and start it.
 +<sxh bash>
 +rcctl enable ftpproxy
 +rcctl start  ftpproxy
 +</​sxh>​
 +
 +Now we need to create the directory to store the files with the tables
 +<sxh bash>
 +mkdir -p /etc/tables
 +</​sxh>​
 +
 +Now let's change the permission of the directory
 +<sxh bash>
 +chmod 750 /etc/tables
 +</​sxh>​
 +
 +**Note:** Here I will use the ip address rather than the dns name because will boost the speed in the processing the tables without resolving the dns name.
 +
 +Now we need to create the files with the content, the first file will be the CentOS repositories file
 +<sxh bash>
 +vim /​etc/​tables/​centosrepositories
 +# http://​mirrorlist.centos.org/?​release=7&​arch=x86_64&​repo=os
 +# http://​mirrorlist.centos.org/?​release=7&​arch=x86_64&​repo=extras
 +# http://​mirrorlist.centos.org/?​release=7&​arch=x86_64&​repo=updates
 +# http://​mirrorlist.centos.org/?​release=7&​arch=x86_64&​repo=centosplus
 +# http://​mirrorlist.centos.org/?​release=7&​arch=x86_64&​repo=fasttrack
 +# https://​mirrors.fedoraproject.org/​metalink?​repo=epel-7&​arch=x86_64
 +# mirrorlist.centos.org
 +85.236.43.108
 +212.69.166.138
 +216.176.179.218
 +67.219.148.138
 +# centos.brnet.net.br
 +177.124.188.233
 +# centos.brisanet.com.br
 +177.37.220.74
 +# mirror.facom.ufms.br
 +200.129.206.120
 +# mirror.globo.com
 +131.0.25.51
 +# centos.xpg.com.br
 +187.17.123.242
 +# mirror.nbtelecom.com.br
 +189.45.5.90
 +# centos.ufes.br
 +200.137.64.198
 +# mirrors.uprm.edu
 +136.145.216.245
 +# mirror.orbyta.com
 +190.196.215.59
 +# mirror.gtdinternet.com
 +190.196.123.25
 +# linorg.usp.br
 +200.144.183.235
 +# mirror.bytemark.co.uk
 +212.110.161.69
 +# mirrors.coreix.net
 +85.13.241.50
 +# ftp.free.fr
 +212.27.60.27
 +# ftp.hosteurope.de
 +80.237.136.138
 +# mirror.as24220.net
 +116.66.162.254
 +# centos.mirror.iweb.ca
 +192.175.120.169
 +# mirror.steadfast.net
 +208.100.4.53
 +# mirror.centos.org
 +187.45.181.183
 +# ftp.jaist.ac.jp
 +150.65.7.130
 +# mirror.us.leaseweb.net
 +108.59.10.97
 +# mirror.vtti.vt.edu
 +198.82.152.116
 +# mirror.us.oneandone.net
 +74.208.4.167
 +# mirror.steadfast.net
 +208.100.4.53
 +# vault.centos.org -> CentOS Sources
 +160.10.26.25
 +# mirror.uta.edu.ec
 +200.93.227.165
 +# epel.gtdinternet.com
 +190.196.123.25
 +# apps.fedoraproject.org
 +209.132.181.16
 +152.19.134.142
 +140.211.169.206
 +67.219.144.68
 +174.141.234.172
 +152.19.134.198
 +209.132.181.15
 +140.211.169.196
 +185.141.165.254
 +# pkgs.repoforge.org
 +78.46.17.228
 +</​sxh>​
 +
 +Now we need to create the file that will store the information about the Debian repositories
 +<sxh bash>
 +vim /​etc/​tables/​debianrepositories
 +# ftp.br.debian.org
 +200.236.31.3
 +# security.debian.org
 +200.17.202.197
 +# www.debian-multimedia.org
 +169.47.15.77
 +# packages.dotdeb.org
 +195.154.242.153
 +# nginx.org
 +206.251.255.63
 +95.211.80.227
 +# apt.dockerproject.org
 +52.84.170.153
 +</​sxh>​
 +
 +Now we need to create the file that will store the information about the public NTP servers
 +<sxh bash>
 +vim /​etc/​tables/​ntpservers
 +# http://​www.pool.ntp.org/​zone/​br
 +# a.ntp.br
 +200.160.0.8
 +# 0.br.pool.ntp.org
 +200.186.125.195
 +200.229.193.194
 +# 1.br.pool.ntp.org
 +192.155.90.13
 +200.160.0.8
 +# 2.br.pool.ntp.org
 +200.160.7.193
 +200.189.40.8
 +# 3.br.pool.ntp.org
 +200.192.232.8
 +192.99.2.8
 +</​sxh>​
 +
 +Now we need to create the file that will store the information about the DMZ NTP servers
 +<sxh bash>
 +vim /​etc/​tables/​ntpservers_dmz
 +# labpfntp01.dqs.local
 +172.31.0.210
 +# labpfntp02.dqs.local
 +172.31.0.211
 +</​sxh>​
 +
 +**Note:** Don't forget to change the ip address and the port numbers to fit your needs.
 +
 +Now we can start the pf but before it we need to test the file configuration
 +<sxh bash>
 +pfctl -nvvf /​etc/​pf.conf
 +Loaded 710 passive OS fingerprints
 +lc_if = "{ xnf2 }"
 +le_if = "{ xnf3 }"
 +lc_f61 = "{ carp2 }"
 +le_f108 = "{ carp3 }"
 +dmz_if = "{ xnf0 }"
 +ldmz_f245 = "{ carp1 }"
 +all_if = "{ xnf0 xnf1 xnf2 xnf3 }"
 +sync_if = "{ xnf1 }"
 +carp_if = "{ xnf0 xnf2 xnf3 }"
 +dmz_net = "​172.31.0.0/​24"​
 +lc_net = "​200.200.200.32/​27"​
 +le_net = "​189.189.189.96/​28"​
 +lcc_net = "​10.10.10.0/​24"​
 +lce_net = "​10.20.20.0/​24"​
 +web_srvs = "{ 172.31.0.131 172.31.0.132 }"
 +mysql_srv = "{ 172.31.0.202 }"
 +pgsql_srv = "{ 172.31.0.135 }"
 +dmz_dns_srv = "{ 172.31.0.200 172.31.0.201 172.31.0.202 }"
 +ext_dns_srv = "{ 8.8.8.8 8.8.4.4 }"
 +all_dns_srv = "{ 172.31.0.200 172.31.0.201 8.8.8.8 8.8.4.4 }"
 +ftp_srv = "{ 172.31.0.135 }"
 +table <​debian_repo>​ file "/​etc/​tables/​debianrepositories"​
 +table <​centos_repo>​ file "/​etc/​tables/​centosrepositories"​
 +table <​ntp_srv>​ file "/​etc/​tables/​ntpservers"​
 +table <​ntp_srv_dmz>​ file "/​etc/​tables/​ntpservers_dmz"​
 +icmp_types = "{ echoreq unreach }"
 +dns_port = "​53"​
 +ssh_port = "​22022"​
 +mysql_port = "​3306"​
 +pgsql_port = "​5432"​
 +http_port = "​80"​
 +ftp_port = "​21"​
 +ntp_port = "​123"​
 +repo_ports = "{ 80 443 }"
 +ftp_p_ports = "​49151"​
 +ftp_ports = "{ 20 21 }"
 +set skip on { lo }
 +set block-policy drop
 +set loginterface egress
 +set state-policy floating
 +table <​__automatic_0>​ const { 172.31.0.131 172.31.0.132 }
 +table <​__automatic_1>​ const { 172.31.0.131 172.31.0.132 }
 +@0 match in all scrub (no-df random-id max-mss 1440)
 +@1 block drop log all
 +@2 block drop in on ! xnf0 inet from 172.31.0.0/​24 to any
 +@3 block drop in inet from 172.31.0.243 to any
 +@4 block drop in on ! xnf1 inet from 192.168.0.0/​24 to any
 +@5 block drop in inet from 192.168.0.1 to any
 +@6 block drop in on ! xnf2 inet from 10.10.10.0/​24 to any
 +@7 block drop in inet from 10.10.10.1 to any
 +@8 block drop in on ! xnf3 inet from 10.20.20.0/​24 to any
 +@9 block drop in inet from 10.20.20.1 to any
 +@10 pass log quick on xnf1 proto pfsync all
 +@11 pass log on xnf0 proto carp all
 +@12 pass log on xnf2 proto carp all
 +@13 pass log on xnf3 proto carp all
 +@14 pass in log quick on xnf0 inet proto tcp from 172.31.0.0/​24 to 172.31.0.245 port = 22022 flags S/SA
 +@15 pass in log quick on xnf0 inet proto tcp from 172.31.0.0/​24 to 172.31.0.243 port = 22022 flags S/SA
 +@16 pass out on xnf2 inet proto udp from 200.200.200.32/​27 to any port 33433 >< 33626
 +@17 pass out on xnf3 inet proto udp from 189.189.189.96/​28 to any port 33433 >< 33626
 +@18 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to 172.31.0.200 port = 53
 +@19 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to 172.31.0.201 port = 53
 +@20 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to 8.8.8.8 port = 53
 +@21 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to 8.8.4.4 port = 53
 +@22 pass out log on xnf0 inet proto udp from 172.31.0.200 to any port = 53
 +@23 pass out log on xnf0 inet proto udp from 172.31.0.201 to any port = 53
 +@24 pass out log on xnf0 inet proto udp from 172.31.0.202 to any port = 53
 +@25 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to 172.31.0.200 port = 53
 +@26 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to 172.31.0.201 port = 53
 +@27 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to 8.8.8.8 port = 53
 +@28 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to 8.8.4.4 port = 53
 +@29 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to 172.31.0.200 port = 53
 +@30 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to 172.31.0.201 port = 53
 +@31 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to 8.8.8.8 port = 53
 +@32 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to 8.8.4.4 port = 53
 +@33 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to <​ntp_srv_dmz:​0>​ port = 123
 +@34 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to <​ntp_srv_dmz:​0>​ port = 123
 +@35 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to <​ntp_srv_dmz:​0>​ port = 123
 +@36 pass out log on xnf2 inet proto udp from 200.200.200.32/​27 to <​ntp_srv:​0>​ port = 123
 +@37 pass out log on xnf3 inet proto udp from 189.189.189.96/​28 to <​ntp_srv:​0>​ port = 123
 +@38 pass out log on xnf0 inet proto udp from 172.31.0.0/​24 to any port 33433 >< 33626
 +@39 pass out log inet proto icmp from 200.200.200.32/​27 to any icmp-type echoreq
 +@40 pass out log inet proto icmp from 200.200.200.32/​27 to any icmp-type unreach
 +@41 pass out log inet proto icmp from 189.189.189.96/​28 to any icmp-type echoreq
 +@42 pass out log inet proto icmp from 189.189.189.96/​28 to any icmp-type unreach
 +@43 pass out log inet proto icmp from 10.10.10.0/​24 to any icmp-type echoreq
 +@44 pass out log inet proto icmp from 10.10.10.0/​24 to any icmp-type unreach
 +@45 pass out log inet proto icmp from 10.20.20.0/​24 to any icmp-type echoreq
 +@46 pass out log inet proto icmp from 10.20.20.0/​24 to any icmp-type unreach
 +@47 pass out log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​centos_repo:​0>​ port = 80 flags S/SA
 +@48 pass out log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​centos_repo:​0>​ port = 443 flags S/SA
 +@49 pass out log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​debian_repo:​0>​ port = 80 flags S/SA
 +@50 pass out log on xnf0 inet proto tcp from 172.31.0.200 to any port = 53 flags S/SA
 +@51 pass out log on xnf0 inet proto tcp from 172.31.0.201 to any port = 53 flags S/SA
 +@52 pass out log on xnf0 inet proto tcp from 172.31.0.202 to any port = 53 flags S/SA
 +@53 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to 172.31.0.200 port = 53
 +@54 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to 172.31.0.201 port = 53
 +@55 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to 8.8.8.8 port = 53
 +@56 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to 8.8.4.4 port = 53
 +@57 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to any port 33433 >< 33626
 +@58 pass in log on xnf0 inet proto udp from 172.31.0.0/​24 to <​ntp_srv_dmz:​0>​ port = 123
 +@59 pass in log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​centos_repo:​0>​ port = 80 flags S/SA
 +@60 pass in log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​centos_repo:​0>​ port = 443 flags S/SA
 +@61 pass in log on xnf0 inet proto tcp from 172.31.0.0/​24 to <​debian_repo:​0>​ port = 80 flags S/SA
 +@62 pass in log on xnf0 inet proto icmp from 172.31.0.0/​24 to any icmp-type echoreq
 +@63 pass in log on xnf0 inet proto icmp from 172.31.0.0/​24 to any icmp-type unreach
 +@64 pass in log on xnf0 inet proto udp from 172.31.0.200 to any port = 53
 +@65 pass in log on xnf0 inet proto udp from 172.31.0.201 to any port = 53
 +@66 pass in log on xnf0 inet proto udp from 172.31.0.202 to any port = 53
 +@67 pass in log on xnf0 inet proto tcp from 172.31.0.200 to any port = 53 flags S/SA
 +@68 pass in log on xnf0 inet proto tcp from 172.31.0.201 to any port = 53 flags S/SA
 +@69 pass in log on xnf0 inet proto tcp from 172.31.0.202 to any port = 53 flags S/SA
 +@70 pass in log inet proto icmp from any to 200.200.200.32/​27 icmp-type echoreq
 +@71 pass in log inet proto icmp from any to 200.200.200.32/​27 icmp-type unreach
 +@72 pass in log inet proto icmp from any to 189.189.189.96/​28 icmp-type echoreq
 +@73 pass in log inet proto icmp from any to 189.189.189.96/​28 icmp-type unreach
 +@74 pass in log inet proto icmp from any to 10.10.10.0/​24 icmp-type echoreq
 +@75 pass in log inet proto icmp from any to 10.10.10.0/​24 icmp-type unreach
 +@76 pass in log inet proto icmp from any to 10.20.20.0/​24 icmp-type echoreq
 +@77 pass in log inet proto icmp from any to 10.20.20.0/​24 icmp-type unreach
 +@78 anchor "​ftp-proxy/​*"​ all
 +@79 pass in quick on xnf0 inet proto tcp from any to any port = 21 flags S/SA divert-to 127.0.0.1 port 8021
 +@80 pass out inet proto tcp from (self:*) to <​debian_repo:​0>​ port = 21 flags S/SA
 +@81 pass out inet proto tcp from (self:*) to <​centos_repo:​0>​ port = 21 flags S/SA
 +@82 pass in log on xnf2 inet proto tcp from any to 200.200.200.61 port = 33060 flags S/SA rdr-to 172.31.0.202 port 3306
 +@83 pass out log on xnf0 inet proto tcp from any to 172.31.0.202 port = 3306 flags S/SA
 +@84 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 54320 flags S/SA rdr-to 172.31.0.135 port 5432
 +@85 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 5432 flags S/SA
 +@86 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 20 flags S/SA rdr-to 172.31.0.135
 +@87 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port = 21 flags S/SA rdr-to 172.31.0.135
 +@88 pass in log on xnf3 inet proto tcp from any to 189.189.189.108 port > 49151 flags S/SA rdr-to 172.31.0.135
 +@89 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 20 flags S/SA
 +@90 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port = 21 flags S/SA
 +@91 pass out log on xnf0 inet proto tcp from any to 172.31.0.135 port > 49151 flags S/SA
 +@92 pass out log on xnf0 inet proto tcp from 172.31.0.135 port = 20 to any flags S/SA
 +@93 pass out log on xnf0 inet proto tcp from 172.31.0.135 port = 21 to any flags S/SA
 +@94 pass in log on xnf0 inet proto tcp from 172.31.0.135 port = 20 to any flags S/SA
 +@95 pass in log on xnf0 inet proto tcp from 172.31.0.135 port = 21 to any flags S/SA
 +@96 pass in log on xnf2 inet proto tcp from any to any port = 80 flags S/SA rdr-to <​__automatic_0:​0>​ round-robin sticky-address
 +@97 pass in log on xnf3 inet proto tcp from any to any port = 80 flags S/SA rdr-to <​__automatic_1:​0>​ round-robin sticky-address
 +@98 pass out log on xnf0 inet proto tcp from any to 172.31.0.131 port = 80 flags S/SA
 +@99 pass out log on xnf0 inet proto tcp from any to 172.31.0.132 port = 80 flags S/SA
 +@100 pass out log inet from 172.31.0.0/​24 to any flags S/SA nat-to (egress:*) round-robin
 +</​sxh>​
 +
 +Now as we don't get any problems with our configuration we can load it.
 +<sxh bash>
 +pfctl -f /​etc/​pf.conf
 +</​sxh>​
 +
 +If you don't remember that is inside a table you can check it out with the following command
 +<sxh bash>
 +pfctl -t debian_repo -T show
 +   ​52.84.170.153
 +   ​95.211.80.227
 +   ​169.47.15.77
 +   ​195.154.242.153
 +   ​200.17.202.197
 +   ​200.236.31.3
 +   ​206.251.255.63
 +</​sxh>​
 +
 +If you want to list all the rules
 +<sxh bash>
 +pfctl -vv -sr
 +@0 match in all scrub (no-df random-id max-mss 1440)
 +  [ Evaluations:​ 166       ​Packets:​ 3         ​Bytes:​ 124         ​States:​ 1     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@1 block drop log all
 +  [ Evaluations:​ 166       ​Packets:​ 141       ​Bytes:​ 16396       ​States:​ 0     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@2 block drop in on ! xnf0 inet from 172.31.0.0/​24 to any
 +  [ Evaluations:​ 166       ​Packets:​ 0         ​Bytes:​ 0           ​States:​ 0     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@3 block drop in inet from 172.31.0.243 to any
 +  [ Evaluations:​ 143       ​Packets:​ 0         ​Bytes:​ 0           ​States:​ 0     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@4 block drop in on ! xnf1 inet from 192.168.0.0/​24 to any
 +  [ Evaluations:​ 142       ​Packets:​ 0         ​Bytes:​ 0           ​States:​ 0     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@5 block drop in inet from 192.168.0.1 to any
 +  [ Evaluations:​ 142       ​Packets:​ 0         ​Bytes:​ 0           ​States:​ 0     ]
 +  [ Inserted: uid 0 pid 63813 State Creations: 0     ]
 +@6 block drop in on ! xnf2 inet from 10.10.10.0/​24 to any
 +  [ Evaluations:​ 142       ​Packets:​ 0         ​Bytes:​ 0           ​States:​ 0     ]
 +  [...]
 +</​sxh>​
 +
 +If you want to check the states of the connections
 +<sxh bash>
 +pfctl -vv -s state
 +all pfsync 192.168.0.1 -> 192.168.0.2 ​      ​MULTIPLE:​MULTIPLE
 +   age 23:51:44, expires in 00:01:00, 173059:​172953 pkts, 96057592:​79725872 bytes
 +   id: 57ff8bea00000004 creatorid: 95a434c3
 +all tcp 172.31.0.244:​22022 <- 172.31.0.250:​51356 ​      ​ESTABLISHED:​ESTABLISHED
 +   ​[2698758666 + 130976] wscale 3  [202785879 + 16952] wscale 5
 +   age 23:52:50, expires in 22:51:53, 0:0 pkts, 0:0 bytes
 +   id: 57ff8b7c000001eb creatorid: 7116b56a
 +all carp 200.195.146.61 (172.31.0.243) -> 224.0.0.18 ​      ​SINGLE:​NO_TRAFFIC
 +   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
 +   id: 57ff8bea000000a6 creatorid: 95a434c3
 +all carp 10.10.10.1 -> 224.0.0.18 ​      ​SINGLE:​NO_TRAFFIC
 +   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
 +   id: 57ff8bea000000a7 creatorid: 95a434c3
 +all carp 10.20.20.1 -> 224.0.0.18 ​      ​SINGLE:​NO_TRAFFIC
 +   age 23:51:34, expires in 00:00:30, 85045:0 pkts, 4762520:0 bytes
 +   id: 57ff8bea000000a8 creatorid: 95a434c3
 +   [...]
 +</​sxh>​
 +
 +Now we can copy the **pf.conf** and the **/​etc/​tables** with its content to the Firewall 02 and launch the pfctl and start the script.
 +
 +
 +==== Checking the PF Log ====
 +
 +Sometimes we have a hard time trying to figure out what is happining with a bunch of rules or a client that does not have the properly access, so we need a way to get information about the rules and we can check it out with the pflog0 interface or with /​var/​log/​pflog
 +
 +We can get more information about it in: [[http://​www.openbsd.org/​faq/​pf/​logging.html#​filter|PF:​ Logging]]
 +
 +Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:
 +
 +<sxh bash>
 +tcpdump -n -e -ttt -r /​var/​log/​pflog port 80
 +</​sxh>​
 +
 +This can be further refined by limiting the display of packets to a certain host and port combination:​
 +<sxh bash>
 +tcpdump -n -e -ttt -r /​var/​log/​pflog port 80 and host 172.31.0.220
 +</​sxh>​
 +
 +The same idea can be applied when reading from the pflog0 interface:
 +<sxh bash>
 +tcpdump -n -e -ttt -i pflog0 host 172.31.0.221
 +</​sxh>​
 +
 +Note that this has no impact on which packets are logged to the pflogd log file; the above commands only display packets as they are being logged.
 +
 +In addition to using the standard [[http://​man.openbsd.org/​tcpdump.8|tcpdump(8)]] filter rules, the tcpdump filter language has been extended for reading pflogd output:
 +
 +  * **ip** - address family is IPv4.
 +  * **ip6** - address family is IPv6.
 +  * **on int** - packet passed through the interface int.
 +  * **ifname int** - same as on int.
 +  * **ruleset name** - the ruleset/​anchor that the packet was matched in.
 +  * **rulenum num** - the filter rule that the packet matched was rule number num.
 +  * **action act** - the action taken on the packet. Possible actions are pass and block.
 +  * **reason res** - the reason that action was taken. Possible reasons are match, bad-offset, fragment, short, normalize, memory, bad-timestamp,​ congestion, ip-option, proto-cksum,​ state-mismatch,​ state-insert,​ state-limit,​ src-limit and synproxy.
 +  * **inbound** - packet was inbound.
 +  * **outbound** - packet was outbound.
 +
 +
 +**Example:​**
 +<sxh bash>
 +tcpdump -n -e -ttt -i pflog0 inbound and action block and on xnf0
 +</​sxh>​
 +
 +This display the log, in real-time, of inbound packets that were blocked on the xnf0 interface.
 +
 +===== Ifstated Configuration for both Firewalls =====
 +
 +Now we need to configure the ifstated to check if the internet link is working and if we have some issue with the link such as stop  working for some reason or another we need to disable the route and keep routing the the good internet link until the link come back.
 +
 +The file does not exists so we need to create it.
 +<sxh bash>
 +vim /​etc/​ifstated.conf
 +#/​etc/​ifstated.conf
 +# net.inet.carp.preempt must be enabled (set to 1) for this to work correctly.
 +
 +# LCC -> INTERNET LINK 01
 +lcc_up = "​carp2.link.up"​
 +
 +# LCE -> INTERNET LINK 02
 +lce_up = "​carp3.link.up"​
 +
 +### SETTING UP THE INICIAL STATE
 +init-state all_ok
 +
 +# CHECKING IF THE INTERNET LINK 01 IS WORKING AND REACHING THE GATEWAY
 +xnf2_ok = '( "ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /​dev/​null"​ every 10 )'
 +
 +# CHECKING IF THE INTERNET LINK 02 IS WORKING AND REACHING THE GATEWAY
 +xnf3_ok = '( "ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /​dev/​null"​ every 10 )'
 +
 +### RULES OF LINK STATE
 +state all_ok {
 +    # CHECK IF THE LINK 01 IS NOT WORKING AND THE CARP IS BACKUP
 +    if ! $xnf2_ok && ! $lcc_up {
 +        # CALL THE xnf2_down FUNCTION AND CLEAN UP THE ROUTES
 +        set-state xnf2_down
 +    }
 +
 +    # CHECK IF THE LINK 01 IS NOT WORKING AND THE CARP IS BACKUP
 +    if ! $xnf3_ok && ! $lce_up {
 +        # CALL THE xnf3_down FUNCTION AND CLEAN UP THE ROUTES
 +        set-state xnf3_down
 +    }
 +}
 +
 +
 +# STATE TO SET THE LINK 01 AS DOWN
 +state xnf2_down {
 +    # INITIAL CALL
 +    init {
 +      # CALL THE COMMAND TO START THE FUNCTION. SO CLEANING UP THE ROUTES
 +      run "route flush -iface xnf2; pfctl -k label -k out_xnf2"​
 +    }
 +    # CHECK IF THE CARP IS MASTER AND THE LINK IS OK AGAIN
 +    if $lcc_up && $xnf2_ok {
 +        # IF THE LINK IS OK AGAIN RESTORE THE ROUTES
 +        run "sh /​etc/​netstart xnf2"
 +        # CALL THE CHECK STATE AGAIN
 +        set-state all_ok
 +    }
 +}
 +
 +# STATE TO SET THE LINK 02 AS DOWN
 +state xnf3_down {
 +    # INITIAL CALL
 +    init {
 +      # CALL THE COMMAND TO START THE FUNCTION. SO CLEANING UP THE ROUTES
 +      run "route flush -iface xnf3; pfctl -k label -k out_xnf3"​
 +    }
 +    # CHECK IF THE CARP IS MASTER AND THE LINK IS OK AGAIN
 +    if $lce_up && $xnf3_ok {
 +        # IF THE LINK IS OK AGAIN RESTORE THE ROUTES
 +        run "sh /​etc/​netstart xnf3"
 +        # CALL THE CHECK STATE AGAIN
 +        set-state all_ok
 +    }
 +}
 +</​sxh>​
 +
 +Now we can star the check and make sure if everything is working.
 +
 +Here I will start the daemon manually to check if everything is working as we need.
 +<sxh bash>
 +ifstated -dvv
 +lcc_up = "​carp2.link.up"​
 +lce_up = "​carp3.link.up"​
 +xnf2_ok = "( "ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /​dev/​null"​ every 10 )"
 +xnf3_ok = "( "ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /​dev/​null"​ every 10 )"
 +initial state: all_ok
 +changing state to all_ok
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
 +started
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
 +</​sxh>​
 +
 +Now we need to unplug the cable of the link 01 and check the logs
 +<sxh bash>
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
 +ping: sendto: Network is unreachable
 +changing state to xnf2_down
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +ping: sendto: Network is unreachable
 +running route flush -iface xnf2; pfctl -k label -k out_xnf2
 +killed 0 states
 +</​sxh>​
 +
 +As we can see the link 01 is not working so the route was cleaning up and the ifstated will keep checking the status.
 +
 +Now plug the cable again and check the logs
 +<sxh bash>
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +running sh /​etc/​netstart xnf2
 +changing state to all_ok
 +running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +running ping -q -c 1 -w 1 -I 189.189.189.108 189.189.189.97 > /dev/null
 +running ping -q -c 1 -w 1 -I 200.200.200.61 200.200.200.33 > /dev/null
 +</​sxh>​
 +
 +As we can see when the ifstated figure out the link up again it will bring back the link and re-establish the multipath again.
 +
 +Now we can press ctrl + c and stop the ifstated now we need to add it to the rc.conf.local
 +<sxh bash>
 +vim /​etc/​rc.conf.local
 +ftpproxy_flags="​-D7 -v"
 +ifstated_flags=""​
 +</​sxh>​
 +
 +Now we need to put the ifstated in the boot time and start it
 +<sxh bash>
 +rcctl enable ifstated
 +rcctl start ifstated
 +</​sxh>​
 +
 +Now you can do the same process in the Firewall 02.
 +====== References ======
 +  - http://​www.openbsd.org/​faq/​faq6.html#​Multipath
 +  - https://​www.kotfu.net/​2007/​10/​openbsd-firewall-cluster/​
 +  - http://​man.openbsd.org/​ifstated.conf.5
 +  - http://​man.openbsd.org/​ifstated.8
 +  - https://​calomel.org/​ifstated.html
 +  - https://​seejanedo.com/​library/​configuring-openbsd-load-balancing-outbound/​