mjhjmtu

I'm trying out FreeBSD with bhyve hypervisor and trying to replicate VMware ESXi vSwitch config in FreeBSD so that pfSense VM gets network traffic properly.

VLANs:

• 111 = LAN, 192.168.101.0/24


• 333 = Internet (WAN), DHCP

pfSense virtual machine setup (ESXi & FreeBSD):

NICS:

• vnic0


• vnic0.111 192.168.101.1/24


• vnic0.333 DHCP from ISP


• NAT: VLAN 333 <-> VLAN 111 & DHCP server


• HW offloads off

Old ESXi setup:

• 192.168.101.6/24 on VLAN 111 GW 192.168.101.1


• Only vSwitch, no dvSwitches

vSwitch0:

 ----------------------- -----------------------
| allvlans | | Physical adapters |
| VLAN ID: 4095 (all) |----| * vmnic0, 1000 Mbps |
| * pfSense24 | | ----------------------- 
 ----------------------- |
 |
 ----------------------- |
| Management Network |--/ 
| VLAN ID: 111 |
| VMkernel ports (1): |
| * vmk0: 192.168.101.6 |
 -----------------------

Security policy:

• Allow promiscuous mode: no
  


• Allow forged transmits: no
  


• Allow MAC changes: no
  

Port group allvlans:

• VLAN ID: 4095 (allow all tagged VLAN traffic)


• Allow promiscuous mode: Inherit from vSwitch (no)


• Allow forged transmits: Inherit from vSwitch (no)


• Allow MAC changes: Inherit from vSwitch (no)

FreeBSD ESXi replacement setup attempt:

sysrc -f /boot/loader.conf vmm_load="YES"
sysrc -f /boot/loader.conf nmdm_load="YES"
sysrc -f /boot/loader.conf if_bridge_load="YES"
sysrc -f /boot/loader.conf if_tap_load="YES"
sysrc if_vlan_load="YES"
sysrc cloned_interfaces="bridge0 tap0"
sysrc ifconfig_bridge0="addm em0 addm tap0"
echo "net.link.tap.up_on_open=1" > /etc/sysctl.d/vm_network.conf
sysrc defaultrouter="192.168.101.1"
sysrc ifconfig_em0="up"
sysrc gateway_enable="YES"

ESXi's Management Network equivalent(?) for SSH access:

sysrc vlans_em0="111"
sysrc ifconfig_em0_111="inet 192.168.101.6/24"

Interfaces:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=852099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
 ether 00:25:90:14:95:8c
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 ether 02:eb:00:40:63:00
 nd6 options=9<PERFORMNUD,IFDISABLED>
 groups: bridge
 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
 ifmaxaddr 0 port 5 priority 128 path cost 2000000
 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
 ifmaxaddr 0 port 1 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=80000<LINKSTATE>
 ether 00:bd:f0:02:f7:00
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect
 status: active
 groups: tap
 Opened by PID 45408
em0.111: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=103<RXCSUM,TXCSUM,TSO4>
 ether 00:25:90:14:95:8c
 inet 192.168.101.6 netmask 0xffffff00 broadcast 192.168.101.255
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active
 vlan: 111 vlanpcp: 0 parent interface: em0
 groups: vlan

pciconf -lv:

em0@pci0:1:0:0: class=0x020000 card=0x10d315d9 chip=0x10d38086 rev=0x00 hdr=0x00
 vendor = 'Intel Corporation'
 device = '82574L Gigabit Network Connection'
 class = network
 subclass = ethernet

Running pfSense VM with:

sh /usr/share/examples/bhyve/vmrun.sh -m 2048M -d /dev/zvol/tank/pfsense0 pfsensevm

It uses tap0.

Currently I can access pfSense's web admin but vmnet0.333 doesn't get IP from my ISP. I'd like to set VLAN bridging as securely as possible (private for bridge0?). If it's possible to send all traffic to the physical switch first, that's what I would prefer. openvSwitch is also ok, but I'm not familiar with it.

DHCP request packet is correctly seen in tap0, bridge0, and em0 with tcpdump -lnexv -i <iface> "vlan 333"and not in em0.111, which is correct when requesting new IP in pfSense. There's no DHCP response from ISP. em0 and tap0 are in promisc mode which is disabled in vSwitch version.

More debugging:

Connected Raspberry Pi to my VDSL modem and the DHCP packet is seen. I also tried spoofing the pfSense's MAC address with RPi and it works. So possible culprits currently are packet checksum and packet truncating. At least Linux bridge implementation is very well known to break packets with bridging and VLANs so FreeBSD might be doing it as well? Adding VLAN 333 to em0 DHCP works. So what is bridge0 or tap0 doing?

What I am missing?