Freenas version 11.3-U3.1, PFSense routing defined VLANS for subnets, UNIFI switches with port assigned native networks to provide different subnets. (1) 6 Disk Array Pool + (1) 2 Disk NVME pool.
Objective: VM should use separate NIC due to high overhead of IPcamera software would affect 10gbe interface. Jail should have the ability to use separate nic in order to offload overhead of streaming applications from file serving. (I understand VLANs could be used in order to separate subnets all exiting the same phyiscal interface)
Result: Networking functions as expected upon initial setup, fails after reboot in 2 tested situations described below.
Hardware: 1 - 10Gbe PCI-e NIC - DHCP main interface (ix0), 2 - Intel NIC onboard ASROCK X470D4U(igb4,igb5), 4 intel nic on PCI-e adapter (igb0,1,2,3)
Jail running plex on VNET0 default networking setup
VM running with nic attached to igb4.
Situation 1: VM creation, Jail on DEFAULT bridge0
existing Jail uses ix0 per bridge0(subnet1). VM is created with nic attached to igb4(subnet2), works as expected.
Result: per ifconfig on reboot bridge0 members include both ix0 and igb4 causing a network crash. Must disconnect nic igb4 in order to access the system. can reassign VM to other nic and restart services without issue. Each reboot adds the VM assigned nic into bridge0. Tested multiple times with multiple nics, each reboot will assign bridge0 AND any nic attached to the VM.
Situation 2: create manual bridge5555 attached to igb5, assign jail to use 5555, via editing network properties from vnet0:bridge0 ->vnet0:bridge5555. Assumption bridge0 is not created for jail with dual membership. Jail should use bridge5555 with only one member.
Expectation: On reboot ifconfig bridge5555 shows only members assigned through webgui (igb5), no dual member, no network crash. jail communicates via bridge5555/igb5
Result: upon jail start bridge5555 adds member ix0 in addition to assigned igb5, network crashes. after disconnecting igb5, network recovers jail communicates through bridge5555/ix0 with igb5 down.
Attached gui screenshot AFTER reboot showing bridge5555 assign igb5; ifconfig snapshot AFTER reboot showing bridge 5555 members ix0,igb5.
Subsequent reboots result in same bridge members. VM is created bridge0 with member igb4/tap0. Jail (plex) communicates via bridge5555/ix0.