can't get iocage jails to operate in vlans

Description

https://forums.freenas.org/index.php?threads/iocage-jails-in-vlans.72287/

To summarize: I have a lagg as a primary interface and want a jail in a vlan. The lagg and vlan are setup from the GUI. Packet captures from the firewall (DHCP server) and the lagg0 interface show the vlan traffic in both directions. The vlan interface, bridge, and epair for the jail do not show the incoming vlan traffic, although the outbound traffic is properly tagged.

The hardware is a C627 PCH (X722 + X557-AT2), binding to the Intel ixl driver.

There's an odd log message at the end from the driver "aq_add_macvlan err -53, aq_error 14" as the GUI-established lagg and vlan interfaces are created.

Relevant dmesg:
pcib10: irq 38 at device 3.0 numa-domain 0 on pci6
pcib6: allocated memory range (0xaa100000-0xaa1fffff) for rid 20 of pcib10
pcib6: allocated prefetch range (0xa5000000-0xa79fffff) for rid 24 of pcib10
pcib10: domain 0
pcib10: secondary bus 29
pcib10: subordinate bus 30
pcib10: memory decode 0xaa100000-0xaa1fffff
pcib10: prefetched decode 0xa5000000-0xa79fffff
pci10: numa-domain 0 on pcib10
pcib10: allocated bus range (29-29) for rid 0 of pci10
pci10: domain=0, physical bus=29
found-> vendor=0x8086, dev=0x37d2, revid=0x04
domain=0, bus=29, slot=0, func=0
class=02-00-00, hdrtype=0x00, mfdev=1
cmdreg=0x0146, statreg=0x0010, cachelnsz=16 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=11
powerspec 3 supports D0 D3 current D0
MSI supports 1 message, 64 bit, vector masks
MSI-X supports 129 messages in map 0x1c
map[10]: type Prefetchable Memory, range 64, base 0xa6000000, size 24, enabled
pcib10: allocated prefetch range (0xa6000000-0xa6ffffff) for rid 10 of pci0:29:0:0
map[1c]: type Prefetchable Memory, range 64, base 0xa7808000, size 15, enabled
pcib10: allocated prefetch range (0xa7808000-0xa780ffff) for rid 1c of pci0:29:0:0
pcib5: matched entry for 24.0.INTD
pcib5: slot 0 INTD hardwired to IRQ 38
pcib6: slot 3 INTA is routed to irq 38
pcib10: slot 0 INTA is routed to irq 38
found-> vendor=0x8086, dev=0x37d2, revid=0x04
domain=0, bus=29, slot=0, func=1
class=02-00-00, hdrtype=0x00, mfdev=1
cmdreg=0x0146, statreg=0x0010, cachelnsz=16 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=11
powerspec 3 supports D0 D3 current D0
MSI supports 1 message, 64 bit, vector masks
MSI-X supports 129 messages in map 0x1c
map[10]: type Prefetchable Memory, range 64, base 0xa5000000, size 24, enabled
pcib10: allocated prefetch range (0xa5000000-0xa5ffffff) for rid 10 of pci0:29:0:1
map[1c]: type Prefetchable Memory, range 64, base 0xa7800000, size 15, enabled
pcib10: allocated prefetch range (0xa7800000-0xa7807fff) for rid 1c of pci0:29:0:1
pcib5: matched entry for 24.0.INTD
pcib5: slot 0 INTD hardwired to IRQ 38
pcib6: slot 3 INTA is routed to irq 38
pcib10: slot 0 INTA is routed to irq 38
ixl0: mem 0xa6000000-0xa6ffffff,0xa7808000-0xa780ffff irq 38 at device 0.0 numa-domain 0 on pci10
ixl0: using 1024 tx descriptors and 1024 rx descriptors
ixl0: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl0: attempting to allocate 9 MSI-X vectors (129 supported)
msi: routing MSI-X IRQ 267 to local APIC 68 vector 48
msi: routing MSI-X IRQ 268 to local APIC 70 vector 48
msi: routing MSI-X IRQ 269 to local APIC 72 vector 48
msi: routing MSI-X IRQ 270 to local APIC 82 vector 48
msi: routing MSI-X IRQ 271 to local APIC 84 vector 48
msi: routing MSI-X IRQ 272 to local APIC 96 vector 48
msi: routing MSI-X IRQ 273 to local APIC 100 vector 48
msi: routing MSI-X IRQ 274 to local APIC 102 vector 48
msi: routing MSI-X IRQ 275 to local APIC 114 vector 48
ixl0: using IRQs 267-275 for MSI-X
ixl0: Using MSIX interrupts with 9 vectors
ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl0: bpf attached
ixl0: Ethernet address: XX:XX:XX:XX:XX:48
msi: Assigning MSI-X IRQ 268 to local APIC 0 vector 58
msi: Assigning MSI-X IRQ 269 to local APIC 1 vector 48
msi: Assigning MSI-X IRQ 270 to local APIC 2 vector 48
msi: Assigning MSI-X IRQ 271 to local APIC 3 vector 48
msi: Assigning MSI-X IRQ 272 to local APIC 4 vector 49
msi: Assigning MSI-X IRQ 273 to local APIC 5 vector 48
msi: Assigning MSI-X IRQ 274 to local APIC 6 vector 48
msi: Assigning MSI-X IRQ 275 to local APIC 7 vector 48
ixl0: SR-IOV ready
random: harvesting attach, 8 bytes (4 bits) from ixl0
ixl1: mem 0xa5000000-0xa5ffffff,0xa7800000-0xa7807fff irq 38 at device 0.1 numa-domain 0 on pci10
ixl1: using 1024 tx descriptors and 1024 rx descriptors
ixl1: fw 3.1.52349 api 1.5 nvm 3.25 etid 800009e7 oem 1.262.0
ixl1: PF-ID[1]: VFs 32, MSIX 129, VF MSIX 5, QPs 768, MDIO shared
ixl1: attempting to allocate 9 MSI-X vectors (129 supported)
msi: routing MSI-X IRQ 276 to local APIC 116 vector 48
msi: routing MSI-X IRQ 277 to local APIC 0 vector 59
msi: routing MSI-X IRQ 278 to local APIC 2 vector 49
msi: routing MSI-X IRQ 279 to local APIC 4 vector 50
msi: routing MSI-X IRQ 280 to local APIC 6 vector 49
msi: routing MSI-X IRQ 281 to local APIC 8 vector 48
msi: routing MSI-X IRQ 282 to local APIC 18 vector 48
msi: routing MSI-X IRQ 283 to local APIC 20 vector 48
msi: routing MSI-X IRQ 284 to local APIC 32 vector 48
ixl1: using IRQs 276-284 for MSI-X
ixl1: Using MSIX interrupts with 9 vectors
ixl1: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl1: bpf attached
ixl1: Ethernet address: XX:XX:XX:XX:XX:49
msi: Assigning MSI-X IRQ 278 to local APIC 1 vector 49
msi: Assigning MSI-X IRQ 279 to local APIC 2 vector 49
msi: Assigning MSI-X IRQ 280 to local APIC 3 vector 49
msi: Assigning MSI-X IRQ 281 to local APIC 4 vector 50
msi: Assigning MSI-X IRQ 282 to local APIC 5 vector 49
msi: Assigning MSI-X IRQ 283 to local APIC 6 vector 49
msi: Assigning MSI-X IRQ 284 to local APIC 7 vector 49
ixl1: SR-IOV ready
...
lagg0: bpf attached
lagg0: link state changed to UP
ixl1: ixl_init_locked: reconfigure MAC addr
vlan20: bpf attached
ixl0: aq_add_macvlan err -53, aq_error 14

Problem/Justification

None

Impact

None

relates to

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Ryan Moeller 
October 12, 2021 at 6:00 PM

I have not been active in the network stack for some time. This will have to be fixed by someone else in FreeBSD.

Matthew Latin 
January 28, 2021 at 7:58 PM
(edited)

I'm having this exact same issue, except I'm using bhyve VMs. The VM can communicate to FreeNAS over the bridge just fine. The bridge does NOT pass traffic outside of it. External devices can communicate directly to FreeNAS's VLAN only when it is not part of a premade bridge. If the IP address is placed on the VLAN itself instead of the bridge, VLAN traffic works. It seems like any time there is a VLAN in a bridge, something in the network stack breaks. See below for my configs:

ifconfig

Alex Rosenberg 
August 30, 2019 at 7:03 PM

Did the upgrade to U5 that weekend, but it didn't resolve the issue. Also, I could not successfully boot with kernel verbose logging.

At this point, I think it's best to just offer to let you have access to my box.

Ryan Moeller 
June 26, 2019 at 12:39 AM

No I don't think anything affecting this has changed in U5.

Alex Rosenberg 
June 25, 2019 at 8:03 PM

Do you have reason to believe that something changed between 11.2 and U5 that would have changed this? If so, can you point to the change? Upgrading is a major effort, which I may be able to try for regression purposes over the weekend.

Third Party to Resolve

Details

Assignee

Reporter

Components

Fix versions

Priority

More fields

Katalon Platform

Created January 16, 2019 at 5:11 PM
Updated July 1, 2022 at 4:26 PM
Resolved October 12, 2021 at 6:00 PM