Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Slow transfers with iflib(4) (Intel, Broadcom, VMware) and Chelsio NICs

Description

Upgraded from 11.3-U4.1 to 12-RC1, and noticed that both iSCSI transfers and small file transfers via SMB took a huge hit in performance.

SMB for a 25GB photo archive I usually get 250MBps, and iSCSI I get 600MBps.

Large SMB transfers max out at 1GBps. Reads from iSCSI max out at 1GBps, writes at 25MBps.

I was using two laggs for mutlipath iSCSI, but have since removed them and tried assigning the IPs directly to the 10gb Chelsio interfaces. No change, even after reboot. No other configuration items like mtu set at all.

Removed IPs from Chelsio NICs, and re-assigned to the 1GB intel NICs. That drastically improved speeds for SMB and iSCSI, so to me it looks like maybe an issue with the Chelsio.

Problem/Justification

None

Impact

None

is duplicated by

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Alexander Motin February 2, 2021 at 3:48 AM
Edited

I'm sorry, but I doubt you have the same issue, since em(4)/igb(4) driver already has the same interrupt usage as this patch introduced for 10G+ drivers.  Make sure you do not have other network issues (by running `iperf` in both directions) or disk bottlenecks (by running `gstat -I 1s -p`) or CPU bottlenecks (by running `top -SHIz`).  If neither that nor update to coming tomorrow 12.0-U2 release give you anything, then open separate ticket providing there TrueNAS debug and all relevant information, since this ticket is done.

Bojan Milojkovic February 2, 2021 at 2:29 AM

 Same here with Intel igb drivers. I using Intel i210-AT NICs. On FreeNAS 11.3 I had 113-115 MB/s for both read and write (SMB). On TrueNAS 12.0-U1.1 I have 113-115 MB/s for read and only 25MB/s for write.

Alexander Motin January 7, 2021 at 10:18 PM
Edited

No, , those patches are in no way related to Mellanox NICs, only bnxt(4), ice(4), ixgbe(4), ixl(4), vmx(4) and cxgbe(4).  Whatever issue you have is different.  I'd recommend you to create a separate ticket, providing TrueNAS debug archive, full description of your tests and all technical details you have.  I am going to reclassify and close this ticket as fixed.

Christian Rohmann January 7, 2021 at 9:13 PM

any chance this would help with my issues running a 10G Mellanox Connect-X3 card ... (https://jira.ixsystems.com/browse/NAS-107593?focusedCommentId=123642&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-123642)? Would I be worth testing your patch already?

Alexander Motin January 7, 2021 at 7:35 PM

I've just committed the performance fix for iflib(4) (Intel, Broadcom, VMware) drivers and temporary workaround for Chelsio (T4 and up) driver.  Both should be a part of TrueNAS 12.0-U2 update, unless Chelsio provide better fix in time.  Anybody interested can test after updating to 12.0-U1 can replace /boot/kernel directory with the content of attached kernel.tgz archive (don't forget to create boot environment first in case something go wrong).

Complete
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created September 16, 2020 at 4:50 PM
Updated July 1, 2022 at 4:55 PM
Resolved January 7, 2021 at 10:22 PM