Using proxmox 6.3-3 attempting to add iscsi from a TrueNAS core 12.0-U1.1.
Adding iScsi through iscsiadm works, but it ends up creating TWO devices for the same target, which then has LVM creation fail.
The truenas box works fine to provide iscsi targets to other machines.
scanning shows the diffferent targets served by the box. root@proxmox:~# pvesm scan iscsi 192.168.1.30 iqn.2005-10.org.freenas.ctl:pbs-backup 192.168.1.30:3260 iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks 192.168.1.30:3260 iqn.2005-10.org.freenas.ctl:windows-data 192.168.1.30:3260
However, when adding it, it seems to create one device using the DNS name as the portal name and one device using the IP as the portal name. It doesn't matter whether I specify the DNS name or the IP address as the portal in iscsiadm
root@proxmox:~# iscsiadm --mode node -d 3 --targetname iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks --portal 192.168.1.30 --login iscsiadm: ip 192.168.1.30, port -1, tgpt -1 iscsiadm: Max file limits 1024 1048576 iscsiadm: exec_node_op: : node [iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks,192.168.1.30,3260] sid 0 iscsiadm: default: Creating session 1/1 Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: truenas.theweigels.org,3260] (multiple) iscsiadm: default: Creating session 1/1 Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] (multiple) Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: truenas.theweigels.org,3260] successful. Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] successful.
And trying to create LVM fails. Doing it manually it results in the reasonable error about not being able to use the device as duplicates exist: root@proxmox:~# pvcreate /dev/sde /dev/sdd: open failed: No medium found WARNING: Not using device /dev/sdf for PV aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41. WARNING: PV aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41 prefers device /dev/sde because device was seen first. Cannot use device /dev/sde with duplicates.
multipath on the proxmox box also does not seem to detect multiple routes to the TrueNas box
root@proxmox:~# multipath -v 3 -ll Jan 21 10:24:00 | set open fds limit to 1048576/1048576 Jan 21 10:24:00 | loading //lib/multipath/libchecktur.so checker Jan 21 10:24:00 | checker tur: message table size = 3 Jan 21 10:24:00 | loading //lib/multipath/libprioconst.so prioritizer Jan 21 10:24:00 | foreign library "nvme" loaded successfully Jan 21 10:24:00 | sda: udev property ID_WWN whitelisted Jan 21 10:24:00 | sda: mask = 0x7 Jan 21 10:24:00 | sda: dev_t = 8:0 Jan 21 10:24:00 | sda: size = 976773168 Jan 21 10:24:00 | sda: vendor = ATA Jan 21 10:24:00 | sda: product = Samsung SSD 860 Jan 21 10:24:00 | sda: rev = 3B6Q Jan 21 10:24:00 | sda: h:b:t:l = 0:0:4:0 Jan 21 10:24:00 | sda: tgt_node_name = Jan 21 10:24:00 | sda: path state = running Jan 21 10:24:00 | sda: 60801 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 10:24:00 | sda: serial = S4CMNE0M703486B Jan 21 10:24:00 | sda: get_state Jan 21 10:24:00 | sda: detect_checker = yes (setting: multipath internal) Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9 Jan 21 10:24:00 | sda: path_checker = tur (setting: multipath internal) Jan 21 10:24:00 | sda: checker timeout = 90 s (setting: kernel sysfs) Jan 21 10:24:00 | sda: tur state = up Jan 21 10:24:00 | sdb: udev property ID_WWN whitelisted Jan 21 10:24:00 | sdb: mask = 0x7 Jan 21 10:24:00 | sdb: dev_t = 8:16 Jan 21 10:24:00 | sdb: size = 976773168 Jan 21 10:24:00 | sdb: vendor = ATA Jan 21 10:24:00 | sdb: product = Samsung SSD 860 Jan 21 10:24:00 | sdb: rev = 3B6Q Jan 21 10:24:00 | sdb: h:b:t:l = 0:0:5:0 Jan 21 10:24:00 | sdb: tgt_node_name = Jan 21 10:24:00 | sdb: path state = running Jan 21 10:24:00 | sdb: 60801 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 10:24:00 | sdb: serial = S4CMNE0M703477N Jan 21 10:24:00 | sdb: get_state Jan 21 10:24:00 | sdb: detect_checker = yes (setting: multipath internal) Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9 Jan 21 10:24:00 | sdb: path_checker = tur (setting: multipath internal) Jan 21 10:24:00 | sdb: checker timeout = 90 s (setting: kernel sysfs) Jan 21 10:24:00 | sdb: tur state = up Jan 21 10:24:00 | sdc: udev property ID_WWN whitelisted Jan 21 10:24:00 | sdc: mask = 0x7 Jan 21 10:24:00 | sdc: dev_t = 8:32 Jan 21 10:24:00 | sdc: size = 570949632 Jan 21 10:24:00 | sdc: vendor = DELL Jan 21 10:24:00 | sdc: product = PERC H310 Jan 21 10:24:00 | sdc: rev = 2.12 Jan 21 10:24:00 | sdc: h:b:t:l = 0:2:0:0 Jan 21 10:24:00 | sdc: tgt_node_name = Jan 21 10:24:00 | sdc: path state = running Jan 21 10:24:00 | sdc: 35539 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 10:24:00 | sdc: serial = 0048b250098d003c27000f0f17c0110b Jan 21 10:24:00 | sdc: get_state Jan 21 10:24:00 | sdc: detect_checker = yes (setting: multipath internal) Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9 Jan 21 10:24:00 | sdc: path_checker = tur (setting: multipath internal) Jan 21 10:24:00 | sdc: checker timeout = 90 s (setting: kernel sysfs) Jan 21 10:24:00 | sdc: tur state = up Jan 21 10:24:00 | sdd: blacklisted, udev property missing Jan 21 10:24:00 | sr0: udev property ID_WWN whitelisted Jan 21 10:24:00 | sr0: device node name blacklisted Jan 21 10:24:00 | sde: udev property ID_WWN whitelisted Jan 21 10:24:00 | sde: mask = 0x7 Jan 21 10:24:00 | sde: dev_t = 8:64 Jan 21 10:24:00 | sde: size = 21474836544 Jan 21 10:24:00 | sde: vendor = TrueNAS Jan 21 10:24:00 | sde: product = iSCSI Disk Jan 21 10:24:00 | sde: rev = 0123 Jan 21 10:24:00 | sde: h:b:t:l = 8:0:0:21 Jan 21 10:24:00 | sde: tgt_node_name = iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks Jan 21 10:24:00 | sde: path state = running Jan 21 10:24:00 | sde: 65535 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 10:24:00 | sde: serial = 0cc47a6a5abe003 Jan 21 10:24:00 | sde: get_state Jan 21 10:24:00 | sde: detect_checker = yes (setting: multipath internal) Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9 Jan 21 10:24:00 | sde: path_checker = tur (setting: storage device autodetected) Jan 21 10:24:00 | sde: checker timeout = 30 s (setting: kernel sysfs) Jan 21 10:24:00 | sde: tur state = up Jan 21 10:24:00 | sdf: udev property ID_WWN whitelisted Jan 21 10:24:00 | sdf: mask = 0x7 Jan 21 10:24:00 | sdf: dev_t = 8:80 Jan 21 10:24:00 | sdf: size = 21474836544 Jan 21 10:24:00 | sdf: vendor = TrueNAS Jan 21 10:24:00 | sdf: product = iSCSI Disk Jan 21 10:24:00 | sdf: rev = 0123 Jan 21 10:24:00 | sdf: h:b:t:l = 9:0:0:21 Jan 21 10:24:00 | sdf: tgt_node_name = iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks Jan 21 10:24:00 | sdf: path state = running Jan 21 10:24:00 | sdf: 65535 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 10:24:00 | sdf: serial = 0cc47a6a5abe003 Jan 21 10:24:00 | sdf: get_state Jan 21 10:24:00 | sdf: detect_checker = yes (setting: multipath internal) Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9 Jan 21 10:24:00 | sdf: path_checker = tur (setting: storage device autodetected) Jan 21 10:24:00 | sdf: checker timeout = 30 s (setting: kernel sysfs) Jan 21 10:24:00 | sdf: tur state = up Jan 21 10:24:00 | loop0: blacklisted, udev property missing Jan 21 10:24:00 | loop1: blacklisted, udev property missing Jan 21 10:24:00 | loop2: blacklisted, udev property missing Jan 21 10:24:00 | loop3: blacklisted, udev property missing Jan 21 10:24:00 | loop4: blacklisted, udev property missing Jan 21 10:24:00 | loop5: blacklisted, udev property missing Jan 21 10:24:00 | loop6: blacklisted, udev property missing Jan 21 10:24:00 | loop7: blacklisted, udev property missing ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st 0:0:4:0 sda 8:0 -1 undef undef ATA,Samsung SSD 860 unknown 0:0:5:0 sdb 8:16 -1 undef undef ATA,Samsung SSD 860 unknown 0:2:0:0 sdc 8:32 -1 undef undef DELL,PERC H310 unknown 8:0:0:21 sde 8:64 -1 undef undef TrueNAS,iSCSI Disk unknown 9:0:0:21 sdf 8:80 -1 undef undef TrueNAS,iSCSI Disk unknown Jan 21 10:24:00 | libdevmapper version 1.02.155 (2018-12-18) Jan 21 10:24:00 | DM multipath kernel driver v1.13.0 Jan 21 10:24:00 | unloading const prioritizer Jan 21 10:24:00 | unloading tur checker
Problem/Justification
None
Impact
None
SmartDraw Connector
Katalon Manual Tests (BETA)
Activity
Show:
William Gryzbowski
February 9, 2021 at 12:46 PM
No, I am saying is that this looks like a initiator problem, not a target problem.
Frank Weigel
February 9, 2021 at 6:58 AM
Sorry for delay. Are you saying that it is normal that TrueNAS reports the same target as appearing on two portals? None of the examples I found seemed to show that this is expected behavior.
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] (multiple) Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: truenas.theweigels.org,3260] successful. Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] successful.
William Gryzbowski
January 27, 2021 at 5:21 PM
Hi.
I am confused why you think this is a TrueNAS issue? Especially because it works on other clients?
Using proxmox 6.3-3 attempting to add iscsi from a TrueNAS core 12.0-U1.1.
Adding iScsi through iscsiadm works, but it ends up creating TWO devices for the same target, which then has LVM creation fail.
The truenas box works fine to provide iscsi targets to other machines.
scanning shows the diffferent targets served by the box.
root@proxmox:~# pvesm scan iscsi 192.168.1.30
iqn.2005-10.org.freenas.ctl:pbs-backup 192.168.1.30:3260
iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks 192.168.1.30:3260
iqn.2005-10.org.freenas.ctl:windows-data 192.168.1.30:3260
However, when adding it, it seems to create one device using the DNS name as the portal name and one device using the IP as the portal name. It doesn't matter whether I specify the DNS name or the IP address as the portal in iscsiadm
root@proxmox:~# iscsiadm --mode node -d 3 --targetname iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks --portal 192.168.1.30 --login
iscsiadm: ip 192.168.1.30, port -1, tgpt -1
iscsiadm: Max file limits 1024 1048576
iscsiadm: exec_node_op: : node [iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks,192.168.1.30,3260] sid 0
iscsiadm: default: Creating session 1/1
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: truenas.theweigels.org,3260] (multiple)
iscsiadm: default: Creating session 1/1
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: truenas.theweigels.org,3260] successful.
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks, portal: 192.168.1.30,3260] successful.
And trying to create LVM fails. Doing it manually it results in the reasonable error about not being able to use the device as duplicates exist:
root@proxmox:~# pvcreate /dev/sde
/dev/sdd: open failed: No medium found
WARNING: Not using device /dev/sdf for PV aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41.
WARNING: PV aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41 prefers device /dev/sde because device was seen first.
Cannot use device /dev/sde with duplicates.
root@proxmox:~# lsblk -p -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
/dev/sde LVM2_member aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41
/dev/sdf LVM2_member aqqceH-TDt5-ojEn-Evtl-bjan-sEUd-JL1I41
/dev/sr0
multipath on the proxmox box also does not seem to detect multiple routes to the TrueNas box
root@proxmox:~# multipath -v 3 -ll
Jan 21 10:24:00 | set open fds limit to 1048576/1048576
Jan 21 10:24:00 | loading //lib/multipath/libchecktur.so checker
Jan 21 10:24:00 | checker tur: message table size = 3
Jan 21 10:24:00 | loading //lib/multipath/libprioconst.so prioritizer
Jan 21 10:24:00 | foreign library "nvme" loaded successfully
Jan 21 10:24:00 | sda: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sda: mask = 0x7
Jan 21 10:24:00 | sda: dev_t = 8:0
Jan 21 10:24:00 | sda: size = 976773168
Jan 21 10:24:00 | sda: vendor = ATA
Jan 21 10:24:00 | sda: product = Samsung SSD 860
Jan 21 10:24:00 | sda: rev = 3B6Q
Jan 21 10:24:00 | sda: h:b:t:l = 0:0:4:0
Jan 21 10:24:00 | sda: tgt_node_name =
Jan 21 10:24:00 | sda: path state = running
Jan 21 10:24:00 | sda: 60801 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 10:24:00 | sda: serial = S4CMNE0M703486B
Jan 21 10:24:00 | sda: get_state
Jan 21 10:24:00 | sda: detect_checker = yes (setting: multipath internal)
Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9
Jan 21 10:24:00 | sda: path_checker = tur (setting: multipath internal)
Jan 21 10:24:00 | sda: checker timeout = 90 s (setting: kernel sysfs)
Jan 21 10:24:00 | sda: tur state = up
Jan 21 10:24:00 | sdb: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sdb: mask = 0x7
Jan 21 10:24:00 | sdb: dev_t = 8:16
Jan 21 10:24:00 | sdb: size = 976773168
Jan 21 10:24:00 | sdb: vendor = ATA
Jan 21 10:24:00 | sdb: product = Samsung SSD 860
Jan 21 10:24:00 | sdb: rev = 3B6Q
Jan 21 10:24:00 | sdb: h:b:t:l = 0:0:5:0
Jan 21 10:24:00 | sdb: tgt_node_name =
Jan 21 10:24:00 | sdb: path state = running
Jan 21 10:24:00 | sdb: 60801 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 10:24:00 | sdb: serial = S4CMNE0M703477N
Jan 21 10:24:00 | sdb: get_state
Jan 21 10:24:00 | sdb: detect_checker = yes (setting: multipath internal)
Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9
Jan 21 10:24:00 | sdb: path_checker = tur (setting: multipath internal)
Jan 21 10:24:00 | sdb: checker timeout = 90 s (setting: kernel sysfs)
Jan 21 10:24:00 | sdb: tur state = up
Jan 21 10:24:00 | sdc: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sdc: mask = 0x7
Jan 21 10:24:00 | sdc: dev_t = 8:32
Jan 21 10:24:00 | sdc: size = 570949632
Jan 21 10:24:00 | sdc: vendor = DELL
Jan 21 10:24:00 | sdc: product = PERC H310
Jan 21 10:24:00 | sdc: rev = 2.12
Jan 21 10:24:00 | sdc: h:b:t:l = 0:2:0:0
Jan 21 10:24:00 | sdc: tgt_node_name =
Jan 21 10:24:00 | sdc: path state = running
Jan 21 10:24:00 | sdc: 35539 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 10:24:00 | sdc: serial = 0048b250098d003c27000f0f17c0110b
Jan 21 10:24:00 | sdc: get_state
Jan 21 10:24:00 | sdc: detect_checker = yes (setting: multipath internal)
Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9
Jan 21 10:24:00 | sdc: path_checker = tur (setting: multipath internal)
Jan 21 10:24:00 | sdc: checker timeout = 90 s (setting: kernel sysfs)
Jan 21 10:24:00 | sdc: tur state = up
Jan 21 10:24:00 | sdd: blacklisted, udev property missing
Jan 21 10:24:00 | sr0: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sr0: device node name blacklisted
Jan 21 10:24:00 | sde: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sde: mask = 0x7
Jan 21 10:24:00 | sde: dev_t = 8:64
Jan 21 10:24:00 | sde: size = 21474836544
Jan 21 10:24:00 | sde: vendor = TrueNAS
Jan 21 10:24:00 | sde: product = iSCSI Disk
Jan 21 10:24:00 | sde: rev = 0123
Jan 21 10:24:00 | sde: h:b:t:l = 8:0:0:21
Jan 21 10:24:00 | sde: tgt_node_name = iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks
Jan 21 10:24:00 | sde: path state = running
Jan 21 10:24:00 | sde: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 10:24:00 | sde: serial = 0cc47a6a5abe003
Jan 21 10:24:00 | sde: get_state
Jan 21 10:24:00 | sde: detect_checker = yes (setting: multipath internal)
Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9
Jan 21 10:24:00 | sde: path_checker = tur (setting: storage device autodetected)
Jan 21 10:24:00 | sde: checker timeout = 30 s (setting: kernel sysfs)
Jan 21 10:24:00 | sde: tur state = up
Jan 21 10:24:00 | sdf: udev property ID_WWN whitelisted
Jan 21 10:24:00 | sdf: mask = 0x7
Jan 21 10:24:00 | sdf: dev_t = 8:80
Jan 21 10:24:00 | sdf: size = 21474836544
Jan 21 10:24:00 | sdf: vendor = TrueNAS
Jan 21 10:24:00 | sdf: product = iSCSI Disk
Jan 21 10:24:00 | sdf: rev = 0123
Jan 21 10:24:00 | sdf: h:b:t:l = 9:0:0:21
Jan 21 10:24:00 | sdf: tgt_node_name = iqn.2005-10.org.freenas.ctl:r620proxmox-vmdisks
Jan 21 10:24:00 | sdf: path state = running
Jan 21 10:24:00 | sdf: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 10:24:00 | sdf: serial = 0cc47a6a5abe003
Jan 21 10:24:00 | sdf: get_state
Jan 21 10:24:00 | sdf: detect_checker = yes (setting: multipath internal)
Jan 21 10:24:00 | failed to issue vpd inquiry for pgc9
Jan 21 10:24:00 | sdf: path_checker = tur (setting: storage device autodetected)
Jan 21 10:24:00 | sdf: checker timeout = 30 s (setting: kernel sysfs)
Jan 21 10:24:00 | sdf: tur state = up
Jan 21 10:24:00 | loop0: blacklisted, udev property missing
Jan 21 10:24:00 | loop1: blacklisted, udev property missing
Jan 21 10:24:00 | loop2: blacklisted, udev property missing
Jan 21 10:24:00 | loop3: blacklisted, udev property missing
Jan 21 10:24:00 | loop4: blacklisted, udev property missing
Jan 21 10:24:00 | loop5: blacklisted, udev property missing
Jan 21 10:24:00 | loop6: blacklisted, udev property missing
Jan 21 10:24:00 | loop7: blacklisted, udev property missing
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:0:4:0 sda 8:0 -1 undef undef ATA,Samsung SSD 860 unknown
0:0:5:0 sdb 8:16 -1 undef undef ATA,Samsung SSD 860 unknown
0:2:0:0 sdc 8:32 -1 undef undef DELL,PERC H310 unknown
8:0:0:21 sde 8:64 -1 undef undef TrueNAS,iSCSI Disk unknown
9:0:0:21 sdf 8:80 -1 undef undef TrueNAS,iSCSI Disk unknown
Jan 21 10:24:00 | libdevmapper version 1.02.155 (2018-12-18)
Jan 21 10:24:00 | DM multipath kernel driver v1.13.0
Jan 21 10:24:00 | unloading const prioritizer
Jan 21 10:24:00 | unloading tur checker