[gentoo-user] SAN multipathing

[gentoo-user] SAN multipathing

[Grant Taylor]

Hi,

I'm trying to set up SAN multipathing via dm-multipath for the first 
time in about a decade.

I am seeing the test LUNs (1 x 10 GB and 1 x 100 GB) twice on my 
relatively recent (< 60 days out of date) Gentoo system.  But I'm not 
able to get multipath to see anything.

Before I go too deep I was curious what the current state on Fibre 
Channel multi-path LUNs is in Linux, specifically Gentoo Linux.

The last time I did this was RHEL / CentOS 6.x, probably more than a 
decade ago.



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing

[Joost Roeleveld]

> Hi,
>
> I'm trying to set up SAN multipathing via dm-multipath for the first  
> time in about a decade.
>
> I am seeing the test LUNs (1 x 10 GB and 1 x 100 GB) twice on my  
> relatively recent (<60 days out of date) Gentoo system. But I'm not  
> able to get multipath to see anything.
>
> Before I go too deep I was curious what the current state on Fibre  
> Channel multi-path LUNs is in Linux, specifically Gentoo Linux.
>
> The last time I did this was RHEL / CentOS 6.x, probably more than a  
> decade ago.

I don't use FibreChannel myself (can't justify the cost).
But with SAS drives with dual expanders and 2 HBAs, I do get multipath  
to work.
The OS sees every drive 4 times (2 HBAs, each 2 paths to every drive).
On top of this, I run multipath with the default configuration and use  
the corresponding /dev/mapper/... entries.
I NEVER use the any of the other entries in the /dev/... tree.

Based on that, do you see the fibrechannel drives show up as  
/dev/sd... entries on your system at all?
If yes, then multipath should pick them up.
If not, I would expect you need to get them seen via all the paths first.

--
Joost

Re: [gentoo-user] SAN multipathing

[Grant Taylor]

Hi Joost,

On 6/4/24 12:29 AM, Joost Roeleveld wrote:
> I don't use FibreChannel myself (can't justify the cost).

I hear that.  I manage an old EMC for a friend who's still using it as 
clustered storage for an old VMware install.  It started giving us 
occasional hiccups (corrected memory errors) so we're assembling a cold 
spare.  He's footing the bill and I get to play with the equipment. 
}:-)  Though honestly, the bill ins't that bad any more.

> But with SAS drives with dual expanders and 2 HBAs, I do get multipath 
> to work.

ACK

> The OS sees every drive 4 times (2 HBAs, each 2 paths to every drive).

Four times surprises me.  --  I know that SAS drives have two SAS 
channels / lanes / terms? on them.  But I don't know how those are both 
connected to two controllers.  I would have naively assumed each drive 
showed up as two drives, one for each channel / lane / ???.  I guess if 
there is some sort of multiplexor / breakout / backplane there's a way 
to connect both controllers to both channels / lanes / ??? on each drive 
and four paths would make sense.

Aside:  I'd like to know more about how you're doing that physical 
connection.

> On top of this, I run multipath with the default configuration and use 
> the corresponding /dev/mapper/... entries.

So dev-mapper multipathing is working in a contemporary Gentoo system. 
Thank you for the confirmation.  That's what I was wanting.

I'm going to assume that I have something mis-configured in dev-mapper 
as it wasn't originally used on my test system and only added for the FC 
testing.

> I NEVER use the any of the other entries in the /dev/... tree.

I understand what you're saying.  I'd be following suit if multi-path 
was working.  --  I'm /well/ aware of the hazards of using the member / 
backing disks vs the multi-path disk.

> Based on that, do you see the fibrechannel drives show up as /dev/sd... 
> entries on your system at all?

Yes, both test LUNs (1 x 10 GB and 1 x 100 GB) show up once per client 
HBA port.

Aside:  I don't have the fiber switch in place yet, so I'm using cables 
directly out of an HBA port into an EMC controller port.

I am seeing the expected number of /dev/sd* for the LUNs.

/dev/sdd = 10 GB
/dev/sdf = 100 GB
/dev/sdg = 10 GB
/dev/sdh = 100 GB

> If yes, then multipath should pick them up.

That's what I thought.

As I get into the multipath command and debug things it seems like 
something thinks the paths are offline despite the underlying member / 
backing disks being online and accessible.

> If not, I would expect you need to get them seen via all the paths first.

Yep.  I can see them via their underlying member / backing paths without 
any problem.

This feels like a device-mapper -> multipath issue to me.

It /may/ be complicated by the old EMC but I'd be surprised by that.

Thank you for confirming that device-mapper multipathing should be 
working.  --  Now that I know that I'm not chasing a ghost, I'll spend 
some more time making sure that device-mapepr and multipathing is 
installed and configured properly (USE flags, kernel options, user space 
utilties, etc) before taking another swing at dm-multipath for the SAN LUNs.



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing

[Joost Roeleveld]

<p>Grant,

> On 6/4/24 12:29 AM, Joost Roeleveld wrote:
>> I don't use FibreChannel myself (can't justify the cost).
>
> I hear that. I manage an old EMC for a friend who's still using it  
> as clustered storage for an old VMware install. It started giving us  
> occasional hiccups (corrected memory errors) so we're assembling a  
> cold spare. He's footing the bill and I get to play with the  
> equipment. }:-) Though honestly, the bill ins't that bad any more.
>
>> But with SAS drives with dual expanders and 2 HBAs, I do get  
>> multipath to work.
>
> ACK
>
>> The OS sees every drive 4 times (2 HBAs, each 2 paths to every drive).
>
> Four times surprises me. -- I know that SAS drives have two SAS  
> channels / lanes / terms? on them. But I don't know how those are  
> both connected to two controllers. I would have naively assumed each  
> drive showed up as two drives, one for each channel / lane / ???. I  
> guess if there is some sort of multiplexor / breakout / backplane  
> there's a way to connect both controllers to both channels / lanes /  
> ??? on each drive and four paths would make sense.
>
> Aside: I'd like to know more about how you're doing that physical connection.

I have 2 HBAs.
Each HBA is connected to expander A and expander B (seperate expander  
chips on the same backplane)
Each SAS drive has 2 connections. Connection 1 is connected to  
expander A. 2 is connected to B.
(might be the other way round, but this explains the situation)

That means that each HBA has 2 paths to each drive.
And the OS can use either HBA to each drive.

This totals to 4 times :)

>> On top of this, I run multipath with the default configuration and  
>> use the corresponding /dev/mapper/... entries.
>
> So dev-mapper multipathing is working in a contemporary Gentoo  
> system. Thank you for the confirmation. That's what I was wanting.

Yes, I have dev-mapper multipath support in the kernel:
# zcat /proc/config.gz | grep -i multipath
# CONFIG_NVME_MULTIPATH is not set
CONFIG_DM_MULTIPATH=y
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_MULTIPATH_HST=m
# CONFIG_DM_MULTIPATH_IOA is not set

I installed multipath:
# eix -I multipath
[U] sys-fs/multipath-tools
  Available versions: 0.9.7^t 0.9.7-r1^t{tbz2} 0.9.8^t{tbz2} {systemd test}
  Installed versions: 0.9.7-r1^t{tbz2}(04:27:38 PM 04/10/2024)(-systemd -test)
  Homepage: http://christophe.varoqui.free.fr/
  Description: Device mapper target autoconfig

(I don't update this particular system constantly as it's critical to  
the infrastructure)

I added 'multipath' and 'multipathd' to the default runlevel:
# rc-status | grep multipath
  multipath [ started ]
  multipathd [ started ]

The configfiles:

# cat /etc/multipath.conf
defaults {
  path_grouping_policy multibus
  path_selector "queue-length 0"
  rr_min_io_rq 100
}

# ls /etc/multipath
bindings wwids
san1 ~ # cat /etc/multipath/bindings
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
san1 ~ # cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:

With all this, I got multipath working:

# multipath -l
35000cca0c444c380 dm-11 HGST,HUS726T4TAL5204
size=3.6T features='0' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=0 status=active
  |- 0:0:10:0 sdk 8:160 active undef running
  |- 0:0:35:0 sdai 66:32 active undef running
  |- 1:0:10:0 sdbg 67:160 active undef running
  `- 1:0:35:0 sdce 69:32 active undef running

> I'm going to assume that I have something mis-configured in  
> dev-mapper as it wasn't originally used on my test system and only  
> added for the FC testing.

All the config I have is listed above, hope it helps

>> I NEVER use the any of the other entries in the /dev/... tree.
>
> I understand what you're saying. I'd be following suit if multi-path  
> was working. -- I'm /well/ aware of the hazards of using the member  
> / backing disks vs the multi-path disk.

I just checked and noticed some new entries under by-id:
/dev/disk/by-id/dm-uuid-mpath-....

These didn't exist when I originally set up this system and am not  
going to risk issues by switching.

>> Based on that, do you see the fibrechannel drives show up as  
>> /dev/sd... entries on your system at all?
>
> Yes, both test LUNs (1 x 10 GB and 1 x 100 GB) show up once per  
> client HBA port.
>
> Aside: I don't have the fiber switch in place yet, so I'm using  
> cables directly out of an HBA port into an EMC controller port.
>
> I am seeing the expected number of /dev/sd* for the LUNs.
>
> /dev/sdd = 10 GB
> /dev/sdf = 100 GB
> /dev/sdg = 10 GB
> /dev/sdh = 100 GB

There is 1 thing that might cause issues. If multipath doesn't detect  
"sdd" and "sdg" are the same physical disk, it won't automagically  
link them. On my system, it identifies it due to them having the same  
serial number. For Fiberchannel to something else, you might need to  
configure this.

Can you check the output of the following commands:
cat /sys/block/sdd/device/wwid
cat /sys/block/sdf/device/wwid
cat /sys/block/sdg/device/wwid
cat /sys/block/sdh/device/wwid

For me, using devicenames that are in the same multipath-group, the  
output is identical.
For different groups, I get different outputs.
I am assuming multipath uses this to group them. Example (for 2 of the  
4 entries shown above):

# cat /sys/block/sdai/device/wwid
naa.5000cca0c444c380
# cat /sys/block/sda/device/wwid
naa.5000cca25d8c17ec

>> If yes, then multipath should pick them up.
>
> That's what I thought.
>
> As I get into the multipath command and debug things it seems like  
> something thinks the paths are offline despite the underlying member  
> / backing disks being online and accessible.

Are the devices used somewhere already? Maybe mounted (automounter) or  
LVM or.... ?

>> If not, I would expect you need to get them seen via all the paths first.
>
> Yep. I can see them via their underlying member / backing paths  
> without any problem.
>
> This feels like a device-mapper -> multipath issue to me.
>
> It /may/ be complicated by the old EMC but I'd be surprised by that.

It could. Check the entries in the /sys/... filesystem referenced  
above. That might show a possible cause.
I think there is a way to force multipath to group disks together even  
when those IDs are different. But that is something we'd need to  
investigate.

> Thank you for confirming that device-mapper multipathing should be  
> working. -- Now that I know that I'm not chasing a ghost, I'll spend  
> some more time making sure that device-mapepr and multipathing is  
> installed and configured properly (USE flags, kernel options, user  
> space utilties, etc) before taking another swing at dm-multipath for  
> the SAN LUNs.

Check my config above and we might be able to figure this out.
It's been running stable for me for over 7 years now with most disks  
being that age as well. :)

--
Joost
</list>

Re: [gentoo-user] SAN multipathing

[Grant Taylor]

On 6/4/24 11:03 AM, Joost Roeleveld wrote:
> I have 2 HBAs.
> Each HBA is connected to expander A and expander B (seperate expander 
> chips on the same backplane)
> Each SAS drive has 2 connections. Connection 1 is connected to expander 
> A. 2 is connected to B.
> (might be the other way round, but this explains the situation)
> 
> That means that each HBA has 2 paths to each drive.
> And the OS can use either HBA to each drive.
> 
> This totals to 4 times :)

That explains things.  Thank you for helping me understand your working 
configuration.

> All the config I have is listed above, hope it helps

I hope that it will to.

It definitely gives me things to check and compare against.  Thank you!

> I just checked and noticed some new entries under by-id:
> /dev/disk/by-id/dm-uuid-mpath-....
> 
> These didn't exist when I originally set up this system and am not going 
> to risk issues by switching.

ACK and +10

> There is 1 thing that might cause issues. If multipath doesn't detect 
> "sdd" and "sdg" are the same physical disk, it won't automagically link 
> them. On my system, it identifies it due to them having the same serial 
> number. For Fiberchannel to something else, you might need to configure 
> this.

Agreed.

I believe I checked and found that the two disks for each LUN did have 
the same serial number.

> Can you check the output of the following commands:
> cat /sys/block/sdd/device/wwid
> cat /sys/block/sdf/device/wwid
> cat /sys/block/sdg/device/wwid
> cat /sys/block/sdh/device/wwid

I will when I work on things this weekend.

> For me, using devicenames that are in the same multipath-group, the 
> output is identical.

ACK

> For different groups, I get different outputs.

As it should be.

> I am assuming multipath uses this to group them. Example (for 2 of the 4 
> entries shown above):
> 
> # cat /sys/block/sdai/device/wwid
> naa.5000cca0c444c380
> # cat /sys/block/sda/device/wwid
> naa.5000cca25d8c17ec

Based on the OUI of 00:0c:ca, that looks like HGST (Hitachi?) a Western 
Digital Company according to the recent OUI list I have.

Network Addressing Authority 2, 5, and 6 are interesting.  :-)

> Are the devices used somewhere already? Maybe mounted (automounter) or 
> LVM or.... ?

No.  Not yet.

I did partition one of the backend member LUNs, caused the system to 
release all of the LUNs (echo 1 > /proc or /sys ... /delete), and then 
re-scan (echo "- - -" /proc or /sys ... /host#/scan).

But nothing is actually using the LUNs yet.

> It could. Check the entries in the /sys/... filesystem referenced above. 
> That might show a possible cause.

I'm not finding anything wrong with the backing / member LUNs.

The only problem that I've found is that multipath / multipathd didn't 
like something about the paths when I cranked verbosity / debug way up.

However, this could be because I don't have device-mapper much less 
multi-path installed / configured properly yet.

I added an HBA to my workstation as a test system.  As such I'm adding 
DM & MP MANY months after the system was configured.

> I think there is a way to force multipath to group disks together even 
> when those IDs are different. But that is something we'd need to 
> investigate.

I've found that forcing things like this is usually doesn't work out as 
well as I would like it to.  :-/

> Check my config above and we might be able to figure this out.

I definitely will.

> It's been running stable for me for over 7 years now with most disks 
> being that age as well. :)

Thank you Joost.



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing

[Grant Taylor]

On 6/4/24 11:03, Joost Roeleveld wrote:
> Yes, I have dev-mapper multipath support in the kernel:
> # zcat /proc/config.gz | grep -i multipath
> # CONFIG_NVME_MULTIPATH is not set
> CONFIG_DM_MULTIPATH=y
> CONFIG_DM_MULTIPATH_QL=m
> CONFIG_DM_MULTIPATH_ST=m
> CONFIG_DM_MULTIPATH_HST=m
> # CONFIG_DM_MULTIPATH_IOA is not set

% zgrep "CONFIG_DM_M" /proc/config.gz
# CONFIG_DM_MIRROR is not set
CONFIG_DM_MULTIPATH=y
CONFIG_DM_MULTIPATH_QL=y
CONFIG_DM_MULTIPATH_ST=y
CONFIG_DM_MULTIPATH_HST=y
CONFIG_DM_MULTIPATH_IOA=y

I suspect that integral to the kernel is probably okay.

> I installed multipath:
> # eix -I multipath
> [U] sys-fs/multipath-tools
>   Available versions: 0.9.7^t 0.9.7-r1^t{tbz2} 0.9.8^t{tbz2} {systemd test}
>   Installed versions: 0.9.7-r1^t{tbz2}(04:27:38 PM 04/10/2024)(-systemd 
> -test)
>   Homepage: http://christophe.varoqui.free.fr/
>   Description: Device mapper target autoconfig

I've got sys-fs/multipath-tools-0.9.8 installed.

> I added 'multipath' and 'multipathd' to the default runlevel:
> # rc-status | grep multipath
>   multipath [ started ]
>   multipathd [ started ]

% rc-status | grep multipath
  multipath    ... [  started  ]
  multipathd   ... [  started  ]

> The configfiles:
> 
> # cat /etc/multipath.conf
> defaults {
>   path_grouping_policy multibus
>   path_selector "queue-length 0"
>   rr_min_io_rq 100
> }

% cat /etc/multipath.conf
defaults {
         path_grouping_policy multibus
         path_selector "queue-length 0"
         rr_min_io_rq 100
}

> # ls /etc/multipath
> bindings wwids
> san1 ~ # cat /etc/multipath/bindings
> # Multipath bindings, Version : 1.0
> # NOTE: this file is automatically maintained by the multipath program.
> # You should not need to edit this file in normal circumstances.
> #
> # Format:
> # alias wwid
> #
> san1 ~ # cat /etc/multipath/wwids
> # Multipath wwids, Version : 1.0
> # NOTE: This file is automatically maintained by multipath and multipathd.
> # You should not need to edit this file in normal circumstances.
> #
> # Valid WWIDs:

Same.

> With all this, I got multipath working:
> 
> # multipath -l
> 35000cca0c444c380 dm-11 HGST,HUS726T4TAL5204
> size=3.6T features='0' hwhandler='0' wp=rw
> `-+- policy='queue-length 0' prio=0 status=active
>   |- 0:0:10:0 sdk 8:160 active undef running
>   |- 0:0:35:0 sdai 66:32 active undef running
>   |- 1:0:10:0 sdbg 67:160 active undef running
>   `- 1:0:35:0 sdce 69:32 active undef running

% multipath -l
%

> All the config I have is listed above, hope it helps

I was hoping so too.

> There is 1 thing that might cause issues. If multipath doesn't detect 
> "sdd" and "sdg" are the same physical disk, it won't automagically link 
> them. On my system, it identifies it due to them having the same serial 
> number. For Fiberchannel to something else, you might need to configure 
> this.

N.B. the devices have changed name since rebooting:

% lsblk | grep 0G
sda           8:0    0    10G  0 disk
sdb           8:16   0   100G  0 disk
sde           8:64   0    10G  0 disk
sdf           8:80   0   100G  0 disk

The system boots off of an NVMe.  I have no idea why other SATA disks 
showed up between sda/sdb and sde/sdf.

> Can you check the output of the following commands:
> cat /sys/block/sdd/device/wwid

% cat /sys/block/sda/device/wwid
naa.600601603b1023004677868ab21aef11

% cat /sys/block/sde/device/wwid
naa.600601603b1023004677868ab21aef11

% cat /sys/block/sdb/device/wwid
naa.600601603b1023008c83299ab21aef11

% cat /sys/block/sdf/device/wwid
naa.600601603b1023008c83299ab21aef11

% fgrep 600601603b1023004677868ab21aef11 /sys/block/sd?/device/wwid
/sys/block/sda/device/wwid:naa.600601603b1023004677868ab21aef11
/sys/block/sde/device/wwid:naa.600601603b1023004677868ab21aef11

% fgrep 600601603b1023008c83299ab21aef11 /sys/block/sd?/device/wwid
/sys/block/sdb/device/wwid:naa.600601603b1023008c83299ab21aef11
/sys/block/sdf/device/wwid:naa.600601603b1023008c83299ab21aef11

> It could. Check the entries in the /sys/... filesystem referenced above. 
> That might show a possible cause.

I'm not seeing anything in /sys.

> Check my config above and we might be able to figure this out.

I was genuinely hoping that I missed something.  But I /think/ I covered 
the things that you mentioned.

> It's been running stable for me for over 7 years now with most disks 
> being that age as well. :)

:-)

Here's some diagnostic that I get from multipath when I crank it up to 5.

% multipath -d -v5 |& grep sda
858065.188852 | Discover device 
/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.1/host1/rport-1:0-0/target1:0:0/1:0:0:0/block/sda
858065.189045 | sda: mask = 0x3f
858065.189061 | sda: dev_t = 8:0
858065.189064 | open 
'/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.1/host1/rport-1:0-0/target1:0:0/1:0:0:0/block/sda/size'
858065.189093 | sda: size = 20971520
858065.189306 | sda: vendor = DGC
858065.189340 | sda: product = RAID 10
858065.189361 | sda: rev = 0429
858065.190057 | sda: h:b:t:l = 1:0:0:0
858065.191262 | sda: tgt_node_name = 0x50060160bce01897
858065.191266 | sda: uid_attribute = ID_SERIAL (setting: multipath internal)
858065.191285 | sda: recheck_wwid = 1 (setting: multipath.conf 
defaults/devices section)
858065.191488 | sda: udev property ID_WWN whitelisted
858065.191568 | sda: path state = running
858065.191851 | sda: 10240 cyl, 64 heads, 32 sectors/track, start at 0
858065.191856 | sda: vpd_vendor_id = 0 "undef" (setting: multipath internal)
858065.191899 | sda: serial = APM00085002348
858065.191902 | sda: detect_checker = no (setting: storage device 
configuration)
858065.191952 | sda checker timeout = 30 s (setting: kernel sysfs)
858065.191958 | sda: path_checker = emc_clariion (setting: storage 
device configuration)
858065.192088 | sda: emc_clariion state = down
858065.192112 | sda: emc_clariion checker: Path not correctly configured 
for failover
858065.192117 | sda: uid = 3600601603b1023004677868ab21aef11 (udev)
858065.192122 | sda: detect_prio = yes (setting: multipath internal)
858065.192391 | sda: prio = alua (setting: storage device autodetected)
858065.192395 | sda: prio args = "" (setting: storage device autodetected)
858065.192413 | sda: reported target port group is 1
858065.192571 | sda: aas = 01 [active/non-optimized]
858065.192575 | sda: alua prio = 10
858065.202314 | sda: udev property ID_WWN whitelisted
858065.202319 | checking if sda should be multipathed
858065.202345 | wwid 3600601603b1023004677868ab21aef11 not in wwids 
file, skipping sda
858065.202349 | sda: orphan path, only one path
3600601603b1023004677868ab21aef11 
                           1:0:0:0 sda     8:0   10  undef undef 
DGC,RAID 10                                   unknown

% multipath -d -v5 |& grep sde
858083.764969 | Discover device 
/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.0/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sde
858083.765119 | sde: mask = 0x3f
858083.765123 | sde: dev_t = 8:64
858083.765127 | open 
'/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.0/host0/rport-0:0-0/target0:0:0/0:0:0:0/block/sde/size'
858083.765144 | sde: size = 20971520
858083.765344 | sde: vendor = DGC
858083.765368 | sde: product = RAID 10
858083.765392 | sde: rev = 0429
858083.765947 | sde: h:b:t:l = 0:0:0:0
858083.766697 | sde: tgt_node_name = 0x50060160bce01897
858083.766702 | sde: uid_attribute = ID_SERIAL (setting: multipath internal)
858083.766706 | sde: recheck_wwid = 1 (setting: multipath.conf 
defaults/devices section)
858083.766839 | sde: udev property ID_WWN whitelisted
858083.766927 | sde: path state = running
858083.767178 | sde: 10240 cyl, 64 heads, 32 sectors/track, start at 0
858083.767186 | sde: vpd_vendor_id = 0 "undef" (setting: multipath internal)
858083.767227 | sde: serial = APM00085002348
858083.767234 | sde: detect_checker = no (setting: storage device 
configuration)
858083.767286 | sde checker timeout = 30 s (setting: kernel sysfs)
858083.767449 | sde: path_checker = emc_clariion (setting: storage 
device configuration)
858083.767557 | sde: emc_clariion state = down
858083.767566 | sde: emc_clariion checker: Path not correctly configured 
for failover
858083.767572 | sde: uid = 3600601603b1023004677868ab21aef11 (udev)
858083.767577 | sde: detect_prio = yes (setting: multipath internal)
858083.768138 | sde: prio = alua (setting: storage device autodetected)
858083.768144 | sde: prio args = "" (setting: storage device autodetected)
858083.768175 | sde: reported target port group is 2
858083.768312 | sde: aas = 80 [active/optimized] [preferred]
858083.768320 | sde: alua prio = 50
858083.782976 | sde: udev property ID_WWN whitelisted
858083.782996 | checking if sde should be multipathed
858083.783049 | wwid 3600601603b1023004677868ab21aef11 not in wwids 
file, skipping sde
858083.783052 | sde: orphan path, only one path
3600601603b1023004677868ab21aef11 
                           0:0:0:0 sde     8:64  50  undef undef 
DGC,RAID 10                                   unknown



% multipath -d -v5 |& grep sdb
858087.190831 | Discover device 
/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.1/host1/rport-1:0-0/target1:0:0/1:0:0:1/block/sdb
858087.190989 | sdb: mask = 0x3f
858087.190992 | sdb: dev_t = 8:16
858087.190995 | open 
'/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.1/host1/rport-1:0-0/target1:0:0/1:0:0:1/block/sdb/size'
858087.191011 | sdb: size = 209715200
858087.191198 | sdb: vendor = DGC
858087.191221 | sdb: product = VRAID
858087.191242 | sdb: rev = 0429
858087.191789 | sdb: h:b:t:l = 1:0:0:1
858087.192443 | sdb: tgt_node_name = 0x50060160bce01897
858087.192448 | sdb: uid_attribute = ID_SERIAL (setting: multipath internal)
858087.192452 | sdb: recheck_wwid = 1 (setting: multipath.conf 
defaults/devices section)
858087.192575 | sdb: udev property ID_WWN whitelisted
858087.192619 | sdb: path state = running
858087.192781 | sdb: 13054 cyl, 255 heads, 63 sectors/track, start at 0
858087.192789 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
858087.192818 | sdb: serial = APM00085002348
858087.192824 | sdb: detect_checker = no (setting: storage device 
configuration)
858087.192861 | sdb checker timeout = 30 s (setting: kernel sysfs)
858087.192865 | sdb: path_checker = emc_clariion (setting: storage 
device configuration)
858087.192945 | sdb: emc_clariion state = down
858087.192949 | sdb: emc_clariion checker: Path not correctly configured 
for failover
858087.192954 | sdb: uid = 3600601603b1023008c83299ab21aef11 (udev)
858087.192958 | sdb: detect_prio = yes (setting: multipath internal)
858087.193089 | sdb: prio = alua (setting: storage device autodetected)
858087.193095 | sdb: prio args = "" (setting: storage device autodetected)
858087.193127 | sdb: reported target port group is 1
858087.193198 | sdb: aas = 80 [active/optimized] [preferred]
858087.193203 | sdb: alua prio = 50
858087.201206 | sdb: udev property ID_WWN whitelisted
858087.201210 | checking if sdb should be multipathed
858087.201250 | wwid 3600601603b1023008c83299ab21aef11 not in wwids 
file, skipping sdb
858087.201253 | sdb: orphan path, only one path
3600601603b1023008c83299ab21aef11 
                           1:0:0:1 sdb     8:16  50  undef undef 
DGC,VRAID                                     unknown

% multipath -d -v5 |& grep sdf
858088.602109 | Discover device 
/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.0/host0/rport-0:0-0/target0:0:0/0:0:0:1/block/sdf
858088.602263 | sdf: mask = 0x3f
858088.602267 | sdf: dev_t = 8:80
858088.602271 | open 
'/sys/devices/pci0000:00/0000:00:1d.0/0000:02:00.0/host0/rport-0:0-0/target0:0:0/0:0:0:1/block/sdf/size'
858088.602286 | sdf: size = 209715200
858088.602472 | sdf: vendor = DGC
858088.602494 | sdf: product = VRAID
858088.602515 | sdf: rev = 0429
858088.603042 | sdf: h:b:t:l = 0:0:0:1
858088.603683 | sdf: tgt_node_name = 0x50060160bce01897
858088.603688 | sdf: uid_attribute = ID_SERIAL (setting: multipath internal)
858088.603691 | sdf: recheck_wwid = 1 (setting: multipath.conf 
defaults/devices section)
858088.603805 | sdf: udev property ID_WWN whitelisted
858088.603842 | sdf: path state = running
858088.604249 | sdf: 13054 cyl, 255 heads, 63 sectors/track, start at 0
858088.604260 | sdf: vpd_vendor_id = 0 "undef" (setting: multipath internal)
858088.604325 | sdf: serial = APM00085002348
858088.604331 | sdf: detect_checker = no (setting: storage device 
configuration)
858088.604373 | sdf checker timeout = 30 s (setting: kernel sysfs)
858088.604377 | sdf: path_checker = emc_clariion (setting: storage 
device configuration)
858088.604468 | sdf: emc_clariion state = down
858088.604473 | sdf: emc_clariion checker: Path not correctly configured 
for failover
858088.604478 | sdf: uid = 3600601603b1023008c83299ab21aef11 (udev)
858088.604482 | sdf: detect_prio = yes (setting: multipath internal)
858088.604705 | sdf: prio = alua (setting: storage device autodetected)
858088.604709 | sdf: prio args = "" (setting: storage device autodetected)
858088.604733 | sdf: reported target port group is 2
858088.604899 | sdf: aas = 01 [active/non-optimized]
858088.604906 | sdf: alua prio = 10
858088.618617 | sdf: udev property ID_WWN whitelisted
858088.618622 | checking if sdf should be multipathed
858088.618649 | wwid 3600601603b1023008c83299ab21aef11 not in wwids 
file, skipping sdf
858088.618653 | sdf: orphan path, only one path
3600601603b1023008c83299ab21aef11 
                           0:0:0:1 sdf     8:80  10  undef undef 
DGC,VRAID                                     unknown


The thing that makes me think that this might be EMC CLARiiON specific 
is the following output.

% multipath -d -v5 |& grep emc_clariion
858194.922991 | loading /lib64/multipath/libcheckemc_clariion.so checker
858194.923088 | checker emc_clariion: message table size = 9
858194.923093 | sde: path_checker = emc_clariion (setting: storage 
device configuration)
858194.923175 | sde: emc_clariion state = down
858194.923179 | sde: emc_clariion checker: Path not correctly configured 
for failover
858194.925776 | sdf: path_checker = emc_clariion (setting: storage 
device configuration)
858194.925848 | sdf: emc_clariion state = down
858194.925852 | sdf: emc_clariion checker: Path not correctly configured 
for failover
858194.928303 | sda: path_checker = emc_clariion (setting: storage 
device configuration)
858194.928386 | sda: emc_clariion state = down
858194.928390 | sda: emc_clariion checker: Path not correctly configured 
for failover
858194.930674 | sdb: path_checker = emc_clariion (setting: storage 
device configuration)
858194.930755 | sdb: emc_clariion state = down
858194.930761 | sdb: emc_clariion checker: Path not correctly configured 
for failover
858194.937861 | emc_clariion checker refcount 4
858194.937989 | emc_clariion checker refcount 3
858194.938070 | emc_clariion checker refcount 2
858194.938125 | emc_clariion checker refcount 1
858194.938305 | unloading emc_clariion checker



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing - SUCCESS!!!

[Grant Taylor]

Pre-Script:  It's working.

On 6/4/24 19:48, Grant Taylor wrote:
> The thing that makes me think that this might be EMC CLARiiON specific 
> is the following output.

It turns out that I needed to change the initiator configuration on the 
EMC for the test system to use fail-over mode 4, which I think is also 
known as ALUA.

Now:

% multipath -l
3600601603b1023004677868ab21aef11 dm-0 DGC,RAID 10
size=10G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='queue-length 0' prio=0 status=enabled
| `- 0:0:0:0 sda 8:0  failed undef running
`-+- policy='queue-length 0' prio=0 status=enabled
   `- 1:0:0:0 sdb 8:16 failed undef running
3600601603b1023008c83299ab21aef11 dm-1 DGC,VRAID
size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='queue-length 0' prio=0 status=enabled
| `- 0:0:0:1 sde 8:64 failed undef running
`-+- policy='queue-length 0' prio=0 status=enabled
   `- 1:0:0:1 sdf 8:80 failed undef running

Thank you for your input Joost.  You confirmed that it should be 
working, which was the encouragement that I needed to dig deeper.



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing - SUCCESS!!!

[Grant Taylor]

On 6/4/24 21:02, Grant Taylor wrote:
> It turns out that I needed to change the initiator configuration on the 
> EMC for the test system to use fail-over mode 4, which I think is also 
> known as ALUA.

I was really close, but not quite there.

Now I've made it all the way.

> % multipath -l
> 3600601603b1023004677868ab21aef11 dm-0 DGC,RAID 10
> size=10G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
> |-+- policy='queue-length 0' prio=0 status=enabled
> | `- 0:0:0:0 sda 8:0  failed undef running
> `-+- policy='queue-length 0' prio=0 status=enabled
>    `- 1:0:0:0 sdb 8:16 failed undef running
> 3600601603b1023008c83299ab21aef11 dm-1 DGC,VRAID
> size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
> |-+- policy='queue-length 0' prio=0 status=enabled
> | `- 0:0:0:1 sde 8:64 failed undef running
> `-+- policy='queue-length 0' prio=0 status=enabled
>    `- 1:0:0:1 sdf 8:80 failed undef running

The member paths were both `failed` and `undef`.

It turns out that dm-multipath *NEEDS* -- what EMC calls -- the 
`ArrayCommPath` a.k.a. `LUN Z`.

It seems as if the ACP/LZ is used as part of detecting if a path is 
viable or not.

With ALUA(4) and ACP/LZ, `mulipath -l` shows the following:

% multipath -l
3600601603b1023004677868ab21aef11 dm-1 DGC,RAID 10
size=10G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 0:0:0:1 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
   `- 1:0:0:1 sde 8:64 active ready running
3600601603b1023008c83299ab21aef11 dm-2 DGC,VRAID
size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 1:0:0:2 sdf 8:80 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
   `- 0:0:0:2 sdc 8:32 active ready running

Notice how the paths are now `active` and `ready`.

Without the ACP/LZ, `multipath -l` would show output, which is what made 
me declare success, but I couldn't actually do anything with the LUNs. 
The following command would simply hang:

% fdisk -c -u -l /dev/mapper/36*

While the following command would work:

% fidsk -c -u -l /dev/sd[bcef]

Now, with the ACP/LZ, `multipath -l` shows that paths are `active` & 
`ready` and the following command works:

% fdisk -c -u -l /dev/mapper/36*
Disk /dev/mapper/3600601603b1023004677868ab21aef11: 10 GiB, 10737418240 
bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/3600601603b1023008c83299ab21aef11: 100 GiB, 
107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I had habitually disabled the ACP/LZ because it causes problems for some 
things at work.

Aside:  ArrayCommPath / LUN Z is an EMCism wherein the EMC will present 
a fake LUN, often showing up as LUN Z / LUNZ /if/ there are no LUNs 
assigned to an initiator.

Further aside:  the ACP/LZ becomes an issue when you have multiple EMC 
SANs (VNXs in my case at work) with hosts being zoned to all EMCs but 
only using LUNs off of some of them (migrations, etc).  So we end up 
with LUN Zs showing up on hosts from EMCs that don't have a LUN assigned 
to the host.

Thank you again Joost.  Your encouragement prompted me to dig deeper and 
I'm now exceedingly happy.  :-D



-- 
Grant. . . .

Re: [gentoo-user] SAN multipathing - SUCCESS!!!

[Joost Roeleveld]

> On 6/4/24 21:02, Grant Taylor wrote:
>> It turns out that I needed to change the initiator configuration on  
>> the EMC for the test system to use fail-over mode 4, which I think  
>> is also known as ALUA.
>
> I was really close, but not quite there.
>
> Now I've made it all the way.
>
>> % multipath -l
>> 3600601603b1023004677868ab21aef11 dm-0 DGC,RAID 10
>> size=10G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
>> |-+- policy='queue-length 0' prio=0 status=enabled
>> | `- 0:0:0:0 sda 8:0  failed undef running
>> `-+- policy='queue-length 0' prio=0 status=enabled
>>    `- 1:0:0:0 sdb 8:16 failed undef running
>> 3600601603b1023008c83299ab21aef11 dm-1 DGC,VRAID
>> size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
>> |-+- policy='queue-length 0' prio=0 status=enabled
>> | `- 0:0:0:1 sde 8:64 failed undef running
>> `-+- policy='queue-length 0' prio=0 status=enabled
>>    `- 1:0:0:1 sdf 8:80 failed undef running
>
> The member paths were both `failed` and `undef`.
>
> It turns out that dm-multipath *NEEDS* -- what EMC calls -- the  
> `ArrayCommPath` a.k.a. `LUN Z`.
>
> It seems as if the ACP/LZ is used as part of detecting if a path is  
> viable or not.
>
> With ALUA(4) and ACP/LZ, `mulipath -l` shows the following:
>
> % multipath -l
> 3600601603b1023004677868ab21aef11 dm-1 DGC,RAID 10
> size=10G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
> |-+- policy='service-time 0' prio=50 status=active
> | `- 0:0:0:1 sdb 8:16 active ready running
> `-+- policy='service-time 0' prio=1 status=enabled
>    `- 1:0:0:1 sde 8:64 active ready running
> 3600601603b1023008c83299ab21aef11 dm-2 DGC,VRAID
> size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
> |-+- policy='service-time 0' prio=50 status=active
> | `- 1:0:0:2 sdf 8:80 active ready running
> `-+- policy='service-time 0' prio=1 status=enabled
>    `- 0:0:0:2 sdc 8:32 active ready running
>
> Notice how the paths are now `active` and `ready`.
>
> Without the ACP/LZ, `multipath -l` would show output, which is what  
> made me declare success, but I couldn't actually do anything with  
> the LUNs. The following command would simply hang:
>
> % fdisk -c -u -l /dev/mapper/36*
>
> While the following command would work:
>
> % fidsk -c -u -l /dev/sd[bcef]
>
> Now, with the ACP/LZ, `multipath -l` shows that paths are `active` &  
> `ready` and the following command works:
>
> % fdisk -c -u -l /dev/mapper/36*
> Disk /dev/mapper/3600601603b1023004677868ab21aef11: 10 GiB,  
> 10737418240 bytes, 20971520 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
>
>
> Disk /dev/mapper/3600601603b1023008c83299ab21aef11: 100 GiB,  
> 107374182400 bytes, 209715200 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
>
> I had habitually disabled the ACP/LZ because it causes problems for  
> some things at work.
>
> Aside: ArrayCommPath / LUN Z is an EMCism wherein the EMC will  
> present a fake LUN, often showing up as LUN Z / LUNZ /if/ there are  
> no LUNs assigned to an initiator.
>
> Further aside: the ACP/LZ becomes an issue when you have multiple  
> EMC SANs (VNXs in my case at work) with hosts being zoned to all  
> EMCs but only using LUNs off of some of them (migrations, etc). So  
> we end up with LUN Zs showing up on hosts from EMCs that don't have  
> a LUN assigned to the host.
>
> Thank you again Joost. Your encouragement prompted me to dig deeper  
> and I'm now exceedingly happy.  :-D

Grant,

I am glad to see you got this working.
Some of the terms you mention mean nothing to me :)
But, I'll definitely read up on them.

--
Joost