Solaris Multipath Incorrect Totals

From time to time we notice that some LUN”s are not optimal. It could be because of off-lining a LUN, changes on the switches I am not sure why exactly it happens. If multipath is not showing the correct Path Counts you may need to run cfgadm.

See how some LUN”s here are showing 4 paths only. We expect 8.

1
# mpathadm list lu
2
        /dev/rdsk/c0t5000CCA04385ED60d0s2
3
                Total Path Count: 1
4
                Operational Path Count: 1
5
[..]
6
        /dev/rdsk/c0t600144F09D7311B500005605A40C0006d0s2
7
                Total Path Count: 8
8
                Operational Path Count: 8
9
        /dev/rdsk/c0t600144F09D7311B50000561ED8AB0007d0s2
10
                Total Path Count: 8
11
                Operational Path Count: 8
12
[..]
13
        /dev/rdsk/c0t600144F09D7311B50000538507080021d0s2
14
                Total Path Count: 8
15
                Operational Path Count: 8
16
        /dev/rdsk/c0t600144F09D7311B50000534309C40011d0s2
17
                Total Path Count: 4
18
                Operational Path Count: 4
19
        /dev/rdsk/c0t600144F09D7311B500005342FE86000Fd0s2
20
                Total Path Count: 4
21
                Operational Path Count: 4
22
        /dev/rdsk/c0t600144F09D7311B5000053D13E130029d0s2
23
                Total Path Count: 4
24
                Operational Path Count: 4
25
        /dev/rdsk/c0t600144F09D7311B50000566AE1CC0008d0s2
26
                Total Path Count: 4
27
                Operational Path Count: 4

I tried a few things and it looks like the cfgadm worked. It could also be a couple other thing that triggered it like destroying an unused LUN or changing a recently added LUN”s target group to be more restrictive but I doubt that is it. Most likely cfgadm.

1
# cfgadm -o show_SCSI_LUN -al
2
Ap_Id                          Type         Receptacle   Occupant     Condition
3
c1                             fc           connected    unconfigured unknown
4
c8                             fc-fabric    connected    configured   unknown
5
c8::20520002ac000f02,254       ESI          connected    configured   unknown
6
c8::21000024ff3db11d,0         disk         connected    configured   unknown
7
c8::21000024ff3db11d,1         disk         connected    configured   unknown
8
[..]
9
c8::21000024ff57d646,54        disk         connected    configured   unknown
10
c8::21000024ff57d646,56        disk         connected    configured   unknown
11
c8::21520002ac000f02,254       ESI          connected    configured   unknown
12
c8::22520002ac000f02,254       ESI          connected    configured   unknown
13
c8::23520002ac000f02,254       ESI          connected    configured   unknown
14
c9                             fc           connected    unconfigured unknown
15
c13                            fc-fabric    connected    configured   unknown
16
c13::20510002ac000f02,254      ESI          connected    configured   unknown
17
c13::21000024ff3db11c,0        disk         connected    configured   unknown
18
[..]
19
c13::21000024ff3db11c,44       disk         connected    configured   unknown
20
c13::21000024ff3db11c,46       disk         connected    configured   unknown
21
c13::21000024ff3db11c,48       disk         connected    configured   unknown
22
c13::21000024ff3db11c,50       disk         connected    unconfigured unknown
23
c13::21000024ff3db11c,52       disk         connected    unconfigured unknown
24
c13::21000024ff3db11c,54       disk         connected    unconfigured unknown
25
c13::21000024ff3db11c,56       disk         connected    configured   unknown
26
c13::21000024ff3db1b4,0        disk         connected    configured   unknown
27
c13::21000024ff3db1b4,1        disk         connected    configured   unknown
28
[..]
29
c13::21510002ac000f02,254      ESI          connected    configured   unknown
30
c13::22510002ac000f02,254      ESI          connected    configured   unknown
31
c13::23510002ac000f02,254      ESI          connected    configured   unknown

After I ran cfgadm..

1
# mpathadm list lu
2
        /dev/rdsk/c0t5000CCA04385ED60d0s2
3
                Total Path Count: 1
4
                Operational Path Count: 1
5
[..]
6
        /dev/rdsk/c0t600144F09D7311B500005605A40C0006d0s2
7
                Total Path Count: 8
8
                Operational Path Count: 8
9
        /dev/rdsk/c0t600144F09D7311B50000561ED8AB0007d0s2
10
                Total Path Count: 8
11
                Operational Path Count: 8
12
[..]
13
        /dev/rdsk/c0t600144F09D7311B5000053BE90620024d0s2
14
                Total Path Count: 8
15
                Operational Path Count: 8
16
        /dev/rdsk/c0t600144F09D7311B50000533C012A0009d0s2
17
                Total Path Count: 8
18
                Operational Path Count: 8
19
        /dev/rdsk/c0t600144F09D7311B50000533AAFF00007d0s2
20
                Total Path Count: 8
21
                Operational Path Count: 8
22
        /dev/rdsk/c0t600144F09D7311B50000538507080021d0s2
23
                Total Path Count: 8
24
                Operational Path Count: 8
25
        /dev/rdsk/c0t600144F09D7311B50000534309C40011d0s2
26
                Total Path Count: 8
27
                Operational Path Count: 8
28
        /dev/rdsk/c0t600144F09D7311B500005342FE86000Fd0s2
29
                Total Path Count: 8
30
                Operational Path Count: 8
31
        /dev/rdsk/c0t600144F09D7311B5000053D13E130029d0s2
32
                Total Path Count: 8
33
                Operational Path Count: 8
34
        /dev/rdsk/c0t600144F09D7311B50000566AE1CC0008d0s2
35
                Total Path Count: 8
36
                Operational Path Count: 8

I can also see the changes in messages…

1
# dmesg
2
Nov  9 03:59:21 solaris11 mac: [ID 469746 kern.info] NOTICE: ldoms-vsw0.vport15 registered
3
Nov  9 05:41:55 solaris11 scsi: [ID 583861 kern.info] ssd98 at scsi_vhci0: unit-address g600144f09d7311b50000534309c40011: f_tpgs
4
[..]
5
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 343 fp16/ssd@w21000024ff3db1b5,38 is standby
6
Dec 14 09:38:01 solaris11 last message repeated 1 time
7
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 344 fp16/ssd@w21000024ff3db1b5,36 is standby
8
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 345 fp16/ssd@w21000024ff3db1b5,34 is standby
9
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 346 fp16/ssd@w21000024ff3db1b5,32 is standby
10
Dec 14 09:38:07 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 347 fp16/ssd@w21000024ff3db11d,38 is standby
11
Dec 14 09:38:07 solaris11 last message repeated 1 time
12
Dec 14 09:38:07 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 348 fp16/ssd@w21000024ff3db11d,36 is standby
13
Dec 14 09:38:08 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 349 fp16/ssd@w21000024ff3db11d,34 is standby
14
Dec 14 09:38:08 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 350 fp16/ssd@w21000024ff3db11d,32 is standby
15
Dec 14 09:38:16 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 351 fp21/ssd@w21000024ff3db1b4,38 is standby
16
Dec 14 09:38:16 solaris11 last message repeated 1 time
17
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 352 fp21/ssd@w21000024ff3db1b4,36 is standby
18
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 353 fp21/ssd@w21000024ff3db1b4,34 is standby
19
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 354 fp21/ssd@w21000024ff3db1b4,32 is standby
20
Dec 14 09:38:22 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 355 fp21/ssd@w21000024ff3db11c,38 is standby
21
Dec 14 09:38:22 solaris11 last message repeated 1 time
22
Dec 14 09:38:22 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 356 fp21/ssd@w21000024ff3db11c,36 is standby
23
Dec 14 09:38:23 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 357 fp21/ssd@w21000024ff3db11c,34 is standby
24
Dec 14 09:38:23 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 358 fp21/ssd@w21000024ff3db11c,32 is standby