Linux/Multipath

Multipath
Turn off multipath: chkconfig multipathd off

This is what is showing up in Knoppix as: /dev/dm-1

Multipath I/O - Wikipedia, the free encyclopedia

Linux Multipathing, Linux Symposium 2005 p. 147

SCSI Rescan
See Linux/SCSI

dmsetup
See Linux/dmsetup

Show wwid
Show device WWID: $ scsi_id -g -u -s /block/sda

Dump all WWID: $ for dev in $( ls /sys/block/ | grep sd ) ; do scsi_id -g -u -s /block/$dev ; done | sort | uniq 360060e80100520e0052faebe00000001 360060e80100520e0052faebe00000002

Blacklisting
[http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/DM_Multipath/config_file_blacklist.html 4.2. Configuration File Blacklist]

By default, all devices are blacklisted, since the following lines appear in the initial configuration file. blacklist { devnode "*" }

Blacklisting by WWID: (World-Wide IDentification) blacklist { wwid 26353900f02796769 }

[http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/DM_Multipath/blacklist_exceptions.html 4.2.4. Blacklist Exceptions]

"You can use the blacklist_exceptions section of the configuration file to enable multipathing on devices that have been blacklisted by default:" blacklist_exceptions { device { vendor "IBM" product "S/390.*" } }

"When specifying devices in the blacklist_exceptions section of the configuration file, you must specify the exceptions in the same way they were specified in the blacklist. For example, a WWID exception will not apply to devices specified by a devnode blacklist entry, even if the blacklisted device is associated with that WWID. Similarly, devnode exceptions apply only to devnode entries, and device exceptions apply only to device entries. "

Citrix Forums : Xen and multipath volumes on a ...

blacklist { wwid "*" devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" } blacklist_exceptions { wwid "36001ec9000dc4e2a000004f14871833e" wwid "36001ec9000dc4e2a00000499486ddf24" }

blacklist { device { vendor "*" product "*" }  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z]" } blacklist_exceptions { device { vendor "DGC" product "*" } }

Red Hat Knowledgebase: How do I setup device-mapper multipathing in Red Hat Enterprise Linux 4?
In order to setup device-mapper-multipathing an updated system with the package device-mapper-multipath is recommended.

This functionality does not exist in releases of Red Hat Enterprise Linux 4 prior to Update 2.

To set up device-mapper multipathing, edit the /etc/multipath.conf file. Comment out the following lines at the top of the file:

devnode_blacklist { devnode "*" }

For example:

# }
 * 1) devnode_blacklist {
 * 2)        devnode "*"

This section keeps device-mapper multipathing from scanning all of the devices. Uncomment the following section of the /etc/multipath.conf file:

defaults { multipath_tool "/sbin/multipath -v0" udev_dir       /dev polling_interval 10 default_selector       "round-robin 0" default_path_grouping_policy   multibus default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n" default_prio_callout   "/bin/true" default_features       "0" rr_min_io             100 failback               immediate } blacklist { wwid 26353900f02796769 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z][[0-9]*]"      devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" }

This sets the default device-mapper behavior and filters out all the devices that are not normally multipathed, such as IDE hard drives and floppy drives. In order to use simple failover behaviour, the default_path_grouping_policy option in the defaults section should be changed to failover as illustrated by the sample below:

defaults { multipath_tool "/sbin/multipath -v0" udev_dir       /dev polling_interval 10 default_selector       "round-robin 0" default_path_grouping_policy   failover default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n" default_prio_callout   "/bin/true" default_features       "0" rr_min_io             100 failback               immediate }

Save the configuration file and exit the editor. Execute the following commands:

modprobe dm-multipath modprobe dm-round-robin service multipathd start multipath -v2

The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed. Execute this command to make sure that the multipath daemon starts on bootup:

chkconfig multipathd on

The devices will be created as /dev/mapper/mpath#, where # is the multipathed group. If /dev/sda is the only multipathed device, /dev/mapper/mpath0 will be the multipathed device for /dev/sda and /dev/sdb. Note that fdisk cannot be used with /dev/dm-# devices. Use fdisk on the underlying disks and execute the following command when device-mapper multipath maps the device to create a /dev/mapper/mpath# device for the partition:

kpartx -a /dev/mapper/mpath#

Note: dmsetup ls --target=multipath is a command that helps determine multipath devices assignments on a system.

If the hardware is not mentioned in the multipathing database, please view the article "How can I add more products into the mutipathing database?" in the Recommended FAQs section.

For an updated sample configuration file, see /usr/share/doc/device-mapper-multipath- /multipath.conf.annotated

Devices
Physical: (fdisk -l) /dev/sda /dev/sdb

Partitions: (fdisk -l) /dev/sda1 /boot /dev/sda2 swap /dev/sda3 / /dev/sdb1 /boot /dev/sdb2 swap /dev/sdb3 /

Mapper: (mount, dmsetup -l) /dev/mapper/mpath0 /dev/mapper/mpath0p1 /boot /dev/mapper/mpath0p2 swap /dev/mapper/mpath0p3 /

dm devices: (fdisk -l) /dev/dm-0p1 /boot /dev/dm-0p2 swap /dev/dm-0p3 / /dev/dm-0 /dev/dm-1 /dev/dm-2 /dev/dm-3
 * 1) but these do not exist under /dev/, but these do:

The wwid matches the Virtual Volume WWN with a prefixed '3'

cat /etc/multipath.conf defaults { user_friendly_names yes }

blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^(hd|xvd)[a-z]*" wwid "*" }


 * 1) Make sure our multipath devices are enabled.

blacklist_exceptions { wwid "350002ac0350003f3" }

multipath -ll: mpath0 (350002ac0350003f3) dm-0 3PARdata,VV [size=12G][features=0][hwhandler=0] \_ round-robin 0 [prio=2][active] \_ 1:0:0:1 sda 8:0  [active][ready] \_ 2:0:0:1 sdc 8:32 [active][ready]

/proc/scsi/scsi: Attached devices: Host: scsi1 Channel: 00 Id: 00 Lun: 01 Vendor: 3PARdata Model: VV              Rev: 0000 Type:  Direct-Access                    ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 02 Vendor: 3PARdata Model: VV              Rev: 0000 Type:  Direct-Access                    ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Id: 00 Lun: 01 Vendor: 3PARdata Model: VV              Rev: 0000 Type:  Direct-Access                    ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Id: 00 Lun: 02 Vendor: 3PARdata Model: VV              Rev: 0000 Type:  Direct-Access                    ANSI SCSI revision: 05

cat /var/lib/multipath/bindings: mpath0 350002ac0350003f3 mpath1 350002ac0350103f3
 * 1) Multipath bindings, Version : 1.0
 * 2) NOTE: this file is automatically maintained by the multipath program.
 * 3) You should not need to edit this file in normal circumstances.
 * 4) This file was automatically generated by anaconda.
 * 5) Format:
 * 6) alias wwid
 * 1) Format:
 * 2) alias wwid

350002ac0350003f3
 * 1) scsi_id -g -s /block/sdc

More Information
Gentoo Linux Documentation -- Multipathing for Gentoo
 * "Multipathing services, generally deployed in enterprise environments, provide a means for high performance, load-balanced, and fault-tolerant data storage either locally or via a storage area network (SAN). Multipathing facilitates a single storage device to be transparently accessed across one or more paths. For example, if there are two connections from a server Host Bus Adapter (HBA) to two Fibre Channel switches and then to a SAN, when the HBA module loads and scans the bus, it will read four paths to the SAN: the paths from the server HBA to and from each Fibre Channel switch and at the storage device. Taking advantage of this situation, Multipath allows you to make use of each path simultaneously or independently to ensure a constant and reliable connection to the data in storage. Multipath serves as a failover for all connections points in the event of losing one path making critical data always available due to redundancy in the design and implementation. "

Linux SAN Multipathing « Thomas Vogt’s IT Blog
 * " There are a lot of SAN multipathing solutions on Linux at the moment. Two of them are discussesed in this blog. The first one is device mapper multipathing that is a failover and load balancing solution with a lot of configuration options. The second one (mdadm multipathing) is just a failover solution with manuel re-anable of a failed path. The advantage of mdadm multiphating is that it is very easy to configure.


 * Before using a multipathing solution for a production environment on Linux it is also important to determine if the used solution is supportet with the used Hardware. For example HP doesn’t support the Device Mapper Multipathing solution on their servers yet."

[http://www.centos.org/docs/5/html/5.2/Virtualization/sect-Virtualization-Virtualized_block_devices-Configuring_persistent_storage_in_a_Red_Hat_Enterprise_Linux_5_environment.html 7.3. Configuring persistent storage in a Red Hat Enterprise Linux 5 environment]
 * "In an environment where external storage (for example, Fibre Channel or iSCSI) is used it is advised to configure persistent device names on your hosts. This will also aid in using Red Hat Virtualization's (live) migration feature to implement consistent device names across multiple systems."

3600a0b800013275100000015427b625e
 * 1) scsi_id -g -s /block/sdc

More Links
RHEL4 dm-multipath on root HOWTO - From the Recliner...

kpartx: Linux Create Device Maps From Partition Tables ( mpath devices for partitions ):

Q. How can I create mpath devices for partitions under CentOS Linux 5?

A. You need to use tool called kpartx. This is derived from util-linux partx package, which reads partition tables on specified device and create device maps over partitions segments detected. It is called from hotplug upon device maps creation and deletion. This is useful for setting multipath which is used to detect multiple paths to devices for fail-over or performance reasons and coalesces them.

To Setup your multipath type the following command at a shell prompt (login as the root user):
 * 1) for d in $(ls /dev/mapper/mpath*); do kpartx -a -v $d ; done

Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3

Gentoo Linux Documentation -- Multipathing for Gentoo

Linux/VM: Enabling root-on-multipath for SLES9 on zSeries

Linux SAN Multipathing « Thomas Vogt’s IT Blog: (GOOD SOURCE)

Procedure for configuring the system with DM-Multipath:
 * 1) Install device-mapper-multipath rpm
 * 2) Edit the multipath.conf configuration file:
 * 3) * comment out the default blacklist
 * 4) * change any of the existing defaults as needed
 * 5) Start the multipath daemons
 * 6) Create the multipath device with the multipath

Right To Your Own Devices LG #114
 * "Linux 2.4.x had the Logical Volume Manager (LVM) and other multi-disk/multi-partition block device constructs. These have been enhanced by the Device Mapper in Linux 2.6.x."

dmsetup(8): low level logical volume management - Linux man page

DMLoop - Device-mapper and LVM2 Wiki

[http://www.linuxquestions.org/questions/red-hat-31/how-to-detect-a-new-disk-with-red-hat-advace-server-3-461814/ How to detect a new disk with red hat advace server 3?? - LinuxQuestions.org] echo "scsi scan-new-devices" > /proc/scsi/scsi

Red Hat Knowledgebase: How do I get the kernel to recognize a hot-added SCSI drive or change the logging/dump detail level for the SCSI sub-system?
 * "You have a SCSI controller and the kernel does not see a newly added SCSI drive"

Symptom: You have a SCSI controller and the kernel does not see a newly added SCSI drive.

Solution: CAUTION: This is not for hotplugging your peripheral hardware. SCSI was not designed for this and you could damage your hardware! That said, it may be possible to switch on an already connected device or add a new drive recently hot-added into hot-plug capable enclosure. However, we make no guarantees, and using these commands could corrupt your data and/or thoroughly confuse your SCSI subsystem. Follow the instructions below at your own risk. Rebooting the system is still the only supported method for adding a new SCSI device to a running system.

Assuming that you have the necessary permissions for writing to /proc/scsi/scsi. The following are some individual commands available for viewing/manipulating scsi controllers and devices:

Channel , Id , and Lun . Will add the device specified so that it is accessible to the system.
 * echo "scsi add-single-device    " > /proc/scsi/scsi where     represents Host ,

Channel , Id , and Lun . Will remove the device specified so that it is no longer accessible to the system.
 * echo "scsi remove-single-device  <C> <I> <L>" > /proc/scsi/scsi where <H> <C> <I> <L> represents Host <H>,


 * echo "scsi scan-new-devices" > /proc/scsi/scsi will scan all host adapters again to see if there are any new devices.

specifing the level of detail for the dump, 0-9 are valid.
 * echo "scsi dump <#>" > /proc/scsi/scsi will dump the status of all current SCSI commands. <#> is the number

scan, mlqueue, mlcomplete, llqueue, llcomplete, hlqueue, or hlcomplete, to the level specified by the <#> number.
 * echo "scsi log <T> <#>" > /proc/scsi/scsi will change the detail level of SCSI logging represented by <T> of error,

Linux Multipath Howto (RHAS4) 1. up2date device-mapper-multipath 2. Edit /etc/multipath.conf For detailed information, see: "SAN Persistent Binding and Multipathing in the 2.6 Kernel" 3. modprobe dm-multipath 4. modprobe dm-round-robin 5. service multipathd start 6. multipath -v2 Will show multipath luns and groups. Look for the multipath group number, this is the dm-# listed in /proc/partitions. The multipath lun is accessed via the /dev/dm-# device entry. 7. Format each SCSI DEVICE: * sfdisk /dev/sdX 8. (Optional) Create multipath devices for each partition: (not needed if using LVM, since you will just mount the logical volume device) * kpartx -a /dev/dm-# 9. Enable multipath to start on bootup: * chkconfig multipathd on

multipath -F
 * Clear all multipath bindings. Useful when making changes to /etc/multipath.conf (clear multipath map, then run multipath to read the config file and build a new map).

multipath -v3 -ll
 * List lots of information about all known disks, what multipath groups they belong to, settings, etc...
 * (NOTE: the -ll also seems to force multipathd to pick up (rescan for) new devices that might have been added to the system but not recognized by mutlipathd yet.)

dmsetup ls --target=multipath
 * Determine multipath device assignments on a system.

Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3

lsmpath
DV=$1 if [ "$DV" = "" ] ; then DVS=$( ls /dev/mapper/mpath? | cut -d 'h' -f 2 ) for dv in $DVS ; do $0 $dv ; done exit 0 fi if [ ! -e /dev/mapper/mpath$DV ] ; then echo "mpath$DV does not exist!" exit 1 fi DEV=$( multipath -ll mpath$DV | tr -s " " | grep "sd[a-z] " | head -n 1 | cut -d " " -f 4 ) SMT=$( smartctl -a /dev/$DEV | grep "Device:" ) SIZE=$( fdisk -l /dev/mapper/mpath$DV 2>/dev/null | grep "Disk /dev/mapper/mpath$DV:" | cut -d " " -f 3-4 | cut -d "," -f 1 )
 * 1) !/bin/sh
 * 2) Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
 * 1) Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
 * 1) DM=$( dmsetup ls | grep "mpath$DV.(" | cut -d ' ' -f 2 | cut -d ')' -f 1 )
 * 2) DEV=$( ls /sys/block/dm-$DM/slaves/ | head -n 1 )
 * 1) VENDOR=$( cat /sys/block/$DEV/device/vendor )
 * 2) MODEL=$( cat /sys/block/$DEV/device/model )
 * 3) SMT=$( echo "$VENDOR - $MODEL" )

echo "mpath$DV ($SIZE) = $SMT"

lsdisks
DVS=$( ls /sys/block/ | grep "sd[a-z]*$" ) for DEV in $DVS ; do VENDOR=$( cat /sys/block/$DEV/device/vendor ) MODEL=$( cat /sys/block/$DEV/device/model ) DEVINFO=$( echo "$VENDOR - $MODEL" ) SIZE=$( fdisk -l /dev/$DEV 2>/dev/null | grep "Disk /dev/$DEV:" | cut -d " " -f 3-4 | cut -d "," -f 1 ) echo "/dev/$DEV = ($SIZE) = $DEVINFO" done
 * 1) !/bin/sh
 * 2) Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
 * 1) Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>

SLES 11 Multipath
Novell Doc: SLES 11: Storage Administration Guide - Configuring Multipath I/O for an Existing Software RAID
 * "Ideally, you should configure multipathing for devices before you use them as components of a software RAID device. If you add multipathing after creating any software RAID devices, the DM-MP service might be starting after the multipath service on reboot, which makes multipathing appear not to be available for RAIDs. You can use the procedure in this section to get multipathing running for a previously existing software RAID."

keywords
linux multipath multi path multipathd kpartx dmsetup