Linux/Multipath

From Omnia
Revision as of 18:58, 25 June 2014 by Kenneth (talk | contribs) (→‎SCSI Rescan)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Multipath

Turn off multipath:

chkconfig multipathd off

This is what is showing up in Knoppix as:

/dev/dm-1

Multipath I/O - Wikipedia, the free encyclopedia


Linux Multipathing, Linux Symposium 2005 p. 147

SCSI Rescan

See Linux/SCSI

dmsetup

See Linux/dmsetup

Show wwid

Show device WWID:

$ scsi_id -g -u -s /block/sda

Dump all WWID:

$ for dev in $( ls /sys/block/ | grep sd ) ; do scsi_id -g -u -s /block/$dev ; done | sort | uniq
360060e80100520e0052faebe00000001
360060e80100520e0052faebe00000002

Blacklisting

4.2. Configuration File Blacklist

By default, all devices are blacklisted, since the following lines appear in the initial configuration file.

blacklist {
        devnode "*"
}

Blacklisting by WWID: (World-Wide IDentification)

blacklist {
       wwid 26353900f02796769
}


4.2.4. Blacklist Exceptions

"You can use the blacklist_exceptions section of the configuration file to enable multipathing on devices that have been blacklisted by default:"

blacklist_exceptions {
       device {
               vendor  "IBM"
               product "S/390.*"
       }
}

"When specifying devices in the blacklist_exceptions section of the configuration file, you must specify the exceptions in the same way they were specified in the blacklist. For example, a WWID exception will not apply to devices specified by a devnode blacklist entry, even if the blacklisted device is associated with that WWID. Similarly, devnode exceptions apply only to devnode entries, and device exceptions apply only to device entries. "


Citrix Forums : Xen and multipath volumes on a ...

blacklist {
  wwid "*"
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
}
blacklist_exceptions {
  wwid "36001ec9000dc4e2a000004f14871833e"
  wwid "36001ec9000dc4e2a00000499486ddf24"
}


blacklist {
  device {
    vendor "*"
    product "*"
  }
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
}
blacklist_exceptions {
  device {
    vendor "DGC"
    product "*"
  }
}

Red Hat Knowledgebase: How do I setup device-mapper multipathing in Red Hat Enterprise Linux 4?

In order to setup device-mapper-multipathing an updated system with the package device-mapper-multipath is recommended.

This functionality does not exist in releases of Red Hat Enterprise Linux 4 prior to Update 2.

To set up device-mapper multipathing, edit the /etc/multipath.conf file. Comment out the following lines at the top of the file:

devnode_blacklist {
        devnode "*"
}

For example:

# devnode_blacklist {
#        devnode "*"
# }

This section keeps device-mapper multipathing from scanning all of the devices. Uncomment the following section of the /etc/multipath.conf file:

defaults {
       multipath_tool  "/sbin/multipath -v0"
       udev_dir        /dev
       polling_interval 10
       default_selector        "round-robin 0"
       default_path_grouping_policy    multibus
       default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
       default_prio_callout    "/bin/true"
       default_features        "0"
       rr_min_io              100
       failback                immediate
}
blacklist {
       wwid 26353900f02796769
       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
       devnode "^hd[a-z][[0-9]*]"
       devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

This sets the default device-mapper behavior and filters out all the devices that are not normally multipathed, such as IDE hard drives and floppy drives. In order to use simple failover behaviour, the default_path_grouping_policy option in the defaults section should be changed to failover as illustrated by the sample below:

defaults {
       multipath_tool  "/sbin/multipath -v0"
       udev_dir        /dev
       polling_interval 10
       default_selector        "round-robin 0"
       default_path_grouping_policy    failover
       default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
       default_prio_callout    "/bin/true"
       default_features        "0"
       rr_min_io              100
       failback                immediate
}

Save the configuration file and exit the editor. Execute the following commands:

modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath -v2

The multipath -v2 command prints out multipathed paths that show which devices are multipathed. If the command does not print anything out, ensure that all SAN connections are set up properly and the system is multipathed. Execute this command to make sure that the multipath daemon starts on bootup:

chkconfig multipathd on

The devices will be created as /dev/mapper/mpath#, where # is the multipathed group. If /dev/sda is the only multipathed device, /dev/mapper/mpath0 will be the multipathed device for /dev/sda and /dev/sdb. Note that fdisk cannot be used with /dev/dm-# devices. Use fdisk on the underlying disks and execute the following command when device-mapper multipath maps the device to create a /dev/mapper/mpath# device for the partition:

kpartx -a /dev/mapper/mpath#

Note: dmsetup ls --target=multipath is a command that helps determine multipath devices assignments on a system.

If the hardware is not mentioned in the multipathing database, please view the article "How can I add more products into the mutipathing database?" in the Recommended FAQs section.

For an updated sample configuration file, see /usr/share/doc/device-mapper-multipath-<version>/multipath.conf.annotated

Devices

Physical: (fdisk -l)

/dev/sda

/dev/sdb

Partitions: (fdisk -l)

/dev/sda1  /boot
/dev/sda2  swap
/dev/sda3  /

/dev/sdb1  /boot
/dev/sdb2  swap
/dev/sdb3  /

Mapper: (mount, dmsetup -l)

/dev/mapper/mpath0

/dev/mapper/mpath0p1  /boot
/dev/mapper/mpath0p2  swap
/dev/mapper/mpath0p3  /

dm devices: (fdisk -l)

/dev/dm-0p1  /boot
/dev/dm-0p2  swap
/dev/dm-0p3  /

#but these do not exist under /dev/, but these do:
/dev/dm-0
/dev/dm-1
/dev/dm-2
/dev/dm-3

The wwid matches the Virtual Volume WWN with a prefixed '3'

cat /etc/multipath.conf
defaults {
        user_friendly_names yes
}

blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(hd|xvd)[a-z]*"
        wwid "*"
}

# Make sure our multipath devices are enabled.

blacklist_exceptions {
        wwid "350002ac0350003f3"
}

multipath -ll:

mpath0 (350002ac0350003f3) dm-0 3PARdata,VV
[size=12G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:0:1 sda 8:0   [active][ready]
 \_ 2:0:0:1 sdc 8:32  [active][ready]

/proc/scsi/scsi:

Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 01
  Vendor: 3PARdata Model: VV               Rev: 0000
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 02
  Vendor: 3PARdata Model: VV               Rev: 0000
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 01
  Vendor: 3PARdata Model: VV               Rev: 0000
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 02
  Vendor: 3PARdata Model: VV               Rev: 0000
  Type:   Direct-Access                    ANSI SCSI revision: 05

cat /var/lib/multipath/bindings:

# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# This file was automatically generated by anaconda.
#
# Format:
# alias wwid
#
mpath0 350002ac0350003f3
mpath1 350002ac0350103f3
  1. scsi_id -g -s /block/sdc
350002ac0350003f3

More Information

Gentoo Linux Documentation -- Multipathing for Gentoo

"Multipathing services, generally deployed in enterprise environments, provide a means for high performance, load-balanced, and fault-tolerant data storage either locally or via a storage area network (SAN). Multipathing facilitates a single storage device to be transparently accessed across one or more paths. For example, if there are two connections from a server Host Bus Adapter (HBA) to two Fibre Channel switches and then to a SAN, when the HBA module loads and scans the bus, it will read four paths to the SAN: the paths from the server HBA to and from each Fibre Channel switch and at the storage device. Taking advantage of this situation, Multipath allows you to make use of each path simultaneously or independently to ensure a constant and reliable connection to the data in storage. Multipath serves as a failover for all connections points in the event of losing one path making critical data always available due to redundancy in the design and implementation. "

Linux SAN Multipathing « Thomas Vogt’s IT Blog

" There are a lot of SAN multipathing solutions on Linux at the moment. Two of them are discussesed in this blog. The first one is device mapper multipathing that is a failover and load balancing solution with a lot of configuration options. The second one (mdadm multipathing) is just a failover solution with manuel re-anable of a failed path. The advantage of mdadm multiphating is that it is very easy to configure.
Before using a multipathing solution for a production environment on Linux it is also important to determine if the used solution is supportet with the used Hardware. For example HP doesn’t support the Device Mapper Multipathing solution on their servers yet."

7.3. Configuring persistent storage in a Red Hat Enterprise Linux 5 environment

"In an environment where external storage (for example, Fibre Channel or iSCSI) is used it is advised to configure persistent device names on your hosts. This will also aid in using Red Hat Virtualization's (live) migration feature to implement consistent device names across multiple systems."
# scsi_id -g -s /block/sdc 
3600a0b800013275100000015427b625e

More Links

RHEL4 dm-multipath on root HOWTO - From the Recliner...


kpartx: Linux Create Device Maps From Partition Tables ( mpath devices for partitions ):

Q. How can I create mpath devices for partitions under CentOS Linux 5?

A. You need to use tool called kpartx. This is derived from util-linux partx package, which reads partition tables on specified device and create device maps over partitions segments detected. It is called from hotplug upon device maps creation and deletion. This is useful for setting multipath which is used to detect multiple paths to devices for fail-over or performance reasons and coalesces them.

To Setup your multipath type the following command at a shell prompt (login as the root user):

# for d in $(ls /dev/mapper/mpath*); do kpartx -a -v $d ; done


Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3

Gentoo Linux Documentation -- Multipathing for Gentoo

Linux/VM: Enabling root-on-multipath for SLES9 on zSeries


Linux SAN Multipathing « Thomas Vogt’s IT Blog: (GOOD SOURCE)

Procedure for configuring the system with DM-Multipath:

  1. Install device-mapper-multipath rpm
  2. Edit the multipath.conf configuration file:
    • comment out the default blacklist
    • change any of the existing defaults as needed
  3. Start the multipath daemons
  4. Create the multipath device with the multipath


Right To Your Own Devices LG #114

"Linux 2.4.x had the Logical Volume Manager (LVM) and other multi-disk/multi-partition block device constructs. These have been enhanced by the Device Mapper in Linux 2.6.x."

dmsetup(8): low level logical volume management - Linux man page

DMLoop - Device-mapper and LVM2 Wiki


How to detect a new disk with red hat advace server 3?? - LinuxQuestions.org

echo "scsi scan-new-devices" > /proc/scsi/scsi


Red Hat Knowledgebase: How do I get the kernel to recognize a hot-added SCSI drive or change the logging/dump detail level for the SCSI sub-system?

"You have a SCSI controller and the kernel does not see a newly added SCSI drive"

Symptom:
You have a SCSI controller and the kernel does not see a newly added SCSI drive.

Solution:
CAUTION: This is not for hotplugging your peripheral hardware. SCSI was not designed for this and you could damage your hardware! That said, it may be possible to switch on an already connected device or add a new drive recently hot-added into hot-plug capable enclosure. However, we make no guarantees, and using these commands could corrupt your data and/or thoroughly confuse your SCSI subsystem. Follow the instructions below at your own risk. Rebooting the system is still the only supported method for adding a new SCSI device to a running system.

Assuming that you have the necessary permissions for writing to /proc/scsi/scsi. The following are some individual commands available for viewing/manipulating scsi controllers and devices:

* echo "scsi add-single-device <H> <C> <I> <L>" > /proc/scsi/scsi where <H> <C> <I> <L> represents Host <H>,
 Channel <C>, Id <I>, and Lun <L>. Will add the device specified so that it is accessible to the system.

* echo "scsi remove-single-device <H> <C> <I> <L>" > /proc/scsi/scsi where <H> <C> <I> <L> represents Host <H>,
 Channel <C>, Id <I>, and Lun <L>. Will remove the device specified so that it is no longer accessible to the system.

* echo "scsi scan-new-devices" > /proc/scsi/scsi will scan all host adapters again to see if there are any new devices.

* echo "scsi dump <#>" > /proc/scsi/scsi will dump the status of all current SCSI commands. <#> is the number
 specifing the level of detail for the dump, 0-9 are valid.

* echo "scsi log <T> <#>" > /proc/scsi/scsi will change the detail level of SCSI logging represented by <T> of error,
 scan, mlqueue, mlcomplete, llqueue, llcomplete, hlqueue, or hlcomplete, to the level specified by the <#> number. 

Linux Multipath Howto (RHAS4)

1. up2date device-mapper-multipath
2. Edit /etc/multipath.conf
   For detailed information, see: "SAN Persistent Binding and Multipathing in the 2.6 Kernel"
3. modprobe dm-multipath
4. modprobe dm-round-robin
5. service multipathd start
6. multipath -v2
   Will show multipath luns and groups. Look for the multipath group number, this is the dm-# listed
   in /proc/partitions. The multipath lun is accessed via the /dev/dm-# device entry.
7. Format each SCSI DEVICE:
   * sfdisk /dev/sdX
8. (Optional) Create multipath devices for each partition:
   (not needed if using LVM, since you will just mount the logical volume device)
   * kpartx -a /dev/dm-#
9. Enable multipath to start on bootup:
  * chkconfig multipathd on

multipath -F

Clear all multipath bindings. Useful when making changes to /etc/multipath.conf (clear multipath map, then run multipath to read the config file and build a new map).

multipath -v3 -ll

List lots of information about all known disks, what multipath groups they belong to, settings, etc...
(NOTE: the -ll also seems to force multipathd to pick up (rescan for) new devices that might have been added to the system but not recognized by mutlipathd yet.)

dmsetup ls --target=multipath

Determine multipath device assignments on a system.


Multipath-usage.txt File for Red Hat Enterprise Linux 4 Update 3

Tools

lsmpath

#!/bin/sh
#
# Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
#
DV=$1
if [ "$DV" = "" ] ; then
  DVS=$( ls /dev/mapper/mpath? | cut -d 'h' -f 2 )
  for dv in $DVS ; do $0 $dv ; done
  exit 0
fi
if [ ! -e /dev/mapper/mpath$DV ] ; then
  echo "mpath$DV does not exist!"
  exit 1
fi
#DM=$( dmsetup ls | grep "mpath$DV.(" | cut -d ' ' -f 2 | cut -d ')' -f 1 )
#DEV=$( ls /sys/block/dm-$DM/slaves/ | head -n 1 )
DEV=$( multipath -ll mpath$DV | tr -s " " | grep "sd[a-z] " | head -n 1 | cut -d " " -f 4 )
#VENDOR=$( cat /sys/block/$DEV/device/vendor )
#MODEL=$( cat /sys/block/$DEV/device/model )
#SMT=$( echo "$VENDOR - $MODEL" )
SMT=$( smartctl -a /dev/$DEV | grep "Device:" )
SIZE=$( fdisk -l /dev/mapper/mpath$DV 2>/dev/null | grep "Disk /dev/mapper/mpath$DV:" | cut -d " " -f 3-4 | cut -d "," -f 1 )

echo "mpath$DV ($SIZE) = $SMT"

lsdisks

#!/bin/sh
#
# Programmed by Kenneth Burgener <kenneth.burgener@applabs.com>
#
DVS=$( ls /sys/block/ | grep "sd[a-z]*$" )
for DEV in $DVS ; do
  VENDOR=$( cat /sys/block/$DEV/device/vendor )
  MODEL=$( cat /sys/block/$DEV/device/model )
  DEVINFO=$( echo "$VENDOR - $MODEL" )
  SIZE=$( fdisk -l /dev/$DEV 2>/dev/null | grep "Disk /dev/$DEV:" | cut -d " " -f 3-4 | cut -d "," -f 1 )
  echo "/dev/$DEV = ($SIZE) = $DEVINFO"
done

SLES 11 Multipath

Novell Doc: SLES 11: Storage Administration Guide - Configuring Multipath I/O for an Existing Software RAID

"Ideally, you should configure multipathing for devices before you use them as components of a software RAID device. If you add multipathing after creating any software RAID devices, the DM-MP service might be starting after the multipath service on reboot, which makes multipathing appear not to be available for RAIDs. You can use the procedure in this section to get multipathing running for a previously existing software RAID."

keywords

linux multipath multi path multipathd kpartx dmsetup

[[Category::Linux]]