Linux/mdadm

mdadm
"mdadm is a Linux utility by Neil Brown that is used to manage RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software. Mdadm derives its name from the “md” (multiple disk) device nodes it manages." 

Commands
Show all md devices status: cat /proc/mdstat

Show md device details: mdadm --detail /dev/md0

Create RAID 0 (stripe): mdadm --create --verbose /dev/md0 --level=raid0 --raid-devices=3 /dev/hd[cde]1

Create RAID 5 (stripe with parity): mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[cde]1 mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sd[cdef]1

Add partition to md device: mdadm /dev/md0 --add /dev/sdb1

Fail and remove partition of md device: mdadm /dev/md0 --fail /dev/sdb1 mdadm /dev/md0 --remove /dev/sdb1

Start drive without resync: mdadm -S /dev/md0 mdadm -A --assume-clean /dev/md0

Create /etc/mdadm.conf (high to low detail, any will work) mdadm --detail --scan --verbose > /etc/mdadm.conf mdadm --detail --scan > /etc/mdadm.conf mdadm --examine --scan > /etc/mdadm.conf

sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf >> /etc/mdadm/mdadm.conf sudo mdadm -A /dev/md0
 * 1) ubuntu uses /etc/mdadm/mdadm.conf

Stop array: mdadm --stop /dev/md0

Start (assemble) array: mdadm --assemble /dev/md0

Add disk to RAID 5: unmount /dev/md1 mdadm --add /dev/md1 /dev/sdb3 mdadm --grow --raid-devices=4 /dev/md1 watch -dc cat /proc/mdstat e2fsck -f /dev/md3 resize2fs /dev/md3
 * 1) wait for rebuild...

fdisk to 'Linux raid': type 'fd'

Device Boot     Start         End      Blocks   Id  System /dev/sdb1           2048  1953525167   976761560   fd  Linux raid autodetect

Common mdadm commands
Generate mdadm.conf cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.save /usr/share/mdadm/mkconf --generate > /etc/mdadm/mdadm.conf

Create RAID mdadm --create /dev/md2 --raid-devices=3 --spare-devices=0 --level=5 --run /dev/sd[cde]1

Remove disk from RAID mdadm --fail /dev/md0 /dev/sda1 mdadm --remove /dev/md0 /dev/sda1

Copy the partition structure (when replacing a failed drive) sfdisk -d /dev/sda | sfdisk /dev/sdb mdadm --zero-superblock /dev/sdb

Add a disk to a RAID array (to replace a removed failed drive) mdadm --add /dev/md0 /dev/sdf1

Check RAID status cat /proc/mdstat mdadm --detail /dev/md0

Reassemble a group of RAID disks mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
 * 1) This works to move an assembly from one physical machine to another.

Steps to emulate mdrun (which has been depreciated) mdadm --examine --scan --config=partitions > /tmp/mdadm.conf mdadm --assemble --scan --config=/tmp/mdadm.conf
 * 1) haven't tested this. Use with care

Add a disk to an existing RAID and resize the filesystem mdadm --add /dev/md0 /dev/sdg1 mdadm --grow /dev/md0 -n 5 e2fsck -f /dev/md0 resize2fs /dev/md0 e2fsck -f /dev/md0

Replace all disks in an array with larger drives and resize mdadm --fail /dev/md0 /dev/sda1 mdadm --remove /dev/md0 /dev/sda1 mdadm --add /dev/md0 /dev/sda1
 * 1) For each drive in the existing array
 * 1) physically replace the drive
 * 1) now, wait until md0 is rebuilt.
 * 2) this can literally take days

mdadm --grow /dev/md0 --size=max
 * 1) All drives have been replaced and sync'd, but they still use the original size.
 * 2) Issue the following command to use all available space:

e2fsck -f /dev/md0 && resize2fs /dev/md0 && e2fsck -f /dev/md0 pvresize /dev/md0 ntfsresize /dev/md0
 * 1) Do not forget to resized the file system which sits on the raid set:
 * 2) for ext2/3/4
 * 1) for lvm pv
 * 1) for ntfs
 * 1) note, most likely ntfs is NOT exported as a single partition. In the case
 * 2) of a Xen hvm machine, it is a "disk device" so you will need to resize the
 * 3) partition itself, then resize ntfs.

Stop and remove the RAID device mdadm --stop /dev/md0 mdadm --remove /dev/md0

Destroy an existing array mdadm --manage /dev/md2 --fail /dev/sd[cde]1 mdadm --manage /dev/md2 --remove /dev/sd[cde]1 mdadm --manage /dev/md2 --stop mdadm --zero-superblock /dev/sd[cde]1

Speed up a sync (after drive replacement) cat /proc/sys/dev/raid/speed_limit_max 200000

Rename an existing array mdadm --stop /dev/md127 mdadm -A /dev/md0 -m127 --update=super-minor /dev/sd[bcd]

Source: Linux Server Tech FAQ - Common mdadm commands - http://wiki.linuxservertech.com/index.php?action=artikel&cat=7&id=11&artlang=en

LVM
See LVM

Larger than 2TB RAID
To get larger than 2TB you need GPTs

parted /dev/sdb mklabel gpt print free mkpart primary 1M 4001GB p set 1 raid on  align-check optimal 1 q

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1 mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1

mkfs.ext4 /dev/md0 -L /ci

cat /proc/mdstat

parted -a optimal /dev/sdf

Ref:
 * server - How can I create a RAID array with >2TB disks? - Ask Ubuntu - https://askubuntu.com/questions/350266/how-can-i-create-a-raid-array-with-2tb-disks
 * A guide to mdadm - Linux Raid Wiki - https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm
 * Using parted to create a RAID primary partition — Lucid Solutions - https://plone.lucidsolutions.co.nz/linux/io/using-parted-to-create-a-raid-primary-partition

Handle Drive Failure
Fail device: mdadm /dev/md0 --fail /dev/sdb1   # -f mdadm: set /dev/sdb1 faulty in /dev/md0

Remove failed device: mdadm /dev/md0 --remove /dev/sdb1

Verify failed device: mdadm --detail /dev/md0

Add device to md: mdadm /dev/md0 --add /dev/sdb1   # -a

Verify rebuild: mdadm --detail /dev/md0 cat /proc/mdstat

References:
 * Handling a Drive Failure - http://www.gagme.com/greg/linux/raid-lvm.php#failure

Force Partial Failure Recovery
inactive array (missing 2 devices due to io failure) md4 : inactive sdf3[5] sde3[4] sdd3[3] sdc3[2] 5780957696 blocks

/etc/mdadm.conf ARRAY /dev/md4 level=raid5 num-devices=6 UUID=f0ce1e02:2cd38a68:6ffb704c:fe6e32b0

Stop array: mdadm --stop /dev/md4

Forcefully Rebuild Array: mdadm -A --force /dev/md4 /dev/sd[acdef]3
 * 1) notice 'b' is missing as it is dead dead
 * 2) but 'a' is only partially dead (random io errors)

References:
 * linux - How to get an inactive RAID device working again? - Super User - http://superuser.com/questions/117824/how-to-get-an-inactive-raid-device-working-again

Bring online auto-read-only
cat /proc/mdstat Personalities : [raid1] md1 : active (auto-read-only) raid1 sde1[1] sdd1[0]

Bring online:

mdadm --readwrite /dev/md1

Ref:
 * raid - New md array is auto-read-only and has resync=PENDING - Unix & Linux Stack Exchange - https://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
 * linux - How do I reactivate my MDADM RAID5 array? - Super User - https://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array