How to add Hot Spare Volume to the existing mdadm software RAID array

Having a Hot Spare significantly increases the security of our data in a RAID array. In the case of a single disk failure, a Hot Spare jumps in the place of a faulty drive. Working as a temporary replacement, a Hot Spare can potentially buy time for us before we swap the faulty drive with the new one.

In the below scenario we have an example CentOS 7, installed on top of RAID 1 (mirror) using mdadm software RAID. The array was created by the Anaconda installer script during OS installation, for more details refer to:
CentOS 7 Installation with LVM RAID 1 – Mirroring.

The array consists of two physical disks: /dev/sda and /dev/sdb and two personalities: /dev/md127 (for standard /boot partition) and /dev/md126 (for LVM Physical Volume):

[root@compute ~]# cat /proc/mdstat
Personalities : [raid1] 
md126 : active raid1 sda2[0] sdb2[1]
      142191616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/2 pages [4KB], 65536KB chunk

md127 : active raid1 sda1[0] sdb1[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: 

Partition layout:

[root@compute ~]# lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0 136.8G  0 disk  
├─sda1              8:1    0     1G  0 part  
│ └─md127           9:127  0  1023M  0 raid1 /boot
└─sda2              8:2    0 135.7G  0 part  
  └─md126           9:126  0 135.6G  0 raid1 
    ├─centos-root 253:0    0    50G  0 lvm   /
    └─centos-swap 253:1    0    16G  0 lvm   [SWAP]
sdb                 8:16   0 136.8G  0 disk  
├─sdb1              8:17   0     1G  0 part  
│ └─md127           9:127  0  1023M  0 raid1 /boot
└─sdb2              8:18   0 135.7G  0 part  
  └─md126           9:126  0 135.6G  0 raid1 
    ├─centos-root 253:0    0    50G  0 lvm   /
    └─centos-swap 253:1    0    16G  0 lvm   [SWAP]
sdc                 8:32   0 136.8G  0 disk  
sdd                 8:48   0 232.9G  0 disk

Now we want to add /dev/sdc as a Hot Spare volume.

Steps:

1. Copy partition table from the active physical volume in RAID array (i.e. /dev/sda) to the spare volume

[root@compute ~]# sfdisk -d /dev/sda | sfdisk /dev/sdc

Spare volume /dev/sdc should have now the same partition table as /dev/sda or /dev/sdb:

[root@compute ~]# lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0 136.8G  0 disk  
├─sda1              8:1    0     1G  0 part  
│ └─md127           9:127  0  1023M  0 raid1 /boot
└─sda2              8:2    0 135.7G  0 part  
  └─md126           9:126  0 135.6G  0 raid1 
    ├─centos-root 253:0    0    50G  0 lvm   /
    └─centos-swap 253:1    0    16G  0 lvm   [SWAP]
sdb                 8:16   0 136.8G  0 disk  
├─sdb1              8:17   0     1G  0 part  
│ └─md127           9:127  0  1023M  0 raid1 /boot
└─sdb2              8:18   0 135.7G  0 part  
  └─md126           9:126  0 135.6G  0 raid1 
    ├─centos-root 253:0    0    50G  0 lvm   /
    └─centos-swap 253:1    0    16G  0 lvm   [SWAP]
sdc                 8:32   0 136.8G  0 disk  
├─sdc1              8:33   0     1G  0 part  
└─sdc2              8:34   0 135.7G  0 part  
sdd                 8:48   0 232.9G  0 disk

2. Add suitable spare volume partitions /dev/sdc1 and /dev/sdc2 to the corresponding personalities /dev/md127 and /dev/md126

[root@compute ~]# mdadm --add /dev/md127 /dev/sdc1
mdadm: added /dev/sdc1

[root@compute ~]# mdadm --add /dev/md126 /dev/sdc2
mdadm: added /dev/sdc2

3. Verify the configuration

Hot Spare volume partitions /dev/sdc1 and /dev/sdc2 are now present in corresponding personalities with the (S) mark:

[root@compute ~]# cat /proc/mdstat
Personalities : [raid1] 
md126 : active raid1 sdc2[2](S) sda2[0] sdb2[1]
      142191616 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md127 : active raid1 sdc1[2](S) sda1[0] sdb1[1]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: 

Spare volumes can also be verified using mdadm command:

[root@compute ~]# mdadm --detail /dev/md127 | head -n 20
/dev/md127:
           Version : 1.2
     Creation Time : Mon Dec 28 23:04:15 2020
        Raid Level : raid1
        Array Size : 1047552 (1023.00 MiB 1072.69 MB)
     Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB)
      Raid Devices : 2
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Dec 29 19:30:42 2020
             State : clean 
    Active Devices : 2
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : bitmap

[root@compute ~]# mdadm --detail /dev/md126 | head -n 20
/dev/md126:
           Version : 1.2
     Creation Time : Mon Dec 28 23:04:23 2020
        Raid Level : raid1
        Array Size : 142191616 (135.60 GiB 145.60 GB)
     Used Dev Size : 142191616 (135.60 GiB 145.60 GB)
      Raid Devices : 2
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Dec 29 23:10:20 2020
             State : clean 
    Active Devices : 2
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : bitmap