→ Applies to: Hyperion 9.x and above

IMPORTANT
If this procedure fails or SMART indicates physical disk failure, proceed with: (“How to Replace a Faulted OS Drive on Systems with MD RAID in Hyperion”)
Sometimes, mdadm marks a disk as faulty not because of physical failure, but due to a transient I/O error. In this case, the array becomes degraded even though the disk itself remains healthy. This procedure covers how to diagnose the root cause and recover the array without replacing the disk, using md126 as the example degraded MD RAID device.
 

Step 1. Connect to Hyperion appliance via SSH as admin

ssh admin@<your_ip_address_or_hostname>

Step 2. Become Super User

sudo su

Step 3. Display a quick health dashboard for all RAID arrays

cat /proc/mdstat

Step 4. Check md126 RAID array

mdadm --detail /dev/md126

EXAMPLE OUTPUT

/dev/md126:
           Version : 1.2
     Creation Time : Fri Jun 12 18:19:47 2026
        Raid Level : raid1
        Array Size : 204667904 (195.19 GiB 209.58 GB)
     Used Dev Size : 204667904 (195.19 GiB 209.58 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Jun 17 12:43:17 2026
             State : active, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 1
     Spare Devices : 0

Consistency Policy : bitmap

              Name : localhost.localdomain:root
              UUID : e59181d6:7bd23ef6:c1ad5cf9:b4c818ab
            Events : 54568

    Number   Major   Minor   RaidDevice State
         -       0       0        0     removed
         1       8      35        1     active sync   /dev/sdc3
         0       8       3        -     faulty        /dev/sda3

Step 5. Check md127 RAID array

mdadm --detail /dev/md127

Example OUTPUT

/dev/md127:
           Version : 1.2
     Creation Time : Fri Jun 12 18:19:41 2026
        Raid Level : raid1
        Array Size : 2094080 (2045.00 MiB 2144.34 MB)
     Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Jun 16 21:54:04 2026
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : localhost.localdomain:boot
              UUID : f7addf86:6d5c6124:e63d9781:4aec22ec
            Events : 20

    Number   Major   Minor   RaidDevice State
         0       8       2        0     active sync   /dev/sda2
         1       8      33        1     active sync   /dev/sdc1

Step 6. Check Physical Disk Health (replace <faulty_disk> with the correct information)

smartctl -a /dev/<faulty_disk>

Example

smartctl -a /dev/sda

Step 7 (Optional). List all block devices

lsblk -o name,size,mountpoint,vendor,model,serial

Step 8. Remove faulty partition (replace <faulty_partition> with the correct information)

mdadm /dev/md126 --remove /dev/<faulty_partition>

EXAMPLE

mdadm /dev/md126 --remove /dev/sda3

IMPORTANT
Run this command only if the faulty partition belongs to the md126 array.

Step 9. Remove faulty partition (replace <faulty_partition> with the correct information)

mdadm /dev/md127 --remove /dev/<faulty_partition>

EXAMPLE

mdadm /dev/md127 --remove /dev/sda2

 

IMPORTANT
Run this command only if the faulty partition belongs to the md127 array.

Step 10. Re-add the removed partition (replace <faulty_partition> with the correct information)

mdadm /dev/md126 --add /dev/<faulty_partition>

EXAMPLE

mdadm /dev/md126 --add /dev/sda3

IMPORTANT
Run this command only if the faulty partition belongs to the md126 array.

Step 11. Re-add the removed partition (replace <faulty_partition> with the correct information)

mdadm /dev/md127 --add /dev/<faulty_partition>

EXAMPLE

mdadm /dev/md127 --add /dev/sda2

IMPORTANT
Run this command only if the faulty partition belongs to the md127 array.

Step 12. Monitor the rebuild process

watch -n 2 cat /proc/mdstat

Step 13. Check md126 RAID array again after rebuild completed

mdadm --detail /dev/md126

EXAMPLE OUTPUT

/dev/md126:
           Version : 1.2
     Creation Time : Fri Jun 12 18:19:47 2026
        Raid Level : raid1
        Array Size : 204667904 (195.19 GiB 209.58 GB)
     Used Dev Size : 204667904 (195.19 GiB 209.58 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Jun 17 14:53:40 2026
             State : active
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : localhost.localdomain:root
              UUID : e59181d6:7bd23ef6:c1ad5cf9:b4c818ab
            Events : 72332

    Number   Major   Minor   RaidDevice State
         0       8       3        0     active sync   /dev/sda3
         1       8      35        1     active sync   /dev/sdc3

Step 14. Check md127 RAID array again after rebuild completed

mdadm --detail /dev/md127

EXAMPLE OUTPUT

/dev/md127:
           Version : 1.2
     Creation Time : Fri Jun 12 18:19:41 2026
        Raid Level : raid1
        Array Size : 2094080 (2045.00 MiB 2144.34 MB)
     Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Tue Jun 16 21:54:04 2026
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : localhost.localdomain:boot
              UUID : f7addf86:6d5c6124:e63d9781:4aec22ec
            Events : 20

    Number   Major   Minor   RaidDevice State
         0       8       2        0     active sync   /dev/sda2
         1       8      33        1     active sync   /dev/sdc1