I accidentally bumped the power button on my USB hub for my (Raspberry Pi) Jellyfin server. I quickly reattached the one drive that got disconnected and did a full power cycle of the Pi, and now it says the RAID is incomplete in journalctl. It is stating that 5 of the 8 attached devices have failed.
Oct 30 23:59:13 JellyPi kernel: md/raid:mdX: device dm-11 operational as raid disk 5
Oct 30 23:59:13 JellyPi kernel: md/raid:mdX: device dm-13 operational as raid disk 6
Oct 30 23:59:13 JellyPi kernel: md/raid:mdX: device dm-15 operational as raid disk 7
Oct 30 23:59:13 JellyPi kernel: md/raid:mdX: not enough operational devices (5/8 failed)
Oct 30 23:59:13 JellyPi kernel: md/raid:mdX: failed to run raid set.
Oct 30 23:59:13 JellyPi kernel: md: pers->run() failed ...
Oct 30 23:59:13 JellyPi kernel: device-mapper: table: 253:16: raid: Failed to run raid array (-EIO)
Oct 30 23:59:13 JellyPi kernel: device-mapper: ioctl: error adding target to table
Oct 30 23:59:13 JellyPi lvm[584]: device-mapper: reload ioctl on (253:16) failed: Input/output error
However, all 8 SSDs are nearly brand new, purchased at different times to spread out the batches. I don’t believe there could be 5 devices failing simultaneously like this, and LVM seems to agree. All PVs are visible, with no errors:
PV VG Fmt Attr PSize PFree
/dev/sda1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdb1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdc1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdd1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sde1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdf1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdg1 jellyfinvg lvm2 a-- <953.87g 253.13g
/dev/sdh1 jellyfinvg lvm2 a-- <953.87g 253.13g
As far as I can tell, vgck produces no errors. pvck produces no errors. I’ve swapped out the SSD enclosure to ensure that wasn’t the problem, as well as migrated this to other LVM2 capable devices to test if it was bad USB ports. Everything seems to be completely visible and working… just not available to mount.
Running vgchange with verbose, I can see the first 15 objects being created and the 16th failing, but I don’t really understand what that means. I did another test, and LVM is capable of creating new LVs on all 8 PVs, so I don’t think it’s an issue with LVM specifically, but perhaps the RAID metadata. Is there a way to flush that or restore from a backup?
Activating logical volume jellyfinvg/media-youtube.
activation/volume_list configuration setting not defined: Checking only host tags for jellyfinvg/media-youtube.
Creating jellyfinvg-media--youtube_rmeta_0
Loading table for jellyfinvg-media--youtube_rmeta_0 (253:0).
Resuming jellyfinvg-media--youtube_rmeta_0 (253:0).
Creating jellyfinvg-media--youtube_rimage_0
Loading table for jellyfinvg-media--youtube_rimage_0 (253:1).
Resuming jellyfinvg-media--youtube_rimage_0 (253:1).
Creating jellyfinvg-media--youtube_rmeta_1
Loading table for jellyfinvg-media--youtube_rmeta_1 (253:2).
Resuming jellyfinvg-media--youtube_rmeta_1 (253:2).
Creating jellyfinvg-media--youtube_rimage_1
Loading table for jellyfinvg-media--youtube_rimage_1 (253:3).
Resuming jellyfinvg-media--youtube_rimage_1 (253:3).
Creating jellyfinvg-media--youtube_rmeta_2
Loading table for jellyfinvg-media--youtube_rmeta_2 (253:4).
Resuming jellyfinvg-media--youtube_rmeta_2 (253:4).
Creating jellyfinvg-media--youtube_rimage_2
Loading table for jellyfinvg-media--youtube_rimage_2 (253:5).
Resuming jellyfinvg-media--youtube_rimage_2 (253:5).
Creating jellyfinvg-media--youtube_rmeta_3
Loading table for jellyfinvg-media--youtube_rmeta_3 (253:6).
Resuming jellyfinvg-media--youtube_rmeta_3 (253:6).
Creating jellyfinvg-media--youtube_rimage_3
Loading table for jellyfinvg-media--youtube_rimage_3 (253:7).
Resuming jellyfinvg-media--youtube_rimage_3 (253:7).
Creating jellyfinvg-media--youtube_rmeta_4
Loading table for jellyfinvg-media--youtube_rmeta_4 (253:8).
Resuming jellyfinvg-media--youtube_rmeta_4 (253:8).
Creating jellyfinvg-media--youtube_rimage_4
Loading table for jellyfinvg-media--youtube_rimage_4 (253:9).
Resuming jellyfinvg-media--youtube_rimage_4 (253:9).
Creating jellyfinvg-media--youtube_rmeta_5
Loading table for jellyfinvg-media--youtube_rmeta_5 (253:10).
Resuming jellyfinvg-media--youtube_rmeta_5 (253:10).
Creating jellyfinvg-media--youtube_rimage_5
Loading table for jellyfinvg-media--youtube_rimage_5 (253:11).
Resuming jellyfinvg-media--youtube_rimage_5 (253:11).
Creating jellyfinvg-media--youtube_rmeta_6
Loading table for jellyfinvg-media--youtube_rmeta_6 (253:12).
Resuming jellyfinvg-media--youtube_rmeta_6 (253:12).
Creating jellyfinvg-media--youtube_rimage_6
Loading table for jellyfinvg-media--youtube_rimage_6 (253:13).
Resuming jellyfinvg-media--youtube_rimage_6 (253:13).
Creating jellyfinvg-media--youtube_rmeta_7
Loading table for jellyfinvg-media--youtube_rmeta_7 (253:14).
Resuming jellyfinvg-media--youtube_rmeta_7 (253:14).
Creating jellyfinvg-media--youtube_rimage_7
Loading table for jellyfinvg-media--youtube_rimage_7 (253:15).
Resuming jellyfinvg-media--youtube_rimage_7 (253:15).
Creating jellyfinvg-media--youtube
Loading table for jellyfinvg-media--youtube (253:16).
device-mapper: reload ioctl on (253:16) failed: Input/output error
Removing jellyfinvg-media--youtube (253:16)
Removing jellyfinvg-media--youtube_rmeta_0 (253:0)
Removing jellyfinvg-media--youtube_rimage_0 (253:1)
Removing jellyfinvg-media--youtube_rmeta_1 (253:2)
Removing jellyfinvg-media--youtube_rimage_1 (253:3)
Removing jellyfinvg-media--youtube_rmeta_2 (253:4)
Removing jellyfinvg-media--youtube_rimage_2 (253:5)
Removing jellyfinvg-media--youtube_rmeta_3 (253:6)
Removing jellyfinvg-media--youtube_rimage_3 (253:7)
Removing jellyfinvg-media--youtube_rmeta_4 (253:8)
Removing jellyfinvg-media--youtube_rimage_4 (253:9)
Removing jellyfinvg-media--youtube_rmeta_5 (253:10)
Removing jellyfinvg-media--youtube_rimage_5 (253:11)
Removing jellyfinvg-media--youtube_rmeta_6 (253:12)
Removing jellyfinvg-media--youtube_rimage_6 (253:13)
Removing jellyfinvg-media--youtube_rmeta_7 (253:14)
Removing jellyfinvg-media--youtube_rimage_7 (253:15)
Activated 0 logical volumes in volume group jellyfinvg.
