Zend certified PHP/Magento developer

Killed fstrim my bcache btrfs raid?

Imagine the following setup: 6x 2TB SATA drives, one 1TB NVME SSD, Ubuntu 18.04, Kernel 5.3.0-40

I configured the SSD as a bcache caching device and the HDDs as bcache backing device, formatted them as BTRFS in Raid 1. The SSD was in writeback mode, which means writes go to the SSD first and will be written down to the disks later. I ran a couple of VMs from this filesystem, everything functioning normal. It is a hard guess, but I think the SSD buffered around 10-20 GB of data for each backing device, but I did not monitor this metric, but when I looked sometimes, there was always that much unsynced data. In the night from Sunday to Monday had only one VM running from this FS an when I unlocked the PC Monday morning, this VM was frozen. The hosts logs tell the following:

Mar 16 00:00:00 hostname systemd[1]: Starting Discard unused blocks...
Mar 16 00:00:00 hostname systemd[1]: Started Timeline of Snapper Snapshots.
Mar 16 00:00:00 hostname dbus-daemon[1782]: [system] Activating service name='org.opensuse.Snapper' requested by ':1.383' (uid=0 pid=75263 comm="/usr/lib/snapper/systemd-helper --timeline " label="unconfined") (using servicehelper)
Mar 16 00:00:00 hostname dbus-daemon[1782]: [system] Successfully activated service 'org.opensuse.Snapper'
Mar 16 00:00:00 hostname kernel: [222595.172893] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 1, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172906] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 2, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172916] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 3, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172925] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 4, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172935] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 5, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172940] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 6, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172940] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 7, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172949] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 8, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172955] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 9, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.172956] BTRFS error (device bcache4): bdev /dev/bcache0 errs: wr 347, rd 11, flush 0, corrupt 0, gen 0
Mar 16 00:00:00 hostname kernel: [222595.173329] BTRFS error (device bcache4): error reading free space cache
Mar 16 00:00:00 hostname kernel: [222595.173337] BTRFS warning (device bcache4): failed to load free space cache for block group 11557819908096, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.173908] BTRFS error (device bcache4): error reading free space cache
Mar 16 00:00:00 hostname kernel: [222595.173911] BTRFS warning (device bcache4): failed to load free space cache for block group 11558893649920, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.174453] BTRFS error (device bcache4): error reading free space cache
Mar 16 00:00:00 hostname kernel: [222595.174456] BTRFS warning (device bcache4): failed to load free space cache for block group 11559967391744, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.174960] BTRFS error (device bcache4): error reading free space cache
# ... about 180 of these lines
Mar 16 00:00:00 hostname kernel: [222595.229687] BTRFS warning (device bcache4): failed to load free space cache for block group 11762904596480, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.230393] BTRFS error (device bcache4): error reading free space cache
Mar 16 00:00:00 hostname kernel: [222595.230395] BTRFS warning (device bcache4): failed to load free space cache for block group 11765052080128, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.231171] BTRFS error (device bcache4): error reading free space cache
Mar 16 00:00:00 hostname kernel: [222595.231173] BTRFS warning (device bcache4): failed to load free space cache for block group 11774715756544, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.231749] BTRFS warning (device bcache4): failed to load free space cache for block group 11785453174784, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.232340] BTRFS warning (device bcache4): failed to load free space cache for block group 11786526916608, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.233284] BTRFS warning (device bcache4): failed to load free space cache for block group 11791895625728, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.234165] BTRFS warning (device bcache4): failed to load free space cache for block group 11802633043968, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.236123] BTRFS warning (device bcache4): failed to load free space cache for block group 11803706785792, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.236723] BTRFS warning (device bcache4): failed to load free space cache for block group 11804780527616, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.237310] BTRFS warning (device bcache4): failed to load free space cache for block group 11809075494912, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.237984] BTRFS warning (device bcache4): failed to load free space cache for block group 11813370462208, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.238496] BTRFS warning (device bcache4): failed to load free space cache for block group 11823034138624, rebuilding it now
Mar 16 00:00:00 hostname kernel: [222595.239040] BTRFS warning (device bcache4): failed to load free space cache for block group 11824107880448, rebuilding it now
Mar 16 00:00:01 hostname fstrim[75262]: fstrim: /media/raid: FITRIM ioctl fehlgeschlagen: Eingabe-/Ausgabefehler # meaning I/O-Error

I restarted the PC (no hard reset, OS is running on another drive) and now trying to mount the fs gives the following errors:

sudo mount -t btrfs -o ro,recovery,nospace_cache /dev/bcache5 /mnt
# stderr:
mount: wrong fs type, bad option, bad superblock on /dev/, missing codepage or helper program, or other error.
# Syslog:
BTRFS info (device bcache5): disk space caching is enabled
BTRFS info (device bcache5): has skinny extents
BTRFS error (device bcache5): parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
BTRFS error (device bcache5): parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
BTRFS warning (device bcache5): failed to read tree root
BTRFS error (device bcache5): open_ctree failed

The transid is off by 17442, I consider this as a very bad sign.

I have tried:

sudo btrfs rescue super-recover -v /dev/bcache5
All Devices:
        Device: id = 12, name = /dev/bcache4
        Device: id = 11, name = /dev/bcache3
        Device: id = 10, name = /dev/bcache2
        Device: id = 8, name = /dev/bcache1
        Device: id = 9, name = /dev/bcache0
        Device: id = 7, name = /dev/bcache5

Before Recovering:
        [All good supers]:
                device name = /dev/bcache5
                superblock bytenr = 65536

                device name = /dev/bcache5
                superblock bytenr = 67108864

        [All bad supers]:
                device name = /dev/bcache4
                superblock bytenr = 65536

                device name = /dev/bcache4
                superblock bytenr = 67108864

                device name = /dev/bcache4
                superblock bytenr = 274877906944

                device name = /dev/bcache3
                superblock bytenr = 65536

                device name = /dev/bcache3
                superblock bytenr = 67108864

                device name = /dev/bcache3
                superblock bytenr = 274877906944

                device name = /dev/bcache2
                superblock bytenr = 65536

                device name = /dev/bcache2
                superblock bytenr = 67108864

                device name = /dev/bcache2
                superblock bytenr = 274877906944

                device name = /dev/bcache1
                superblock bytenr = 65536

                device name = /dev/bcache1
                superblock bytenr = 67108864

                device name = /dev/bcache1
                superblock bytenr = 274877906944

                device name = /dev/bcache0
                superblock bytenr = 65536

                device name = /dev/bcache0
                superblock bytenr = 67108864

                device name = /dev/bcache0
                superblock bytenr = 274877906944

                device name = /dev/bcache5
                superblock bytenr = 274877906944


Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: y
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
Ignoring transid failure
Couldn't setup extent tree
Failed to recover bad superblocks

sudo btrfs rescue zero-log /dev/bcache5
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
parent transid verify failed on 15188638056448 wanted 1734776 found 1717334
Ignoring transid failure
Couldn't setup extent tree
Couldn't setup device tree
ERROR: could not open ctree

The following process is still running:

sudo btrfs rescue chunk-recover -v /dev/bcache5

Unfortunately the dirty data on the SSD seems to be gone, but obviously not to the disks:

for I in {0..5}; do sudo sh -c "cat /sys/block/bcache$I/bcache/dirty_data"; done
0.0k
0.0k
0.0k
0.0k
0.0k
0.0k

Any ideas what might have happened? Is fstrim on a bached volume generally a bad idea? Any suggestions what I can do to mount and recover some files? (I have a backup from the important things, but some minor important thing are only stored on this array…)

Thank you for your time and help!