I am trying to see how these two set of commands differ if I just have one nvme device.
Block device
ls -l /sys/block/nvme0n1
lrwxrwxrwx 1 root root 0 Apr 28 11:59 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.1/0000:09:00.0/nvme/nvme0/nvme0n1
sudo sh -c "echo 1 > /sys/block/nvme0n1/device/device/remove"
# LINK DISABLE/DOWN
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x40:0x40
# LINK ENABLE
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x00:0x40
sudo sh -c "echo 1 > /sys/bus/pci/rescan"
If I run sudo nvme list after the above, the device is present.
sudo modprobe -r nvme
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x40:0x40
sudo setpci -s 0000:00:03.1 BRIDGE_CONTROL.W=0x00:0x40
sudo modprobe nvme
If I run sudo nvme list after the above, the device is missing. However, if I remove the device from the PCI bus and rescan, it is able to come back up.
There are no issues with either remove method when running without the link down commands.
Does link down and up require mmu space to be reset? According to dmesg
echo 1 > .. /remove; echo 1 > .. ./rescan commands do reset mmu and BAR
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: Removing from iommu group 23
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: [1d82:0401] type 00 class 0x010802
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x10: [mem 0xea1e0000-0xea1e3fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x18: [mem 0xea1e4000-0xea1e7fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x20: [mem 0xd0000000-0xdfffffff 64bit pref]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: supports D1
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: PME# supported from D0 D1 D3hot
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1d8: [mem 0xea1e8000-0xea1ebfff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR0 space: [mem 0xea1e8000-0xea1fffff 64bit] (contains BAR0 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1e0: [mem 0xea180000-0xea18ffff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR2 space: [mem 0xea180000-0xea1dffff 64bit] (contains BAR2 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: reg 0x1ec: [mem 0xea000000-0xea03ffff]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: VF(n) BAR5 space: [mem 0xea000000-0xea17ffff] (contains BAR5 for 6 VFs)
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: Adding to iommu group 23
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 4: assigned [mem 0xd0000000-0xdfffffff 64bit pref]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 12: assigned [mem 0xea000000-0xea17ffff]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 9: assigned [mem 0xea180000-0xea1dffff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 0: assigned [mem 0xea1e0000-0xea1e3fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 2: assigned [mem 0xea1e4000-0xea1e7fff 64bit]
[Fri Apr 28 12:11:04 2023] pci 0000:09:00.0: BAR 7: assigned [mem 0xea1e8000-0xea1fffff 64bit]
[Fri Apr 28 12:11:04 2023] nvme nvme0: pci function 0000:09:00.0
[Fri Apr 28 12:11:04 2023] nvme nvme0: 15/0/0 default/read/poll queues
but not modprobe
[Fri Apr 28 12:23:31 2023] nvme nvme0: pci function 0000:09:00.0
[Fri Apr 28 12:23:31 2023] nvme 0000:09:00.0: enabling device (0000 -> 0002)
[Fri Apr 28 12:23:31 2023] nvme nvme0: 15/0/0 default/read/poll queues
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 264 op 0x0:(READ) flags 0x80700 phys_seg 30 prio class 0
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 520 op 0x0:(READ) flags 0x80700 phys_seg 39 prio class 0
[Fri Apr 28 12:23:31 2023] blk_update_request: I/O error, dev nvme0n1, sector 16777215096 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 0
