Im trying to install two K80 GPU’s into each available slot of my motherboard. And would like some help and advice on how to potentially solve this. Im guessing it might include a BIOS patch or linux boot args.
The card installed into the
00:1c.0 Intel Corporation 100/C230 Series Chipset
fully allocates its nested BARs. But the one using which I guess is provided by the cpu
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller
fails to allocate all nested BARs.
My BIOS has above 4G decoding enabled, and I tried to patch with ReBarUEFI (set to 512GB) without any success. Ive played a lot with my boot parameters as provided by docs, searches and GPT’s(I know), but currently its set to pci=nocrs,realloc=on,assign-busses
Ive also tried a mixed bag of and eventually just trying random combinations: pci=nocrs,realloc=on,assign-busses,big_root_window,pcie_bus_perf resource_alignment=64G iomem=relaxed pcie_ports=compat
I feel like one thing that would help is an “unlocked” bios, where all the menus are visible and accessible. When I was playing around with the bios there seemed to be many hidden options and values that I had no idea how to expose.
Hopefully someone out there knows what they are doing more than me! Any advice or questions is greatly appreciated.
System specs:
Motherboard GA-B150M-D3H (1.0)
BIOS B150MD3H.F25d
GPU Tesla K80
CPU i7-6700K
OS Arch Linux (7.0.5)
RAM 32G
In dmesg it fails with(reduced):
[ 0.272573] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.272574] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]
[ 0.272576] pci_bus 0000:00: root bus resource [bus 00-7e]
[ 0.329966] pci 0000:00:01.0: bridge window [mem 0xc00000000-0x1403ffffff 64bit pref]: assigned
[ 0.329969] pci 0000:00:01.0: bridge window [mem 0x60000000-0x61ffffff]: assigned
[ 0.329971] pci 0000:01:00.0: bridge window [mem 0xc00000000-0x1403ffffff 64bit pref]: assigned
[ 0.329973] pci 0000:01:00.0: bridge window [mem 0x60000000-0x61ffffff]: assigned
[ 0.329975] pci 0000:02:08.0: bridge window [mem 0xc00000000-0x1001ffffff 64bit pref]: assigned
[ 0.329977] pci 0000:02:10.0: bridge window [mem size 0x402000000 64bit pref]: can't assign; no space
[ 0.329978] pci 0000:02:10.0: bridge window [mem size 0x402000000 64bit pref]: failed to assign
[ 0.329980] pci 0000:02:08.0: bridge window [mem 0x60000000-0x60ffffff]: assigned
[ 0.329981] pci 0000:02:10.0: bridge window [mem 0x61000000-0x61ffffff]: assigned
[ 0.329982] pci 0000:02:08.0: bridge window [mem 0xc00000000-0x1001ffffff 64bit pref]: releasing
[ 0.329993] pci 0000:03:00.0: BAR 1 [mem 0xc00000000-0xfffffffff 64bit pref]: assigned
[ 0.330000] pci 0000:03:00.0: BAR 3 [mem 0x1000000000-0x1001ffffff 64bit pref]: assigned
[ 0.330006] pci 0000:03:00.0: BAR 0 [mem 0x60000000-0x60ffffff]: assigned
[ 0.330008] pci 0000:02:08.0: PCI bridge to [bus 03]
[ 0.330011] pci 0000:02:08.0: bridge window [mem 0x60000000-0x60ffffff]
[ 0.330014] pci 0000:02:08.0: bridge window [mem 0xc00000000-0x1001ffffff 64bit pref]
[ 0.330018] pci 0000:04:00.0: BAR 1 [mem size 0x400000000 64bit pref]: can't assign; no space
[ 0.330019] pci 0000:04:00.0: BAR 1 [mem size 0x400000000 64bit pref]: failed to assign
[ 0.330020] pci 0000:04:00.0: BAR 3 [mem size 0x02000000 64bit pref]: can't assign; no space
[ 0.330021] pci 0000:04:00.0: BAR 3 [mem size 0x02000000 64bit pref]: failed to assign
[ 0.330022] pci 0000:04:00.0: BAR 0 [mem 0x61000000-0x61ffffff]: assigned
And the PCI devices have the layout:
00:00.0 Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
02:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
03:00.0 NVIDIA Corporation GK210GL [Tesla K80]
02:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
04:00.0 NVIDIA Corporation GK210GL [Tesla K80] (###FAILING###)
00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 (rev f1) (prog-if 00 [Normal decode])
05:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode])
06:08.0 CI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
07:00.0 NVIDIA Corporation GK210GL [Tesla K80]
06:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
08:00.0 NVIDIA Corporation GK210GL [Tesla K80]
More complete logs and some hopefully appropriate command outputs can be found at https://pastebin.com/aissh96i
