ffmpeg x264 encoder crashes an otherwise stable machine

I have a newly built machine based on Threadripper, running Windows 11 LTSC IoT, which seems to be stable and has worked well through initial stress tests (Prime95, Memtest, Passmark) and most of the real-world usage it has seen so far, but there’s one problem: it instantly crashes when I try to encode a video file using ffmpeg and the x264 encoder.

The crash is immediate – it creates a blank output file, then the machine restarts with no warning. There’s nothing in the Windows event logs, it happens too fast. This happens both with the regular command-line ffmpeg and Shutter Encoder.

Using the h264_amf encoder to encode h264 with ffmpeg works fine. I’ve tested transcoding to a couple of other formats too (Prores, HAP) with no issue.

I’m fairly sure it’s not a thermal or power issue, as 100% load testing works fine with stable temperatures and voltages, and the crash happens before the temperature has a chance to increase anyway. I’d suspect a bad CPU or RAM, but it doesn’t make sense to me that only this one particular operation would cause a crash every single time, with no other signs of stability issues.

The hardware spec is

    • AMD Threadripper 9960X
    • Gigabyte Radeon AI PRO R9700
    • Supermicro H13SRA-TF
    • 2x Hynix 32 GB DDR5-5600MHz ECC RDIM
    • 2x Samsung 990 Pro 4TB SSD (RAID1)
    • 3x Blackmagic DeckLink 8K Pro G2
    • Corsair HX1200i PSU
    • Silverstone XED 120 WS CPU cooler
    • Silversone RM45-360 case

The chipset drivers are downloaded from Supermicro. The GPU driver is the “Pro” version. OS, drivers and ffmpeg are the latest versions (as of yesterday). The BIOS version is recent, but not the very newest. I could upgrade, but I want to wait until after a project this machine will be use for in case of any trouble.

Apart from that I’m completely stumped on how to approach this. I guess I could try using the same encoder on Linux instead and see how that goes, but I’m not convinced that’s really useful either.

Any ideas would be much appreciated!