UCX devices visible to root not to user

I’m currently using DOCA-OFED as the UCX driver, and IPoIB and UCX on the same device. The system can see the device from ibstat

CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.36.1010
        Hardware version: 0
        Node GUID: 0x946dae03006f4936
        System image GUID: 0x946dae03006f4936
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 8
                LMC: 0
                SM lid: 8
                Capability mask: 0xa651e84a
                Port GUID: 0x946dae03006f4936
                Link layer: InfiniBand

This device is ibs1 on the node, and works perfectly on IPoIB

However I can’t get UCX to work, as a user, notably it does work as root, confirmed by ibping.

I’ve worked through the udev rules

KERNEL=="issm*", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="ucm*", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="umad*", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="uverbs*", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="uat", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="ucma", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"
KERNEL=="rdma_cm", SYMLINK+="infiniband/%k", MODE="0776", GROUP="rdma"

In /etc/udev/rules.d/90-rdma.rules and confirmed that sets the /dev/infiniband folders sensibly (note-this server is airgapped so we can run a little bit lighter on the security. I also know I think thats not quite correct, but the documentation for this issue spans a decade and a half.

Running ucx_info -d as root produces sensible results with rdma memory domains for mlx5_0 but running it as a user who is a member of the group rdma does not produce the memory domains expected.

I don’t even know how to diagnose this at this point (rocky 9.7 for reference, connectx-7 compatible device).