Why does Display Port audio freezes from time to time on Ubuntu 24.04?

I’m having trouble with audio playback; it freezes from time to time (for half a second), no matter what app I’m using. I’ve found out that this problem is only occur when audio is working via Display Port.

I tried changing the audio settings, switch video ports, reinstalling the audio and video drivers, but nothing worked.

The video card is GTX 1660 and Nvidia driver is 590.