VSCode Remote Target connections hang my ssh Server cannot connect to box

Whenever I use VS Code Remote Connect features to setup ssh and connect to boxes, it doesn’t establish a connection, even worse, it hangs when I try other methods to connect via ssh which would have worked if I haven’t attempted with VS Code Remote Connect first. Only I have to restart the box to connect via SSH again. VS Code Remote never works but causes to hang the SSH deamons.