Single download effectively turns off internet access for everything else

I have dual link DSL internet, something around 24mb/1mb. The problem I have is that no matter what I seem to do, any time a device on my network is downloading something at near peak bandwith, it effectively DoS’s the rest of the network until the download is finished, rather that what I would expect to happen is that my DSL router engages in some kind of load balancing. I have a nighthawk RAX120 wifi router in from of the DSL modem, and even using the QoS feature on it (and my previous TP-LINK AC7200 QoS did not rectify this either, hence the nighthawk) does not seem to solve the problem. Any suggestions?