How to ensure Chrome extensions only haave access to specific sites

I wish to restrict a specific chrome extension to have access to only Youtube. I have inputted the below in that extension’s settings.

However, I still seem to be able to manually enable the extension for other sites by clicking on the extension and then this Reload button?

Furthermore, I’ve noticed the extension is enabled anyway on some sites that aren’t youtube (including, say superuser.com).

Is the whitelist feature of Chrome faulty? Any way I can resolve this?