Filters don’t hide out-of-stock products

I have a couple of custom filters, size and warehouse availability. When I select for example, size “XS” and “Warehouse1” I still see the product in the catalog that has size XS in other warehouses, but specifically not in Warehouse1. Why is that so and what can be done to hide such products? The custom attribute for size is dropdown and for warehouses it’s multiple-select, all containing the correct data.