Magento 2: Override theme conditionally fails

I have a certain route in my store for which I want to switch to magento’s luma theme.
I used the layout_load_before event and created an oberserver to do that:

namespace MyModuleObserver;

class LayoutLoadBefore implements MagentoFrameworkEventObserverInterface
{
    protected MagentoFrameworkAppRequestHttp $_request;
    protected MagentoFrameworkViewDesignInterface $_design;

    public function __construct(
        MagentoFrameworkAppRequestHttp $request,
        MagentoFrameworkViewDesignInterface $design
    ) {
        $this->_request = $request;
        $this->_design = $design;
    }

    public function execute($observer): void
    {
        if (substr($this->_request->getPathInfo(), 0, 8) === '/myroute') {
            $this->_design->setDesignTheme('Magento/luma');
        }
    }
}

This works fine, meaning the observer is triggered only when the current route matches that condition.

But unfortunately the design/template switch does not work 100% as the page load fails with an error:

Magento Framework Exception ValidatorException
Invalid template file: ‘Magento_Theme::html/header/header-bar.phtml’ in module: ”
block’s name: ‘header-bar’

The header-bar is from my stores actual template/design where I defined that block inside the theme’s layout file: Magento_Theme/layout/header.xml

That block’s definition looks like this:

<block class="MagentoFrameworkViewElementTemplate" name="header-bar" as="header_bar" template="Magento_Theme::html/header/header-bar.phtml" before="-" />

The header.xml, not being a default xml layout file is being added via layout update in the Magento_Theme/layout/default.xml: <update handle="header"/>

I am confused as I have switched to the luma theme. Why is magento complaining about a block that was defined in my custom theme which should be irrelevant at this point?

When I switch the entire store to the luma theme via admin everything works fine, but of course, that is not the goal here.

EDIT: When I use the event controller_action_predispatch it seems to work fine (so far).