Magento 2.3.2 LayoutXML Update remove canonical

Under Stores > Configuration > Catalog > Catalog, We have “Use Canonical Link Meta Tag For Categories” set to “Yes”

There are a couple of category pages that we want to edit the Canonical Link for.
I am able to add a canonical via XML Update with the following code

However, This causes our category page to have 2 canonical links.
I’m looking for a way to remove the canonical link put in place by the config setting, or a way to edit the existing canonical link for a few of our categories.

Any help would be great.

Thanks,