Wrong pagination for a filtered product collection

I used this event to alter my collection to remove duplicates.
it is showing the correct items, but the wrong way.
Any idea how to correct the following:

• show correct items count (256 is the wrong items count for this filtered collection)
• show correct pagination

public function execute(
MagentoFrameworkEventObserver $observer
) {
$collection = $observer->getEvent()
->getCollection();
$variations = [];
foreach ($collection as $_product) {
$variationGroupId = $_product->getVariationGroupId();
if (!empty($variationGroupId)) {
if (in_array($variationGroupId, $variations)) {
$collection->removeItemByKey($_product->getId());
}
$variations[] = $variationGroupId;
}
}
return $collection;
}