Product variations are causing duplicates to show up in filter results
No, that's not possible, the delta=0 will remove duplicates by restricting to the first product in the product reference field. It won't apply selectively to different attributes on that referenced product. 

A better option may be to use the colour filter only, then on the product display node itself, add the value to select the size in the add to cart form.

That way, customers can filter by colour, then select the size they want when viewing the product detail.