Index on integer column not helping. Help? (EXPLAIN ANALYZE output inside)
With only 5 values the index isn't honing to be very selective for joining so it'll pretty much always index scan unless looking for just one value.

That query plan/anal output suggests that it thinks half the work is in matching the rows there with the CTE, could you try run the group+aggregate over the whole lot. If it is index scanning anyway you might find it faster to do that and then filter out the two you are not looking for results for.