Genetic Algorithm Minimum Population Size
There is no minimum to population size but it has a few drawbacks when it is too low.

when it is too low your genetic algorithm is almost going to be a deterministic or greedy algorithm and besides that you are going to lose the effect of weak answers. it has been proved that even the weakest answer can move the algorithm to a good answer. The nature of genetic algorithm is randomization and bias to better answers, when the population size is too low non of these are regarded. 

When the population size is too low the population is going to lose the diversity so most likely your algorithm will fall in local optimums. 

When the population size is too low you will face a few problems in Replacement step. because the population size is too low you will have to reject the unfeasible answers and it is a very bad idea for more illustration, look at this picture:



Let the red color be all of the answers (feasible and unfeasible), the yellow be the space of feasible answers, the green be the space of good answers that are around the optimums and the little blue spot be an unfeasible answer. As you can see an unfeasible answer is very close to the space of good answers so may be a little variation to this unfeasible answer lead to a very good answer, yes? that is why I say when the population size is too low you are going to limit yourself in Replacement step and you will have to remove the unfeasible answers and removing such answers has the bad effect that I just explained.

If the population size is too big your algorithm will be brute force! I recommended you to read the Schema Theory to know how to provide an appropriate population size.