How do I check the entered quantity before the checkout phase in Drupal Commerce?
You can use rules to change the quantity in a line item... so you could force the quantity up and output a system message. I don't know of a way to prevent them from checking out using rules. 

So: Condition: Data Comparison: Line item: Product: SKU  equals XX
Condition: Data Comparison: Line Item: Quantity less than X
Action: System message explaining why we're increasing quantity
Action: Data Set Value: Set quantity to X+3

Update: this strategy might work for you.