Can you take 2 gems of the same kind for 2-3 players?
You are reading the rules incorrectly, they says when there are 4 chips at the start of your turn not after you take them. The rules don't care how many gems are in a stack after you have taken them just beforehand.

Rules


  Collect Gem Chips – Either:Take 1 Gem from each of 3 different stacks (not gold)
  1: Take 2 Gems from one stack, as long as there are 4 or more chips in that stack


In a 4 player game there can be 3 chips missing at the start of your turn, 3 player 1 chip missing and 2 player no chips missing.

Testing on the iOs version of the game lets me draw two chips of the same color from a stack when there is exactly 4 chips on it in a two player game.