Investment Mechanics
I think the mechanic you came up with doesn't work well for 1 dice per player. Basically you're taking the luck of the dice roll, and adding another luck factor. You can cancel this out by allowing the use of more dice. What if the gems you collect can be used to purchase more dice?

You should take a look at the way resource collection is managed in Stone Age, as an inspiration.