I have limited samples for one class, unlimited samples for the other class. Need to balance?
Yes, train with all the data. 

To adjust for the imbalanced data size, you want your minority class X to be more important than the majority class. You can either (1) use weighted cross-entropy loss, e.g. tf.nn.weighted_cross_entropy_with_logits, or (2) when adding up the loss from different traning examples, weight them inversely proportional to the size of that label, or use both.

This is a good explanation here.