I have limited samples for one class, unlimited samples for the other class. Need to balance? I want my machine learning algorithm to learn the difference between two classes, actually picture of X or picture of something else.

My sample data is:


500 pictures of X (I know it is low, unfortunately I can't do much about it)
1,000,000 pictures of something else


Question: Should I do the training with all of the 1,000,000 pictures of something else?
Or would such an imbalance have negative effects? For instance would it kind of "drown" the other data?

Notes:


Computing power and time is not an issue.
In the real world, pictures of X make up 5% to 10% of the data, so I don't think I have a class imbalance problem.
I think the classification is an easy one, machine learning will probably be able to understand quickly.
I am OK with a reasonable amount of misclassifications.
In case that matters, I am planning to use Keras and Tensorflow with Flatten/Dense relu/Dense softmax/AdamOptimizer/sparse_categorical_crossentropy.
Yes, train with all the data. 

To adjust for the imbalanced data size, you want your minority class X to be more important than the majority class. You can either (1) use weighted cross-entropy loss, e.g. tf.nn.weighted_cross_entropy_with_logits, or (2) when adding up the loss from different traning examples, weight them inversely proportional to the size of that label, or use both.

This is a good explanation here.