How to use "ignore" class with tensorflow object detection API?
Yes, you need to have another Class which is the object you don't want to detect. 
If you don't have this Other Class which includes everything that is not to be detected. The model will compare it to the existing class which is almost identical to the cards of interest.
Some of the factors are:

Similarity of Shape
Similarity of Color 
Similarity of Symbols

This is why even though it is not the card of interest (Skip, Reverse, and Draw 4), it would somehow have high "belongingness" to these three classes.
Having another Class to dump all of these can significantly lessen the "belongingness" to the three classes of interest and as much as possible provide A LOT of Data during Training.
If you don't want to have another class.
You could overfit Skip, Reverse, and Draw 4 cards (close to 100%), then increase your threshold value of detection to (70-90%).
Hope this will help you.
I have trained tensorflow object detection model (for num_steps:50000) using SSD (mobilenet-v1) on custom dataset. I got mAP@.50IOU ~0.98 and loss ~1.17. The dataset consist of uno playing card images (skip, reverse, and draw four). On all these cards, model performs pretty well as I have trained model only on these 3 card (around 278 images with 829 bounding boxes (25% bounding box used for testing i.e. validation) collected using mobile phone).
However, I haven’t trained model on any other card but still it detects other cards (inference using webcam).
How can I fix this? Should I also collect other class images (anything other than skip, reverse and draw four cards) and ignore this class in operation? So that model sees this class i.e. Label: Other images during training and doesn’t put any label during inference.
I am not sure how to inform tensorflow object detection API that it should ignore images from Other class.
Can anyone please provide pointer?
Please share your views!