Machine Learning using NVIDIA DIGITS - Can't Classify Left or Right Direction of a Ball Throw
a few things,
these images look huge. Are they resized or cropped before going into the net. Try resizing instead of cropping. Also, if you use flip/rotation augmentation that will mess up your network.

Second, in the training data what % are right and what % are left throws? ideally, you have about 50:50 in the training set. If images in training are mostly left, it might learn to guess left preferentially. 

BTW, there might be an easier way to solve this problem. First, extract small patches and classify each patch as: 1 (contains a ball) or 0 (doesn't contain a ball) - this is segmentation.

After segmentation, a simple algorithm that tells you if the ball is left of center or right of center will give you the correct answer.