Why is there a difference in performance across the feature descriptors for the same imaging modality?
Feature-extraction mechanisms like GIST, HOG, etc are built and optimized to improve performance on given datasets.  Because of this, they don't perform as well across datasets.  It's kind of like putting specialized fuel in a vehicle that isn't built to utilize it - it might even do harm.

Hand-engineered features are, as a rule, brittle.  I once heard it said that the dirty secret of machine learning is just knowing how to transform your domain-specific information into meaningful features - after that, you can use an extremely simple classifier and it may do surprisingly well.  The drawback is that the rules you built are very specific to your domain.

Deep neural networks, and convolutional neural networks in particular, are an advancement in that they learn what features are useful about raw data - for CNNs, these are the raw pixel or time-series values.  Instead of hand-building feature extraction mechanisms, these architectures automatically build them.  

One benefit of this is that if you use a CNN to identify images in general, you can re-use the top few feature extraction layers of the CNN on a different image recognition dataset, and re-train the bottom few layers to make the network specific to recognizing e.g. dog breeds.  You can transfer what you learned about the statistical structure of natural images in general to other, more specific questions (general -> specific).  In your case, the 'top few layers' are analogous to your GIST/HOG methods - and they wouldn't be expected to perform well when the task changes, because they were constructed for a specific task (specific -> other specific).