Why does nobody use decision trees for visual question answering?
For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better to inputs and maintain performance when inputs have undergone certain linear transformations (e.g. some scaling or a translation along the x-axis). These properties, along with the robust frameworks that exist for developing and deploying neural nets, and the fact that they have been shown to widely produce very good results, are some of the reasons they’re used. 

In terms of being a black box, while this is true for a lot of applications it’s actually less true for image-based tasks. The layers of a well designed and trained convolutional neural network model can actually be visualized and made quite interpretable; from these visualizations, it’s often clear how the representation roughly works. In contrast, I’d argue that while a decision tree is theoretically easier to interpret for some tasks (say medical decision making), this is less the case for vision tasks because we don’t interpret images one pixel at a time. People are almost always interested in the higher level representation within an image (say, a cat, a leaf, or a face). Decision trees tend to have trouble capturing these higher level representations. 

Distill.pub has a nice explanation of feature visualization that may be of interest.