What is the best way of combining audio and visual data to make predictions?
You can try to fuse the image and audio data. You can fuse in different ways:

1) Early fusion - fusion at feature level

2) Late fusion - fusion at output level (similar to your idea)

Here is some papers for your reference:

Look, Listen and Learn
https://arxiv.org/pdf/1705.08168.pdf

Learn to Combine Modalities in Multimodal Deep
Learning
https://arxiv.org/pdf/1805.11730.pdf