Is there a flexible way to get the original data indices from each cell of a confusion matrix?
You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).
You can do all sorts of comparisons of the prediction vectors to get the info you want (assuming you're using numpy arrays):
# Which examples did classifier A get wrong?
A_mistakes = np.invert(pred_A == ground_truths)

# Which examples did classifier B get wrong?
B_mistakes = np.invert(pred_B == ground_truths)

# Where did the classifiers make the same mistakes?
common_mistakes = A_mistakes == B_mistakes

# Where did the classifiers make a wrong prediction that the other didn't?
unique_mistakes = np.logical_xor(A_mistakes, B_mistakes)


You can then compare unique mistakes with A_mistakes or B_mistakes to trace them back to a classifier (or anything else you're interested in). You can also just np.sum the binary vectors to count the number of unique mistakes, common mistakes, etc.