Case when Out of Bag Error and Test error differs a lot in Random Forest
What type of sampling method did you use to split the data? I'm guessing, by looking at the ratio of classes in your OOB confusion matrix and your test set confusion matrix that it wasn't stratified, is this correct? For instance, the class ratio in the OOB data is 4000:800 whereas in the test set it is 7761:43. As a starting point, you may want to split your data using stratified sampling so that the classes are fairly represented (in a statistical sense) in both your training data and test data. The ratio of classes in the test set should be identical or almost the same as the ratio used in the training data.

I think this would be a good starting point to figuring out the problem.