Reviewer questioned credibility of paper and results after first revision, which led to us finding a major error. Should we revise or withdraw?
I suggest that you should withdraw your submission until you can figure out what's going on. Good science should be reproducible, and this is not. You did not appropriately critique your own results, and put forth a result that seemed to &quot;look good&quot;. You've now changed some part of your modeling methodology, and gotten another set of results that &quot;look good&quot;, even though you cannot articulate any theoretical or practical reason why this set of results is any more sound than the last. The first result did not pass the &quot;smell test&quot; of reasonable performance, and you kind of shot yourself in the foot by failing to investigate. Now you have a result that does pass the &quot;smell test&quot;, but the reviewer is aware that your diligence in self-criticism is somewhat lacking.
You're essentially telling the reviewer that &quot;the last result was an error, but this one is not, although we are unable to explain any difference between the two.&quot; It's a really big ask for the reviewer to have confidence in your updated methodology, if you yourself cannot explain why it is any more correct than what you did the first time around.