How to debug weird and random behavior I'm trying to figure out how to debug the following behavior and would appreciate some ideas on how to track down the source of the problem.

I run a website were some users have privileges to create and edit nodes and taxonomy terms.

Upon creating content or taxonomy terms, users report that, on occasion, content is created 3 times, for both entity types.

What steps can I take to find the culprit for this behavior knowing that...?


I have made sure users only click the submit button once, but that hasn't made any difference.
Both Drupal and PHP error logs are clean.  I can see 3 logged messages confirming content has been triplicated but nothing else.
It only happens in live environment.  I haven't been able to reproduce it in development environment
It happens randomly.  Users have reported this only happens ocasionally.
When this happens, any rules that react to the condition 'After saving node ...' or 'After updating node...' are also fired 3 times
Treat it like any bug: 


Find a way to reliably reproduce the problem (this is essential to debugging any code issue for obvious reasons; the question "how do I debug this problem I can't reproduce?" doesn't have an answer, let alone a Drupal-specific one). You may need to set up a clone of your production environment as the issue might not be related to Drupal (e.g. you could have a rogue process duplicating MySQL records that Drupal doesn't get involved with)
Fire up XDebug or your debugger of choice.
Step through every single line of the code that gets involved in the problem you're seeing, keeping an eye on what parameters are being passed to methods, and if they contain the expected data. Basically run through an entire bootstrap for the page(s) you're having problems with, starting at index.php, narrowing down your target search area as you go.
Fix when you've identified the problem (you're not done yet though...)
To help for next time, write a test that fails before you implement your fix, and passes afterwards. Make it part of your deployment workflow to run this test after every change you make. Consider adding tests for other major pieces of functionality as you decide which are the most important, to help avoid this sort of situation in future.