AB testing : When AA testing doesn't work After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old AA testing.
AA testing consists in dividing randomly the users between two branches, making both branches have the exact same experience, and testing that conversion rates on both branches are not significantly different, which would mean that both branches are not actually treated the same (or that the population is not uniformly distribued between two branches)
As you can imagine, this did not happen at all. We sent roughly 3500 mails to each branch, and one of these exhibits a 40% conversion while the other only has 32%, which is a difference that has a p-value of the order of 10^-12
Furthermore, we partitionned the time in 4 periods, each one of these exhibit a very significant difference with the other one.

Now what am I supposed to do? I would like to have a theoretical discussion about this with Oracle's support, but of course, everybody that sees an AA testing go wrong will believe you did not respect a proper methodology.
My question : I would like some pointers of people with more experience than me in data science to help me to :
- Express my problem in the clearest possible way to get the company that sells me this solution to take an interest in the issue here
- Understand if I am forgetting to check anything on my side to explain this discrepancy
The exact process that is used from our server to the final client is :

we send every x hours a list of a few hundred people to Responsys

Responsys is supposed to affect them in an independent and identically distributed way using random numbers generated with java.util.random.nextfloat()

then the list is filtered to remove undeliverable people, and each branch is sent the exact same email


Oracle is pretty laconic in his description, but I think the following lines from their documentation is supposed to describe an i.i.d. uniform distribution.

During launch, for each email recipient:
a. generate a random number ranging in [0, 1] using java.util.Random.nextFloat()
b. decide the bucket this random number is in
c. send the campaign in that bucket to the recipient
I 'solved' the issue with Oracle's support, I still post an answer here because some of what I encountered can be useful :  According to Oracle's support, this bug in AB testing programs was known for the last 9 month, but corrected only this week-end.

So my problem is supposedly solved (I will re-launch a test and see if results are now consistent), but I still have some pointers if someone encounters the same problem I did :


Make sure this is not a statistical artifact : test your hypothesis on several independant periods. I used a Fischer test because it is exact for a binomial framework, it is what you should use if your population/conversion rate is too small for a normal approximation
Respect hypothesis testing framework : start your test after you formulated your hypothesis
Get a very very low p-value : No one will listen to your complaint about faulty AB testing if you come up with a 3% p-value, and they would be right. A bayesian analysis probably points to a false positive. 
Make sure the random numbers that are used to split the populations are actually random (and not, say a 50/50 split)
Imagine all the possibilities that your reporting might be wrong. I thought I had imagined them all, but if you read below the real reason behind the bug, you will find that faulty reporting can be hidden very far away in parts of the environment you don't control or even have a view in.


I have the actual answer from Oracle that describes the bug that happened, in case someone has the exact same problem or wants to know how this kind of bug happens : 

For programs triggered by event, their engine does not have the capability to associate the different open and clicks events to the correct email. In the case several emails have been sent through the same program, Responsys assumes the event concerns the earliest email that has been sent through this program.

In my case, the program could have been sent several times during the period.

In the period where I was not splitting my emails through two branches, the name of the campaign I used was already A  (the same than one of my branches in my testing period)

You can see where this is going : Every customer opening any email was marked as having read the "A" email, as long as they had been into the program in the 3 months before I started my AB testing, which caused a faulty reporting.

So actually both branches were sent to the correct customers, and it is just a reporting problem that comes from an approximative algorithm on Responsys side.

The workaround I will try now is simply to create two brand new "campaign" objects before starting a test rather than using a campaign that already existed.