Finding P value - Explain
p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )


This is the key for your question: The p-value is the probability that the null hypothesis is true.

If the null hypothesis is true: Your model does not find any differences between groups.
If false: Your model finds differences between groups.

Given that you are using a model which its subyacent assumption is normallity (amongst others), the hypothesis test is to be tried comparing the probability in the context of a normal distribution.

The function stats.norm.cdf returns the probability of "lift being close to zero" if lift is supposed to be "normal". If lift is zero, then there is no difference between groups, so a p-value of &lt;0.01 tell us that the probability that the groups are equal is almost 0, meaning that your groups are different.

The 2 is due to a concept called "two-tailed distribution": The difference between groups can be A greater than B or B greater that A, that's why you measure the difference in either two of the ways.

The addition between standard deviations obeys the concept of:
$Var(X+Y) = Var(X) + Var(Y)$ if $X$ and $Y$ are independent.