# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] hide_input=true toc=true
# <h1>Table of Contents<span class="tocSkip"></span></h1>
# <div class="toc"><ul class="toc-item"><li><span><a href="#Input-data" data-toc-modified-id="Input-data-1">Input data</a></span></li><li><span><a href="#Key-Components-of-an-AB-Test" data-toc-modified-id="Key-Components-of-an-AB-Test-2">Key Components of an AB Test</a></span><ul class="toc-item"><li><span><a href="#Running-an-AB-test-in-✨abracadabra✨-is-as-easy-as-✨1,-2,-3✨:" data-toc-modified-id="Running-an-AB-test-in-✨abracadabra✨-is-as-easy-as-✨1,-2,-3✨:-2.1">Running an AB test in ✨abracadabra✨ is as easy as ✨1, 2, 3✨:</a></span><ul class="toc-item"><li><span><a href="#Hypotheses" data-toc-modified-id="Hypotheses-2.1.1">Hypotheses</a></span></li><li><span><a href="#Inference-Methods" data-toc-modified-id="Inference-Methods-2.1.2">Inference Methods</a></span></li><li><span><a href="#Example" data-toc-modified-id="Example-2.1.3">Example</a></span></li></ul></li></ul></li><li><span><a href="#Interpreting-results" data-toc-modified-id="Interpreting-results-3">Interpreting results</a></span><ul class="toc-item"><li><ul class="toc-item"><li><span><a href="#Top-Plot:-Sample-Distributions" data-toc-modified-id="Top-Plot:-Sample-Distributions-3.0.1">Top Plot: Sample Distributions</a></span></li><li><span><a href="#Middle-Plot:-Central-Tendencies" data-toc-modified-id="Middle-Plot:-Central-Tendencies-3.0.2">Middle Plot: Central Tendencies</a></span></li><li><span><a href="#Bottom-Plot:-Deltas" data-toc-modified-id="Bottom-Plot:-Deltas-3.0.3">Bottom Plot: Deltas</a></span></li></ul></li></ul></li><li><span><a href="#Bootstrap-Hypothesis-Tests" data-toc-modified-id="Bootstrap-Hypothesis-Tests-4">Bootstrap Hypothesis Tests</a></span></li><li><span><a href="#Bayesian-Hypothesis-Tests" data-toc-modified-id="Bayesian-Hypothesis-Tests-5">Bayesian Hypothesis Tests</a></span><ul class="toc-item"><li><span><a href="#Bayesian-Model-Specification" data-toc-modified-id="Bayesian-Model-Specification-5.1">Bayesian Model Specification</a></span></li></ul></li><li><span><a href="#Including-Segmentations" data-toc-modified-id="Including-Segmentations-6">Including Segmentations</a></span></li><li><span><a href="#Running-multiple-tests,-and-multiple-comparison-control" data-toc-modified-id="Running-multiple-tests,-and-multiple-comparison-control-7">Running multiple tests, and multiple comparison control</a></span></li><li><span><a href="#Custom-Metrics" data-toc-modified-id="Custom-Metrics-8">Custom Metrics</a></span></li><li><span><a href="#Working-with-other-types-of-variables" data-toc-modified-id="Working-with-other-types-of-variables-9">Working with other types of variables</a></span><ul class="toc-item"><li><span><a href="#Continuous-Variables" data-toc-modified-id="Continuous-Variables-9.1">Continuous Variables</a></span><ul class="toc-item"><li><span><a href="#Bayesian-models-for-continuous-variables" data-toc-modified-id="Bayesian-models-for-continuous-variables-9.1.1">Bayesian models for continuous variables</a></span></li></ul></li><li><span><a href="#Counts-/-Rates-variables" data-toc-modified-id="Counts-/-Rates-variables-9.2">Counts / Rates variables</a></span><ul class="toc-item"><li><span><a href="#Bayesian-models-for-count-variables" data-toc-modified-id="Bayesian-models-for-count-variables-9.2.1">Bayesian models for count variables</a></span></li></ul></li></ul></li></ul></div>
# -

# # ✨abracadabra✨ Basics

# ## Input data
# ✨ABracadabra✨ takes as input a [pandas](https://pandas.pydata.org/) `DataFrame` containing experiment observations data. Each record represents an observation/trial recorded in the experiment and has the following columns:
#
# - **One or more `treatment` columns**: each treatment column contains two or more distinct, discrete values that are used to identify the different groups in the experiment
# - **One or more `metric` columns**: these are the values associated with each observation that are used to compare groups in the experiment.
# - **Zero or more `attributes` columns**: these are associated with additional properties assigned to the observations. These attributes can be used for any additional segmentations across groups.
#
# To demonstrate, let's generate some artificial experiment observations data. The `metric` column in our dataset will be a series of binary outcomes (i.e. `True`/`False`, here stored as `float` values). This binary `metric` is analogous to *conversion* or *success* in AB testing. These outcomes are simulated from three different Bernoulli distributions, each associated with the `treatement`s named `"A"`, `"B"`, and `"C"`. and each of which has an increasing average probability of *conversion*, respectively. The simulated data also contains four `attribute` columns, named `attr_*`.

# +
from abra.utils import generate_fake_observations

# generate some fake binary trial data
binary_data = generate_fake_observations(
    distribution='bernoulli',  # binary data
    n_treatments=3,
    n_attributes=4,
    n_observations=1000
)
binary_data.head()
# -

# The resulting data have a single `treatment` column, called (creatively) `'treatment'`, a single `metric` column, called `'metric'`, and four `attribute` columns, `atr_0`, `attr_1`, `attr_2`, and `attr_3`. The `treatment` column has 3 distinct treatments: `"A"`, `"B"`, and `"C"`, and the metric takes on boolean/binary values drawn from a [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution).

# ## Key Components of an AB Test

# The three key components of running an AB test in ✨abracadabra✨ are:
#
# - **The `Experiment`**, which holds the raw observations data recorded from, and any metadata associated with an AB experiment.
# - **The `HypothesisTest`**, which defines the statistical inference procedure applied to the experiment data.
# - **The `HypothesisTestResults`**, which is the statistical artifact that results from running a `HypothesisTest` against an `Experiment`'s observations. The `HypothesisTestResults` are used to summarize, visulize, and interpret the inference results and make decisions based on these results.

# ### Running an AB test in ✨abracadabra✨ is as easy as ✨1, 2, 3✨:
# 1. Initialize your `Experiment` with observations and any metadata
# 2. Define your `HypothesisTest`. This requires defining the `hypothesis` and `inference_method`. The relevant inference method used will depend on the support of your observations. A list of supported hypotheses and inference methods for different types of observation variables are shown below:
#
# #### Hypotheses
# | Hypothesis | Hypothesis Type | `hypothesis` parameter  |
# |---|---|---|
# | "The treatment is larger than the control" | one-tailed | `"larger"` |
# | "The treatment is smaller than the control" | one-tailed | `"smaller"` |
# | "The treatment is not equal to the control" | two-tailed | `"unequal"` |
#
#
# #### Inference Methods
#
# | Variable Type | Model Class| `inference_method` parameter  |
# |---|---|---|
# | Continuous | Frequentist| `'means_delta'` (t-test) |
# |  | Bayesian| `'gaussian'`, `'exp_student_t'`|
# | Binary / Proportions | Frequentist| `'proportions_delta'` (z-test) |
# |  | Bayesian| `'beta'`, `'beta_binomial'`, `'bernoulli'`  |
# | Counts  |Frequentist| `'rates_ratio'`  |
# |  |Bayesian| `'gamma_poisson'`  |
# |  |Bayesian| `'gamma_poisson'`  |
# | Non-parametric  |Bootstrap| `'bootstrap'` |
#
# 3. Run the test against your experiment and interpret the resulting `HypothesisTestResults`

# #### Example
#
# Below we demonstrate a standard AB test workflow in ✨abracadabra✨. Namely we:
# 1. Initialize an `Experiment` instance `exper` with our artifical binary observations generated above.
# 2. We then initialize a `HypothesisTest` instance `ab_test` that tests if `treatment` `"B"` is `"larger"` than treatment `"A"` based on the metric values in the `"metric"` column of the dataframe. The hypothesis test uses a Frequentist method `'proportions_delta'` that is dedicated to detecting differences between binary samples. 

# +
from abra import Experiment, HypothesisTest

# Initialize the Experiment
exper = Experiment(data=binary_data, name="Demo Experiment")

# Initialize the A/B test
ab_test = HypothesisTest(
    metric="metric",
    treatment="treatment",
    control="A", variation="B",
    hypothesis="unequal",
    inference_method="proportions_delta"
)

# Run the test with an alpha of 0.5; get back a HypothesisTestResults object
ab_test_results = exper.run_test(ab_test, alpha=.05)

# Check the test results decision
assert ab_test_results.accept_hypothesis
# -

# ## Interpreting results
# Each `HypothesisTestResults` has its own `display()` and `visualize()` methods that can be used to interpret the results of the test. The `display()` method prints out the results to the console, while `visualize` plots a visual summary of the results.

# Print the test results to the console
ab_test_results.display()

# Providing an `outfile` argument to the `visualize` method will save the results figure to an image file.

# +
# %pylab inline
# Visualize the test results
outfile = '/tmp/abracadabra_demo_test_results.png'
ab_test_results.visualize(outfile=outfile)

# Show that results figure exists
# !ls -lah {outfile}
# -

# The resulting frequentist test results (displayed and visualized above) indicate that hypothesis `"B != A"` should be accepted. A breakdown of the results plot is as follows:
#
# #### Top Plot: Sample Distributions
# The top plot compares the _parameterized representation of the sample distributions_ with parameters being derived from the experiment observations.
#
# The large degree of separation between the two distributions indicates that the `"B"` group is qualitatively larger than `"A"`
#
# #### Middle Plot: Central Tendencies
# The middle plot compares the _central tendiency estimates_ of the two sample groups--in this case the, average probability of success on a given trial--as well as adds _Standard Errors_ calculate for the central tendency estimates.
#
# We can see that there is no overlap of the standard errors, further indicating that we can be confident that the two groups are likely different, and that `"B"` is larger than `"A"`.
#
# #### Bottom Plot: Deltas
# The bottom plot gives the estimate of the _difference in central tendencies_, in this case `ProportionsDelta`, as well as 95% Confidence Intervals on this difference estimate (This is a two-tailed test, checking that `"B"` is not equal to `"A"`, so the upper bound on ProportionsDelta is `inf`).
#
# We can see that the confidence interval on the difference between the two groups does not intersect with the `ProportionsDelta=0` line, indicating a statistically significant difference between the two samples.

# ## Bootstrap Hypothesis Tests
#
# If your samples do not follow standard parametric distributions (e.g. Gaussian, Binomial, Poisson), or if you're comparing more exotic descriptive statistics (e.g. median, mode, etc) then you might want to consider using a non-parametric [Bootstrap Hypothesis Test](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)).  Running bootstrap tests is easy in ✨abracadabra✨, you simply use the `"bootstrap"` `inference_method`.

# +
# Tests and data can be copied via the `.copy` method.
def my_statistic(samples):
    """Boootstrap tests support custom test statistics. Here
    we use re-define the mean to compare with parametrics
    tests.
    """
    return np.mean(samples)


bootstrap_ab_test = ab_test.copy(
    inference_method='bootstrap',
    infer_kwargs={'statistic_function': my_statistic}
)

# run the test
bootstrap_ab_test_results = exper.run_test(bootstrap_ab_test)
bootstrap_ab_test_results.display()
bootstrap_ab_test_results.visualize()
# -

# Notice that the `"bootstrap"` hypothesis test results above--which are based on resampling the data set with replacent--are very similar to the results returned by the `"proportions_delta"` parametric model, which are based on descriptive statistics and model the data set as a Binomial distribution. The results will converge as the sample sizes grow.

# ## Bayesian Hypothesis Tests
# In addition to common Frequentist test, running Bayesian analogs is simple. Simply intitialize the `HypothesisTest` with a Bayesian `method` parameter. For example, the Bayesian analogs to the `'proportions_delta'` method are the `'binomial'`, `'beta_binomial'`, or `'bernoulli'` methods.

# +
binomial_ab_test = ab_test.copy(inference_method='binomial')

# run the test
binomial_ab_test_results = exper.run_test(binomial_ab_test)
assert binomial_ab_test_results.prob_greater > .95
binomial_ab_test_results.display()
binomial_ab_test_results.visualize()
# -

# Bayesian `HypothesisTestResults` have their own analogous `display` and `visualize` methods that can be used to interpret the results of the analysis. Notice the results of the `ab_test_results` and `binomial_ab_test_results` each support a difference in proportions of approximately 0.14.
#
# Note how the mean parameter and 95% HDI sampled from the Bayesian model are very similar to the mean and 95% confidence interval used in the Frequentist test `ab_test_results`. Similarly, the deltas in proportion parameter $p$ sampled from the model provide aligned with the `ProportionsDelta` estiamted for the Frequentist test.

# ### Bayesian Model Specification
# Bayesian models allow the experimenter to incorporate prior beliefs. This can be helpful when you have little data, or can provide sound domain knowledge of baselines. Specifying custom priors is also straight-forward using `abracadaba`, simply pass in a `model_params` argument during `HypothesisTest` initialization. Below we demonstrate by running another Bayesian hypothesis test, this time with a hierarchical [Beta-Binomial model](https://en.wikipedia.org/wiki/Beta-binomial_distribution#:~:text=In%20probability%20theory%20and%20statistics,is%20either%20unknown%20or%20random.). This model allows the user to specify a prior over the base probability $p$ by setting two hyperparameters for the Beta Distribution $\alpha$ and $\beta$ such that the mean prior has a value of 
#
# $$ p = \frac{\alpha}{\alpha + \beta}$$
#
# where the larger $\alpha$ and $\beta$. Let's put a super-strong prior on $p$ and see how it affects the inference results.

# +
# Run Bayesian test with custom priors

# strong prior that p = alpha / (alpha + beta) = 0.33
beta_prior_params = dict(alpha=500., beta=1000.)

bb_ab_test = HypothesisTest(
    metric='metric',
    control='A', variation='C',
    inference_method='beta_binomial',
    model_params=beta_prior_params
)

# run the test
bb_ab_test_results = exper.run_test(bb_ab_test)
assert bb_ab_test_results.prob_greater > .95
bb_ab_test_results.display()
bb_ab_test_results.visualize()
# -

# Here we see that the strong prior of $p=0.33$ influences the proportion parameters to values around 0.33. This causes our delta samples from the model be much smaller because two distributions that are forced to be near a prior value (here 0.33) will be located close to one another. Even with this strong prior, the model identifies that the two distributions are, in fact different, and that `P(C > A)` near 1.
#
# Note that if there were more data in the experiment, these parameter estimates would gradually move toward the data distribution, rather than the super-confident prior distribution, providing results that are similar to the results provided by the  `binomial`  (which can be thought of as a special case of the Beta-Binomial with a weak, agnostic prior) and `proportions_delta` inference methods.

# ## Including Segmentations
# ✨abracdabra✨ supports the ability to segment experiment observations based on one or more attributes in your dataset using the `segmentation` argument to `HyptothesisTest`. The segmentation can be a string or list of string expressions, each of which follow the [pandas query API](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html)

# +
# Initialize an A/B test with additional segmentation on the 'attr_1' attribute
ab_test_segmented = HypothesisTest(
    metric='metric',
    control='A', variation='B',
    inference_method='proportions_delta',
    hypothesis='larger',
    segmentation="attr_1 == 'A1a'"
)

# Run the segmented test
ab_test_segmented_results = exper.run_test(ab_test_segmented)
assert not ab_test_segmented_results.accept_hypothesis  # B is larger

# Display results (notice reduced sample sizes)
ab_test_segmented_results.display()
ab_test_segmented_results.visualize()
# -

# We now see that if we dig into a particular segment, namely the segement defined by `"attr_1 == 'A1a'"`, we can no longer accept the hypothesis that `"B is larger"`. This is indicated by a `ProportionsDelta` that overlaps substantially with the line indicating `0` difference between the two samples.

# ## Running multiple tests, and multiple comparison control
# When running multiple tests on the same metric, you'll need to control for [multiple comparisons](https://en.wikipedia.org/wiki/Multiple_comparisons_problem). This is handled by running a `HypothesisTestSuite`, which takes in a list of hypothesis tests.
#
# Below we run 3 independent tests comparing A to A, B to A and C to A, and set the correction `method` to `'bonferonni'`, which simply updates the effective $\alpha_{corrected} = \frac{\alpha}{N_{tests}}$. Our original value for `alpha=0.05`, thus the corrected value would be $\frac{0.05}{3} = 0.0167$

# +
from abra.hypothesis_test import HypothesisTestSuite
from copy import deepcopy

# Use the `HypothesisTest.copy` method for duplicating test
# configurations, while overwriting specific parameters, in
# this case `variation` parameter
aa_test = ab_test.copy(variation='A')
ac_test = ab_test.copy(variation='C')

# Initialize the `HypothesisTestSuite`
test_suite = HypothesisTestSuite(
    tests=[aa_test, ab_test, ac_test],
    correction_method='bonferroni'
)

# Run tests
test_suite_results = exper.run_test_suite(test_suite)

print(test_suite_results)

# Print results
test_suite_results.display()
# -

# Note that the alpha has been `corrected` with `MC Correction='bonferroni'`

# +
# test_suite_results.visualize()
# -

# The `HypothesisTestSuite` supports the following multiple comparison strategies:
# - [Sidak](http://en.wikipedia.org/wiki/%C5%A0id%C3%A1k_correction) (default)
# - [Bonferonni](http://en.wikipedia.org/wiki/Bonferroni_correction)
# - [Benjamini-Hochberg false-discovery rate](http://pdfs.semanticscholar.org/af6e/9cd1652b40e219b45402313ec6f4b5b3d96b.pdf)

# ## Custom Metrics
# ✨abracadabra✨ also supports the use of custom metrics, which can transform and combine information from one or more columns. Below we create a `CustomMetric` always makes the `variation` greater than the `control` by adding a constant offset (plus noise) to the value of the the `control`.

# +
from abra import CustomMetric
import numpy as np

def custom_metric(row):
    """
    Define a custom 'metric' where the control is always better.
    """
    return 4 + np.random.rand() if row['treatment'] == 'A' else np.random.rand()

custom_test = HypothesisTest(
    metric=CustomMetric(custom_metric),
    control='A',
    variation='B',
    inference_method='means_delta',  # Note we use a t-test here
    hypothesis='unequal'
)

custom_test = exper.run_test(custom_test)
custom_test.display()
# -

# We see that, as expected, we have highly significant results, accepting the hypothesis that `'B != A'`

# ## Working with other types of variables
# The examples above demonstrate running AB tests for variables that take on binary values--i.e. variables that take on values that exist in the interval $[0, 1]$. This is a pretty common scenario, as a lot of AB tests measure metrics like conversions at various stages in a UX funnel. However, ✨abracadabra✨ also supports inference methods for other types of variables, like continuous variables (e.g. time spent on a page) and counts/rate variables (e.g. number of clicks on a button per unit time).

# ### Continuous Variables
# Many continuous variables follow Gaussian distribution, thus deltas between samples of continuous variables are often modeled based on differences of Gaussian-distributed random variables. The most common statistical inference for this scenario is the good ole' [Student's t-test](https://en.wikipedia.org/wiki/Student%27s_t-test), referred to in ✨abracadabra✨ as `"means_delta"`, as it tests for differences in means of the underlying Gaussian distributions from each of treatments.
#
# Below we generate some Gaussian-distributed observations and show how ✨abracadabra✨can be used to run a t-test using the common `Experiment->HypothesisTest->HypothesisTestResults` workflow.

# generate some fake Gaussian-distributed trial data
gaussian_data = generate_fake_observations(
    distribution='gaussian',  # binary data
    n_treatments=3,
    n_attributes=4,
    n_observations=1000
)
gaussian_data.head()

# Here we use the `inference_method="means_delta"` argument to run the t-test

# +
# Initialize the Experiment
exper = Experiment(data=gaussian_data, name="Demo Experiment")

# Initialize the A/B test
gaussian_ab_test = HypothesisTest(
    metric="metric",
    treatment="treatment",
    control="A", variation="B",
    hypothesis="unequal",
    inference_method="means_delta"
)

# Run the test with an alpha of 0.5; get back a HypothesisTestResults object
gaussian_ab_test_results = exper.run_test(gaussian_ab_test, alpha=.05)

# Check the test results decision
assert gaussian_ab_test_results.accept_hypothesis
gaussian_ab_test_results.display()
gaussian_ab_test_results.visualize()
# -

# #### Bayesian models for continuous variables
# The Bayesian analog to the t-test is called the "Hierarchical Gaussian" and involves modeling the observations as a generative process where each point is sampled from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. The model is "hierarchical" because it also assumes there is a distribution over both $\mu$ and $\sigma^2$ as well, namely $\mu \sim \text{Normal}(\bar{x}, \text{std(x)})$ and $\sigma \sim \text{Uniform}(0, \sigma_{max})$, where $\bar{x}$ and  $\text{std(x)}$ are the empirical mean and standard deviation of the observations, and $\sigma_{max}$ is a user-specified hyperparameter.
#
#
# That all sounds pretty complicated, right? Well, in ✨abracadabra✨ it's easy to run inference using this model. We simply update the `inference_method`:

# +
bayesian_gaussian_ab_test = ab_test.copy(inference_method='gaussian')

# Run the test with an alpha of 0.5; get back a HypothesisTestResults object
bayesian_gaussian_ab_test_results = exper.run_test(bayesian_gaussian_ab_test, alpha=.05)

# Check the test results decision
assert bayesian_gaussian_ab_test_results.accept_hypothesis
bayesian_gaussian_ab_test_results.display()
bayesian_gaussian_ab_test_results.visualize()

# -

# ### Counts / Rates variables
# ✨abracadabra✨ also supports analysis of counts variables such as clicks or page views per unit time. These discrete, countable variables often follow a Poisson distribution. Rather than modeling the _difference_ between the two treatments, we instead model whether the ratio of the two counts is statistically different than 1. The reasoning being that if the two treatments have the same number of counts per the same unit of time (and thus the same _rate_) then their ratio will be close to unity. Accordingly, in ✨abracadabra✨, this is called a `"rates_ratio"` inference method.
#
# Below we'll run an AB test on syntetic data drawn from a Poisson distribution, and test to see if the two distributions are statistically different.

# generate some fake Gaussian-distributed trial data
counts_data = generate_fake_observations(
    distribution='poisson',  # binary data
    n_treatments=3,
    n_observations=1000
)
counts_data.head()

# +
# Initialize the A/B test
exper = Experiment(data=counts_data, name="Demo Experiment")

poisson_ab_test = HypothesisTest(
    metric="metric",
    treatment="treatment",
    control="B", variation="C",
    hypothesis="unequal",
    inference_method="rates_ratio"
)

# Run the test with an alpha of 0.5; get back a HypothesisTestResults object
poisson_ab_test_results = exper.run_test(poisson_ab_test, alpha=.05)

# Check the test results decision
poisson_ab_test_results.display()
poisson_ab_test_results.visualize()
# -

# Here we can see that the ratio of the two rate parameters ranges between 1.38 and 1.68 (95% confidence), with no overlap with the value 1. This indicates that the variation `"C"`s location parameter is approximately 1.5x that of the control `"B"`, which makes sense, given the mean estimates for the two treatments are approximately 3 and 2, respectively.

# #### Bayesian models for count variables
# The Bayesian analog to the rates ratio test is what's called the [Gamma-Poisson model](http://www.math.wm.edu/~leemis/chart/UDR/PDFs/Gammapoisson.pdf). In this model the observations are assumed to be generated from a Poisson distribution with location parameter $\lambda$. Similarly to the "Hierarchical Gaussian" Bayesian, there is a prior distribution associated with $\lambda$ (that's what makes it Bayesian!). Namely $\lambda \sim \text{Gamma}(\alpha, \beta)$. Here the hyperparameters $\alpha$ and $\beta$ can be set by the experimenter to encode any intuitions or domain knowledge about the problem.
#
# Though this sounds complicated, implementing a hypothesis test using an inference method based off of the Gamma-Poisson model is not:

# +
bayesian_poisson_ab_test = poisson_ab_test.copy(inference_method='gamma_poisson')

# Run the test with an alpha of 0.5; get back a HypothesisTestResults object
bayesian_poisson_ab_test_results = exper.run_test(bayesian_poisson_ab_test)

# Check the test results decision
bayesian_poisson_ab_test_results.display()
bayesian_poisson_ab_test_results.visualize()
# -

# Note here that samples drawn from the Bayesian model provide similar central tendency interval estimates to those calculated by the analytical rates ratio model. However, unlike the rates ratio model which looks at the ratio of central tendencies, the Bayesian AB tests provides _deltas_ or differences amongst the rate parameter samples drawn from the model. Thus differences in $\lambda$ samples that are far away from zero (in this case the diffence is--and should be--approximately equal to one) indicate significant difference between the treatments when interpreting the Bayesian counts AB test.

StarcoderdataJupyterScriptsDedupFiltered