What does "Test Group" mean?

Definition of Test Group in the context of A/B testing (online controlled experiments).

What is a Test Group?

A Test Groups is a group of (usually randomly assigned) users/sessions/pageviews/etc. that are exposed to a certain treatment. The test groups are then compared to the Control Group to check for discrepancies large enough to reject the null hypothesis of interest.

In most tests the control and test groups are of equal sizes as it facilitates easier interpretation due to lack of generalizability issues, is easier to compute and is usually an optimal design. In an adaptive sequential design or a multi-armed bandit design the weights of the control and test groups are usually different and vary across the test duration.

Adding or dropping test groups mid-experiment should only be done using proper adaptive sequential design methods, otherwise the error probabilities of the A/B test will suffer.

Related A/B Testing terms

Control Group


Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter

