Buzz supports the concept of Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also sometimes called "A/B Testing" or "Test/Control Methodology". This article explains the concepts of how Experiments are implemented in the Buzz API.
It is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased.
Beeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier.
In Buzz, Experiments are implemented through a two-level hierarchy:
Test Planis associated with a Campaign and represents a static way of segmenting users for experiments.
- Every Line Item m](doc:line-items) within that Campaign is assigned a
Test Group, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users. There is a new API endpoint, campaign_experiment_assignment, that allows you to associate all Line Items within a Campaign to their respective Test Groups in one POST.
The available Test Plans are described below:
|test_plan_id||Test Plan||Test Groups (with IDs)|
|1||Five Percent Hold-out Plan||Control Group (5%) (ID: 1)|
Test Group (95%) (ID: 2)
|2||Ten Percent Hold-out Plan||Control Group (10%) (ID: 3)|
Test Group (90%) (ID: 4)
|3||Twenty Percent Hold-out Plan||Control Group (20%) (ID: 5)|
Test Group (80%) (ID: 6)
|4||Two-Way Split Plan||Group A (50%) (ID: 7)|
Group B (50%) (ID: 8)
|5||Three-Way Split Plan||Group A (33.3%) (ID: 9)|
Group B (33.3%) (ID: 10)
Group C (33.4%) (ID: 11)
|6||Ten-Way Split Plan||Group A (10%) (ID: 12)|
Group B (10%) (ID: 13)
Group C (10%) (ID: 14)
Group D (10%) (ID: 15)
Group E (10%) (ID: 16)
Group F (10%) (ID: 17)
Group G (10%) (ID: 18)
Group H (10%) (ID: 19)
Group I (10%) (ID: 20)
Group J (10%) (ID: 21)
|7||Three-Way Split with Fifty Percent Hold-Out Plan||Control (50%) (ID: 22)|
Group A (25%) (ID: 23)
Group B (25%) (ID: 24)
|8||Twenty Five Percent Hold-out Plan||Control Group (25%) (ID: 25)|
Test Group (75%) (ID: 26)
A list of active Test Plans may be retrieved at any time using the View method:
curl -X GET "https://[host].api.beeswax.com/rest/test_plan" -b cookies.txt
And Test Groups within a plan may similarly be retrieved:
curl -X GET "https://canary.api.beeswax.com/rest/test_group?test_plan_id=1" -b cookies.txt
Within logs, the
experiment_user_index field is provided, which indicates the persistent index of the user, from 0-1000.
Updated over 3 years ago