{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","results":{"codes":[]},"settings":"","params":[]},"next":{"description":"","pages":[]},"title":"Experiments","type":"basic","slug":"experiments","excerpt":"","body":"Buzz supports the concept of Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also sometimes called \"A/B Testing\" or \"Test/Control Methodology\". This article explains the concepts of how Experiments are implemented in the Buzz API.\n[block:api-header]\n{\n  \"title\": \"Experiments Overview\"\n}\n[/block]\nIt is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased.\n\nBeeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier.\n[block:api-header]\n{\n  \"title\": \"Buzz Implementation\"\n}\n[/block]\nIn Buzz, Experiments are implemented through a two-level hierarchy:\n* A `Test Plan` is associated with a [Campaign](doc:campaigns) and represents a static way of segmenting users for experiments.\n* Every [Line Item](doc:line-items) within that Campaign is assigned a `Test Group`, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users. There is a new API endpoint, [campaign_experiment_assignment](doc:campaign_experiment_assignment), that allows you to associate all Line Items within a Campaign to their respective Test Groups in one POST.\n\nThe available Test Plans are described below:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"test_plan_id\",\n    \"h-1\": \"Test Plan\",\n    \"h-2\": \"Test Groups (with IDs)\",\n    \"0-0\": \"1\",\n    \"0-1\": \"Five Percent Hold-out Plan\",\n    \"1-0\": \"2\",\n    \"1-1\": \"Ten Percent Hold-out Plan\",\n    \"0-2\": \"Control Group (5%) (ID: 1)\\nTest Group (95%) (ID: 2)\",\n    \"1-2\": \"Control Group (10%) (ID: 3)\\nTest Group (90%) (ID: 4)\",\n    \"2-0\": \"3\",\n    \"3-0\": \"4\",\n    \"4-0\": \"5\",\n    \"2-1\": \"Twenty Percent Hold-out Plan\",\n    \"2-2\": \"Control Group (20%) (ID: 5)\\nTest Group (80%) (ID: 6)\",\n    \"3-2\": \"Group A (50%) (ID: 7)\\nGroup B (50%) (ID: 8)\",\n    \"3-1\": \"Two-Way Split Plan\",\n    \"4-1\": \"Three-Way Split Plan\",\n    \"4-2\": \"Group A (33.3%) (ID: 9)\\nGroup B (33.3%) (ID: 10)\\nGroup C (33.4%) (ID: 11)\",\n    \"5-0\": \"6\",\n    \"5-1\": \"Ten-Way Split Plan\",\n    \"5-2\": \"Group A (10%) (ID: 12)\\nGroup B (10%) (ID: 13)\\nGroup C (10%) (ID: 14)\\nGroup D (10%) (ID: 15)\\nGroup E (10%) (ID: 16)\\nGroup F (10%) (ID: 17)\\nGroup G (10%) (ID: 18)\\nGroup H (10%) (ID: 19)\\nGroup I (10%) (ID: 20)\\nGroup J (10%) (ID: 21)\",\n    \"6-0\": \"7\",\n    \"6-1\": \"Three-Way Split with Fifty Percent Hold-Out Plan\",\n    \"6-2\": \"Control (50%) (ID: 22)\\nGroup A (25%) (ID: 23)\\nGroup B (25%) (ID: 24)\",\n    \"7-0\": \"8\",\n    \"7-1\": \"Twenty Five Percent Hold-out Plan\",\n    \"7-2\": \"Control Group (25%) (ID: 25)\\nTest Group (75%) (ID: 26)\"\n  },\n  \"cols\": 3,\n  \"rows\": 8\n}\n[/block]\nA list of active Test Plans may be retrieved at any time using the [View](doc:view) method:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl -X GET \\\"https://[host].api.beeswax.com/rest/test_plan\\\" -b cookies.txt\",\n      \"language\": \"curl\"\n    }\n  ]\n}\n[/block]\nAnd Test Groups within a plan may similarly be retrieved:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl -X GET \\\"https://canary.api.beeswax.com/rest/test_group?test_plan_id=1\\\" -b cookies.txt\",\n      \"language\": \"curl\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Logging and Reporting\"\n}\n[/block]\nWithin [logging](doc:data-feeds) and [reporting](doc:report-queue) new fields are available for:\n* `test_plan_id`\n* `test_plan`\n* `test_group_id`\n* `test group`\n\nWithin logs, the `experiment_user_index` field is provided, which indicates the persistent index of the user, from 0-1000. \n[block:api-header]\n{\n  \"title\": \"Bidding Agents and Augmentors\"\n}\n[/block]\nFor [Bidding Agents](doc:bidding-agent) and [Augmentators](doc:data-augmentation) the user's persistent `experiment_user_index` is provided.","updates":[],"order":9,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"5dd6b3b153dabc005e30aa5e","project":"56c35c56c0c4630d004e864c","version":{"version":"0.5","version_clean":"0.5.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["56c35c57c0c4630d004e8650","56c7b9e5379b311700ed8fe3","56c7bab4606ee717003c4766","56c7bb3613e5400d001e8cbd","56cf3f5a5267d70b00494c4b","56cf3f866c5d7a13005ee894","56fd3956caad892200847bce","599da256e7742b002588bb02"],"_id":"56c35c56c0c4630d004e864f","createdAt":"2016-02-16T17:28:54.864Z","project":"56c35c56c0c4630d004e864c","releaseDate":"2016-02-16T17:28:54.864Z","__v":8},"category":{"sync":{"isSync":false,"url":""},"pages":["56c7c193f9aa3b0d00c8458f","56cb80a4c675f50b00a4b826","56cb83859f4ae20b00644f1f","56cb853a245b841300806f82","56cb863c32011d2500681925","56cb88a4245b841300806f8b","56cb9915245b841300806fa7","56cb9a079f4ae20b00644f48","56cb9b5bc675f50b00a4b859","56cba5929f4ae20b00644f5d","56cba5c5d5c6241d00ef5e93","56cbab9c9f4ae20b00644f76","56cbad69c675f50b00a4b881","56cbb060d5c6241d00ef5ebb","56cf3c4d6c5d7a13005ee88c","56cf3d0e287eb20b009f9ec7","56cf3d7c5267d70b00494c42","56cf3ee0287eb20b009f9ecd"],"title":"Buzz Concepts","slug":"buzz-concepts","order":0,"from_sync":false,"reference":false,"_id":"56c7bab4606ee717003c4766","createdAt":"2016-02-20T01:00:36.607Z","project":"56c35c56c0c4630d004e864c","__v":18,"version":"56c35c56c0c4630d004e864f"},"user":"56c39c05bc41330d009f25d7","createdAt":"2019-11-21T15:56:33.472Z","__v":0,"parentDoc":null}
Buzz supports the concept of Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also sometimes called "A/B Testing" or "Test/Control Methodology". This article explains the concepts of how Experiments are implemented in the Buzz API. [block:api-header] { "title": "Experiments Overview" } [/block] It is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased. Beeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier. [block:api-header] { "title": "Buzz Implementation" } [/block] In Buzz, Experiments are implemented through a two-level hierarchy: * A `Test Plan` is associated with a [Campaign](doc:campaigns) and represents a static way of segmenting users for experiments. * Every [Line Item](doc:line-items) within that Campaign is assigned a `Test Group`, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users. There is a new API endpoint, [campaign_experiment_assignment](doc:campaign_experiment_assignment), that allows you to associate all Line Items within a Campaign to their respective Test Groups in one POST. The available Test Plans are described below: [block:parameters] { "data": { "h-0": "test_plan_id", "h-1": "Test Plan", "h-2": "Test Groups (with IDs)", "0-0": "1", "0-1": "Five Percent Hold-out Plan", "1-0": "2", "1-1": "Ten Percent Hold-out Plan", "0-2": "Control Group (5%) (ID: 1)\nTest Group (95%) (ID: 2)", "1-2": "Control Group (10%) (ID: 3)\nTest Group (90%) (ID: 4)", "2-0": "3", "3-0": "4", "4-0": "5", "2-1": "Twenty Percent Hold-out Plan", "2-2": "Control Group (20%) (ID: 5)\nTest Group (80%) (ID: 6)", "3-2": "Group A (50%) (ID: 7)\nGroup B (50%) (ID: 8)", "3-1": "Two-Way Split Plan", "4-1": "Three-Way Split Plan", "4-2": "Group A (33.3%) (ID: 9)\nGroup B (33.3%) (ID: 10)\nGroup C (33.4%) (ID: 11)", "5-0": "6", "5-1": "Ten-Way Split Plan", "5-2": "Group A (10%) (ID: 12)\nGroup B (10%) (ID: 13)\nGroup C (10%) (ID: 14)\nGroup D (10%) (ID: 15)\nGroup E (10%) (ID: 16)\nGroup F (10%) (ID: 17)\nGroup G (10%) (ID: 18)\nGroup H (10%) (ID: 19)\nGroup I (10%) (ID: 20)\nGroup J (10%) (ID: 21)", "6-0": "7", "6-1": "Three-Way Split with Fifty Percent Hold-Out Plan", "6-2": "Control (50%) (ID: 22)\nGroup A (25%) (ID: 23)\nGroup B (25%) (ID: 24)", "7-0": "8", "7-1": "Twenty Five Percent Hold-out Plan", "7-2": "Control Group (25%) (ID: 25)\nTest Group (75%) (ID: 26)" }, "cols": 3, "rows": 8 } [/block] A list of active Test Plans may be retrieved at any time using the [View](doc:view) method: [block:code] { "codes": [ { "code": "curl -X GET \"https://[host].api.beeswax.com/rest/test_plan\" -b cookies.txt", "language": "curl" } ] } [/block] And Test Groups within a plan may similarly be retrieved: [block:code] { "codes": [ { "code": "curl -X GET \"https://canary.api.beeswax.com/rest/test_group?test_plan_id=1\" -b cookies.txt", "language": "curl" } ] } [/block] [block:api-header] { "title": "Logging and Reporting" } [/block] Within [logging](doc:data-feeds) and [reporting](doc:report-queue) new fields are available for: * `test_plan_id` * `test_plan` * `test_group_id` * `test group` Within logs, the `experiment_user_index` field is provided, which indicates the persistent index of the user, from 0-1000. [block:api-header] { "title": "Bidding Agents and Augmentors" } [/block] For [Bidding Agents](doc:bidding-agent) and [Augmentators](doc:data-augmentation) the user's persistent `experiment_user_index` is provided.