{"_id":"57a1f808cd51b22d00f62302","project":"56c35c56c0c4630d004e864c","category":{"_id":"56c7b9e5379b311700ed8fe3","pages":["56c7ba3853cafe0d00a53daa"],"project":"56c35c56c0c4630d004e864c","version":"56c35c56c0c4630d004e864f","__v":1,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-02-20T00:57:09.551Z","from_sync":false,"order":2,"slug":"stinger","title":"Stinger"},"user":"56c39c05bc41330d009f25d7","githubsync":"","__v":2,"parentDoc":null,"version":{"_id":"56c35c56c0c4630d004e864f","project":"56c35c56c0c4630d004e864c","__v":8,"createdAt":"2016-02-16T17:28:54.864Z","releaseDate":"2016-02-16T17:28:54.864Z","categories":["56c35c57c0c4630d004e8650","56c7b9e5379b311700ed8fe3","56c7bab4606ee717003c4766","56c7bb3613e5400d001e8cbd","56cf3f5a5267d70b00494c4b","56cf3f866c5d7a13005ee894","56fd3956caad892200847bce","599da256e7742b002588bb02"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"0.5.0","version":"0.5"},"updates":["599f5badc8dfe7000f6ab87e","5a306256434301002850e988"],"next":{"pages":[],"description":""},"createdAt":"2016-08-03T13:56:24.708Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":5,"body":"One of the most important considerations in working with an RTB bidder is designing the data pipeline by which auction, bid, and win events are processed and analyzed.\n\nUnlike DSPs, with the Beeswax Bidder-as-a-Service technology you can access the full, unfiltered stream of RTB events, in much the same way you would if you were building a bidder from scratch. \n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Streaming vs Batch\"\n}\n[/block]\nThe first consideration for designing your data pipeline is whether you prefer to get the data in batch form, or as a continuous stream. For the very high-volume data, like auction logs, Beeswax only supports batch data. For win logs (impressions) Beeswax supports both methods, and there are pros and cons to each:\n[block:parameters]\n{\n  \"data\": {\n    \"0-0\": \"Batch\",\n    \"1-0\": \"Stream\",\n    \"h-0\": \"Pipeline Method\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Pros\",\n    \"h-3\": \"Cons\",\n    \"0-1\": \"Hourly or daily files of data placed in an S3 bucket.\",\n    \"1-1\": \"Near real-time data in [protobuf](https://developers.google.com/protocol-buffers/) format sent over http or to AWS [Kinesis](https://aws.amazon.com/kinesis/)\",\n    \"0-2\": \"Fairly easy to ingest, fault tolerant\",\n    \"0-3\": \"Delay in utilizing data. Also may include many files written per hour.\",\n    \"1-2\": \"Use data as fast as you can process it\",\n    \"1-3\": \"Higher cost and complexity to support data ingestion\"\n  },\n  \"cols\": 4,\n  \"rows\": 2\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Data Definitions\"\n}\n[/block]\nColumn definitions, protobuf mapping, and field lists can be found on [Google Docs](https://docs.google.com/spreadsheets/d/1cIMgCNyAech19JUqS5mMKQ_UkJ-tOxVCY85BXiSnWNQ/edit?usp=sharing). \n\nA complete **data dictionary** can be found in this [Google Doc](https://docs.google.com/spreadsheets/d/1cIMgCNyAech19JUqS5mMKQ_UkJ-tOxVCY85BXiSnWNQ/edit?usp=sharing).\n\nBeeswax makes three types of data available from Stinger as described in the chart below. Based on your use case you may need some, or all of this data. Because some of this data can be quite large, additional fees may apply (contact your Account Manager for more information). \n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Data Type\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Batch Field Manifest\",\n    \"0-0\": \"Auctions\",\n    \"1-0\": \"Bids\",\n    \"4-0\": \"Wins\",\n    \"0-1\": \"The auction request from the exchange, normalized to OpenRTB fields.\",\n    \"1-1\": \"The bids returned from the [Bidding Agent](doc:bidding-agent) to the exchange, whether the auction was won or not.\",\n    \"4-1\": \"The winning auctions (impressions), clicks, and events (video plays, etc)\",\n    \"0-2\": \"[auction_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/auction_log_headers.csv)\",\n    \"1-2\": \"[bid_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/bid_log_headers.csv)\",\n    \"4-2\": \"[win_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/win_log_headers.csv)\",\n    \"h-3\": \"Column Definitions and Protobuf Mapping\",\n    \"4-3\": \"[ad_log.proto](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/ad_log.proto)\",\n    \"1-3\": \"\",\n    \"2-0\": \"Conversions\",\n    \"2-1\": \"The conversions recorded by Beeswax\",\n    \"2-2\": \"[conversion_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/conversion_log_headers.csv)\",\n    \"3-0\": \"Losses\",\n    \"3-1\": \"Loss logs provided by a limited number of exchanges (Google)\",\n    \"3-2\": \"[bid_response_feedback_logs.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/bid_response_feedback_logs.csv)\"\n  },\n  \"cols\": 4,\n  \"rows\": 5\n}\n[/block]","excerpt":"","slug":"data-feeds","type":"basic","title":"Data Feeds"}
One of the most important considerations in working with an RTB bidder is designing the data pipeline by which auction, bid, and win events are processed and analyzed. Unlike DSPs, with the Beeswax Bidder-as-a-Service technology you can access the full, unfiltered stream of RTB events, in much the same way you would if you were building a bidder from scratch. [block:api-header] { "type": "basic", "title": "Streaming vs Batch" } [/block] The first consideration for designing your data pipeline is whether you prefer to get the data in batch form, or as a continuous stream. For the very high-volume data, like auction logs, Beeswax only supports batch data. For win logs (impressions) Beeswax supports both methods, and there are pros and cons to each: [block:parameters] { "data": { "0-0": "Batch", "1-0": "Stream", "h-0": "Pipeline Method", "h-1": "Description", "h-2": "Pros", "h-3": "Cons", "0-1": "Hourly or daily files of data placed in an S3 bucket.", "1-1": "Near real-time data in [protobuf](https://developers.google.com/protocol-buffers/) format sent over http or to AWS [Kinesis](https://aws.amazon.com/kinesis/)", "0-2": "Fairly easy to ingest, fault tolerant", "0-3": "Delay in utilizing data. Also may include many files written per hour.", "1-2": "Use data as fast as you can process it", "1-3": "Higher cost and complexity to support data ingestion" }, "cols": 4, "rows": 2 } [/block] [block:api-header] { "type": "basic", "title": "Data Definitions" } [/block] Column definitions, protobuf mapping, and field lists can be found on [Google Docs](https://docs.google.com/spreadsheets/d/1cIMgCNyAech19JUqS5mMKQ_UkJ-tOxVCY85BXiSnWNQ/edit?usp=sharing). A complete **data dictionary** can be found in this [Google Doc](https://docs.google.com/spreadsheets/d/1cIMgCNyAech19JUqS5mMKQ_UkJ-tOxVCY85BXiSnWNQ/edit?usp=sharing). Beeswax makes three types of data available from Stinger as described in the chart below. Based on your use case you may need some, or all of this data. Because some of this data can be quite large, additional fees may apply (contact your Account Manager for more information). [block:parameters] { "data": { "h-0": "Data Type", "h-1": "Description", "h-2": "Batch Field Manifest", "0-0": "Auctions", "1-0": "Bids", "4-0": "Wins", "0-1": "The auction request from the exchange, normalized to OpenRTB fields.", "1-1": "The bids returned from the [Bidding Agent](doc:bidding-agent) to the exchange, whether the auction was won or not.", "4-1": "The winning auctions (impressions), clicks, and events (video plays, etc)", "0-2": "[auction_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/auction_log_headers.csv)", "1-2": "[bid_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/bid_log_headers.csv)", "4-2": "[win_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/win_log_headers.csv)", "h-3": "Column Definitions and Protobuf Mapping", "4-3": "[ad_log.proto](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/ad_log.proto)", "1-3": "", "2-0": "Conversions", "2-1": "The conversions recorded by Beeswax", "2-2": "[conversion_log_headers.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/conversion_log_headers.csv)", "3-0": "Losses", "3-1": "Loss logs provided by a limited number of exchanges (Google)", "3-2": "[bid_response_feedback_logs.csv](https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/logs/batch/bid_response_feedback_logs.csv)" }, "cols": 4, "rows": 5 } [/block]