Exporting Large Results via API

There are use cases where exporting large result sets beyond the 30,000 row limit in the 2.0 version of the API is required. Unlike the 0.5 version of the API, pagination is not supported in the 2.0 version of the API. The workflow to receive all results differs from normal.

🚧

Notes on Heavy API Usage

Please note, that exporting large results frequently can be computationally intensive on our underlying systems. We kindly request to bulk requests as much as possible and be wary that data in our reporting API can only update hourly.

If you will be repeatedly pulling the same dataset multiple times, we recommend using our Log Delivery products instead of, or in addition to, the API. We also provide real-time log delivery via Kinesis or HTTP.

Scheduling or Sending a Large Result set

Reports exceeding 30,000 rows should not be run through the standard [/reporting/run-query] (ref:reportingrun-query) endpoint.

Instead, if you anticipate results larger than this you should use the /reporting/report-schedules/run-once for one-time queries and /reporting/report-schedules for recurring queries.

When using this endpoint an optional send_all_results boolean parameter can be included with the request. Setting this to True will overwrite the 30,000 row limit with respect to the following caveats below.

As an example, the following payload would allow running an unlimited results report (Saved Report ID 120) -- in this case to an S3 bucket:

{
    "crontab": "0 7 * * *",
    "saved_report_id": 120,
    "name": "Daily Report -- All Results",
    "scheduled_plan_destination": [
        {
            "format": "csv",
            "address": "s3://beeswax-tmp/daily_reports/",
            "type": "s3",
            "parameters": {"access_key_id": "access_key_here", "region": "us-east-1"},
            "secret_parameters": {"secret_access_key": "secret_key_here"}
        }

    ],
    "send_all_results": true,
    "timezone": "America/New_York"
}

Supported Destinations and Limitations

Large results must be scheduled to a destination, rather than returned in the responding HTTP request. The following destinations are supported, however there are some limitations depending on the destination.

• SMTP (Email) -- Attachments cannot exceed 10MB.
• Webhook -- No known size limitation from Beeswax, however your webhook receiving server may need to be configured to allow large payload sizes.
• S3 -- No known size limitation.
• SFTP -- No known size limitation.