Data Offload (MSP)
Data Offload of your customer's DNS requests in log file format is available in Web Protection using our Data Offload API set. These APIs allow you to download a customer's most recent traffic data (up to the previous 5 days) from an Amazon S3 bucket in Gzip format. Authentication is through the Web Protection API.
An MSP admin can follow the steps below to view the available logs for a single customer and download one or all of the available log files. The following are helpful to complete the steps:
- Access to your preferred API client, such as Postman. See https://www.postman.com/downloads
- Access to the Web Protection Data Offload API documentation for reference. See <Need to Add Link>.
- You will need the Customer ID Number for the customer whose data logs you want to retrieve. This can be found in Web Protection in Overview > All Customers > ID Number. If the ID Number column is not visible, select the columns icon and slide ID Number on.
Important: In Web Protection log files, DNS requests match the time zone specified during Web Protection setup. However, the timestamp used in a log file name is UTC. Therefore, if you are looking for a log file for a specific time, you must adjust that time to UTC and locate the log file with the UTC-adjusted time stamp.
Authentication for data offload is through the Web Protection API. If you are not already authenticated, complete the Web Protection API Authentication steps.
Tip: A Customer UUID and Customer ID Number are different. The Customer ID Number is in your Customer's table in Web Protection (see above) and the Customer UUID is retrieved via API, as described here.
- Use the Get a User API. See Get a User. <Need URL from Robin>
- Pass the Customer ID Number in the id parameter.
The Customer UUID is returned in uuid_str. For example, the Customer UUID below is 4d20e977-0ec4-434a-b21d-2c529fb1d062:
Data log files are retained for 5 days.
- Use the List all customer log files API. See List all customer log files.
Pass the Customer UUID in the prefix parameter.
Tip: To see logs for a specific date, you can append a date to the UUID in YYYY-MM-DD format.
For example, 0130ae8f-3712-4dad-9e1b-f2ad5ebc136c/2022-07-13
A list of the files stored for this customer is returned. For example, the filename below is filename-with-timestamp.extension.
You can download all customer log files or a single customer log file.
- Use the Download all log files for a specified customer and date (System/MSP Admin) API to download all log files for a customer. See Download all log files for a specified customer and date (System/MSP Admin).
- Pass the Customer UUID in the uuid parameter.
- Pass the specific date you want to download log files for in the date parameter in YYYY-MM-DD format.
- Use Download a single log file for a specified date (System/MSP Admin) to download a single log file for a customer. See Download a single log file for a specified customer and date (System/MSP Admin).
- Pass the Customer UUID in the uuid parameter.
- Pass the log file date in YYYY-MM-DD format in the date parameter.
- Pass the log file name of the single log file you want to download in the filename parameter.
Tip: In Web Protection log files, DNS requests match the time zone specified during Web Protection setup. However, the timestamp used in a log file name is UTC. Therefore, if you are looking for a log file for a specific time, you must adjust that time to UTC and locate the log file with the UTC-adjusted time stamp.
- When saving the log file to your local machine, ensure you save it with the .gz extension. For example, filename.gz.