When sharing data to file storage solutions, Funnel can provide a summary file containing information about the shared data.
Trigger the next stages in your data pipeline
Determine which files have been updated
Handle errors for failed runs
Copy the data in all of the files from S3 into an AWS service
The summary json file has a list of files that were sent for each export run. Each file object has status, error message, stats and updated. Only files with
updated=true contains changes and needs to be reloaded in your pipe.
The sharing summary file can be configured for the following sharing destinations:
Google Cloud storage
Microsoft Azure Blob storage
(separate summary tables are available for Biguery and Snowflake)
The summary file can be configured in the 'advanced' settings and defaults to the JSON format for new Shares.
It is also possible to use a custom file name for the Summary file, which can contain the variables 'runId', 'runDate', and 'runHour' to separate summary data from different runs.
This example comes from a Google Cloud storage Share:
The general summary file formats available are:
Contains a list of the files that have been shared and if they were updated or not since the last run.
Contains meta-information about the Share run, such as status (Success or Failed), error messages, unique IDs, start and end dates, file names, and file sizes of what was shared and if data was updated or not since the last run.
No summary file
You can choose to receive no summary file at all.
The extra summary file formats that can be chosen when exporting to S3:
Includes a JSON file containing all the uploaded files (not only the ones included in the last export) in the AWS Redshift format.
Includes a JSON file containing all the uploaded files (not only the ones included in the last export) and file configuration in the AWS QuickSight format.
Summary file format 'Text file' has been deprecated but will still be available in pre-existing shares.
For sharing summaries with BigQuery, see:
For sharing summaries with Snowflake, see: