When sharing data to file storage solutions, Funnel can provide a summary file containing information about the shared data.

Use cases

Trigger the next stages in your data pipeline
Determine which files have been updated
Handle errors for failed runs
Copy the data in all of the files from S3 into an AWS service

The summary json file has a list of files that were sent for each export run. Each file object has status, error message, stats and updated. Only files with updated=true contains changes and needs to be reloaded in your pipe.

Destination types

The sharing summary file can be configured for the following sharing destinations:

Google Cloud storage
Amazon S3
Microsoft Azure Blob storage
SFTP

(separate summary tables are available for Biguery and Snowflake)

Summary formats

The summary file can be configured in the 'advanced' settings and defaults to the JSON format for new Shares.

It is also possible to use a custom file name for the Summary file, which can contain the variables 'runId', 'runDate', and 'runHour' to separate summary data from different runs.

This example comes from a Google Cloud storage Share:

The general summary file formats available are:

CSV file
Contains a list of the files that have been shared and if they were updated or not since the last run.
JSON file
Contains meta-information about the Share run, such as status (Success or Failed), error messages, unique IDs, start and end dates, file names, and file sizes of what was shared and if data was updated or not since the last run.
No summary file
You can choose to receive no summary file at all.

The extra summary file formats that can be chosen when exporting to S3:

Redshift manifest
Includes a JSON file containing all the uploaded files (not only the ones included in the last export) in the AWS Redshift format.
QuickSight manifest
Includes a JSON file containing all the uploaded files (not only the ones included in the last export) and file configuration in the AWS QuickSight format.

Summary file format 'Text file' has been deprecated but will still be available in pre-existing shares.

The (plain) text file format was previously used to trigger the subsequent steps in a data pipeline but was deprecated as it contained no information about the actual Share run, and has been included for backward compatibility with older Shares.

For sharing summaries with BigQuery, see:

https://help.funnel.io/en/articles/5137040-bigquery-summary-table-export

For sharing summaries with Snowflake, see:

https://help.funnel.io/en/articles/6929929-snowflake-summary-table