When sharing data to file storage solutions, Funnel can provide a summary file containing information about the shared data.
This can be used to:
Trigger the next stages in your data pipeline
Determine which files have been updated
Handle errors for failed runs
The sharing summary file can be configured for the following sharing destinations:
Google Cloud storage
Microsoft Azure Blob storage
The summary file can be configured in the 'advanced' settings and defaults to the JSON format for new Shares.
It is also possible to use a custom file name for the Summary file, which can contain the variables 'runId', 'runDate', and 'runHour' to separate summary data from different runs.
This example comes from a Google Cloud storage Share:
The summary file formats available are:
Contains a list of the files that have been shared and if they were updated or not since the last run.
Contains meta-information about the Share run, such as status (Success or Failed), error messages, unique IDs, start and end dates, file names, and file sizes of what was shared and if data was updated or not since the last run.
No summary file
You can choose to receive no summary file at all.
Summary file format 'Text file' has been deprecated but will still be available in pre-existing shares.
For sharing summaries with BigQuery Shares, see: