When sharing data to file storage solutions, Funnel can provide a summary file containing information about the shared data.

This can be used to:

  • Trigger the next stages in your data pipeline

  • Determine which files have been updated

  • Handle errors for failed runs

The sharing summary file can be configured for the following sharing destinations:

  • Google Cloud storage

  • Amazon S3

  • Microsoft Azure Blob storage

  • SFTP

The summary file can be configured in the 'advanced' settings and defaults to the JSON format for new Shares.

It is also possible to use a custom file name for the Summary file, which can contain the variables 'runId', 'runDate', and 'runHour' to separate summary data from different runs.

This example comes from a Google Cloud storage Share:

The summary file formats available are:

  • CSV file
    Contains a list of the files that have been shared and if they were updated or not since the last run.

  • JSON file
    Contains meta-information about the Share run, such as status (Success or Failed), error messages, unique IDs, start and end dates, file names, and file sizes of what was shared and if data was updated or not since the last run. 

  • No summary file
    You can choose to receive no summary file at all.

Summary file format 'Text file' has been deprecated but will still be available in pre-existing shares.

The (plain) text file format was previously used to trigger the subsequent steps in a data pipeline but was deprecated as it contained no information about the actual Share run, and has been included for backward compatibility with older Shares.

For sharing summaries with BigQuery Shares, see:


Did this answer your question?