When sharing data to file storage solutions, Funnel can provide a summary file containing information about the shared data.

This can be used to:

  • Trigger the next stages in your data pipeline

  • Determine which files have been updated

  • Handle errors for failed runs

The sharing summary file can be configured for the following sharing destinations:

  • Google Cloud storage

  • Amazon S3

  • Microsoft Azure Blob storage

  • SFTP

The summary file can be configured in the 'advanced' settings and defaults to the JSON format for new Shares.


This example comes from a Google Cloud storage Share:

The summary file formats that are available are:

  • Text file (legacy)
    This plain text file can be used to trigger the next steps in a data pipeline but is considered legacy as it contains no information about the Share run, and is included for backward compatibility with older Shares.

  • CSV file
    Contains a list of the files that have been shared and if they were updated or not since the last run.

  • JSON file
    Contains meta-information about the Share run, such as status (Success or Failed), error messages, unique IDs, start and end dates, file names, and file sizes of what was shared and if data was updated or not since the last run. 

  • No summary file
    You can choose to receive no summary file at all.

For sharing summaries with BigQuery Shares, see:

https://help.funnel.io/en/articles/5137040-bigquery-summary-table-export

Did this answer your question?