Skip to main content
All CollectionsData Warehouse Sharing
When will new data be sent to your Data Warehouse
When will new data be sent to your Data Warehouse
Niclas Bångman avatar
Written by Niclas Bångman
Updated over 5 months ago

If your Data Share uses the option "Only share periods with updated data" disabled which for most cases is a legacy option this article does not apply because everything will re-export every time.

This article only describes the behaviour of Data Shares to our Data Warehouse destinations.

Overview

The Data Share will by default automatically look for all types of changes in the data and in Funnel that might affect it and re-export only when necessary. This is designed to be as efficient as possible to avoid unnecessarily sending the same data multiple time.

The schedule of the Data Share decides when we will look for new data and maybe trigger a re-export. If any of the criteria below are met then an job run will start, otherwise it will be marked as a success and we will write that no changes were found.

The Data Share will only react to changes of data within the selected Date range of the Data Share. If you have a moving window such as "Last 2 years" and you add data further than that you will not automatically receive it in you Data Warehouse.

Changes that will trigger a new re-export of a period

  1. Adding or editing a Data Source (where we typically collect 2 years of historical data)

    1. If the Data Share contains a field that could contain data from that Data Source and has downloaded data within the date range of the Data Share.

    2. If the Data Share is not actively excluding data from that Data Source or Data Source type with a filter.

  2. A Data Source has downloaded updated data for the interval

    1. Same conditions as #1

  3. Changes in dimension/metric rules within Funnel for a field used in the Data Share

  4. The same part failed it's last job run

  5. Manually updating configuration in the Data Share that affects the data sent

  6. Manually updated exchange rates in Funnel

  7. Changing workspace locale and including a non-ISO date dimension in the Data Share

  8. Manually clicking "Force run all parts" for a Data Share

There are also other conditions such as that we re-export the current period (day or month) at least once a day and all other data within the Data Share interval once a month.

When a change is detected we will process the data and compare it against another layer of cache and if it has not been changed it will also not be re-uploaded (unless you are sharing the data to a new location or manually clicked "Force run all parts").

All this logic means that you can trust that the files/tables that have been updated in your warehouse has all the updated rows. You therefore don't have to waste money and compute on processing identical files. We also write what files were updated in a Summary file or table which can be used to only trigger parts of your pipe, see the best practices around that here.

Did this answer your question?