Funnel integrates a large number of different source platforms, with varying characteristics and limitations. The biggest category of Funnel source platforms are "online advertising data", such as Facebook Ads and Google Ads, and the following will focus on that perspective. For these sources Funnel has opted to download data on a schedule and to store the data internally before serving it to Funnel customers. As Funnel keeps expanding into other areas different choices may be made for those integrations in the future. 

Online advertising data have some typical properties that are important to understand as they come with some important implications for how to design a service like Funnel.

E.g the final advertising cost for each day may not be settled until several days later due to the cost being calculated based on the number of orders, removal of fees for fraudulent clicks, volume discounts etc. Order attribution is also frequently attributed to the time of the last click, where a window of up to 30 days between the click and order is not uncommon. This means getting the data for "yesterday" is usually an early estimation, rather than the final numbers.
One example of this is how Google describes the expected data freshness for Google Ads data.

Rate limits (e.g requests per second) and quotas (requests per day) typically apply to the individual advertising account and/or user used for connecting the advertising account. In addition to this, rate limits and quotas apply to the application making the requests for data; Funnel, in this case. In order to retrieve the data for a single "report" a large number of API requests may be needed as there are limits to the number of columns and rows retrieved with each request.

The initial request will "order" a report, that becomes available a few seconds (best case), or (more commonly) minutes later.

While large platforms (Google, Facebook, ...) have excellent availability, the internet is always a source of intermittent problems, and smaller platforms may have a harder time providing a very high uptime.

Many smaller advertising platforms do not strictly version control their APIs or notify customers and third parties like Funnel when they perform breaking changes. Funnel spends considerable resources on monitoring and adjusting the connections to this type of sources.

Credentials for API access frequently "break". A lot of times Oauth or other tokens are invalidated based on time, or when the connecting user changes their password. While Funnel can notify customers of the need to reconnect, with large customers connecting hundreds, or thousands of sources at any given time there is a risk of not having current credentials. In addition, for large advertisers the people holding the updated credentials may be spread out over several departments or even time zones.

While some platforms offer very granular data, and a large set of breakdowns (age group, country, device ...) a lot of smaller platform do not, and common strategies include "encoding" these in other fields. An example of this is to include codes in campaign names, e.g "SE_FURNITURE_myrealcampaign" for a campaign running in Sweden and promoting furniture.

With the above characteristics in mind, Funnel has opted to download data from the source platforms on a schedule that is adapted to each platform. And to store the data in the Funnel AWS environment before serving or exporting the data to the final destinations.

For the above reasons we believe the foundation for Funnel will continue to be based on scheduled retrieval and storing data, while the future evolutions may also include more flexible options and hybrid solutions.