This article will highlight some of the most common errors occurring in Custom Connector sources and how to troubleshoot them. For a high level introduction of what Custom Connectors and their capabilities are, please refer to this article.
A Custom Connector Data Source receives data in one of two ways; "push" or "pull".
Push means that either the user or the platform sends data to Funnel from somewhere outside of Funnel, usually in a CSV/XLSX file.
Pull means that our team of Data Integration Engineers automates an extraction of data from a platform or system, typically via an API or a database.
Within these two concepts there are several methods of updating data in Funnel, each with their own characteristics to keep in mind when troubleshooting or investigating a data mismatch.
Scheduled exports (push)
The below methods both utilise the push method, where files get sent to a data source in Funnel, either in CSV or XLSX format. The data source can be configured to import either file-formats, but never both at the same time. Sending a file of the other format to a source will break the import.
SFTP (Funnel internal)
Funnel imports data using a date field as a key.
We strongly recommend reading this article on how Funnel treats date ranges when importing files.
Funnel only imports the contents of the file that is being sent.
If you are missing data for a specific date, the first step should be to check if a file containing the data for that day has been sent to the source.
If the data for a specific date is incorrect, it's good practice to double-check if the file sent matches what you see in the Data Explorer. You can always overwrite incorrect data with a new send from the platform or system.
You can send data to a Data Source at any time.
This includes patching missing date ranges and updating historical data as well as correcting en incorrect sending.
Overwriting or 'deleting' data in a Data Source.
While neither the Data Integration Engineering team or you can delete the data in a source completely, previously imported data can be overwritten by a new file. More info on how to go about that can be found here.
Each Data Source can consistently ingest files up to 500 Mb in size.
For cases where the file is larger than 500 Mb, we recommend splitting the data by day. If any of the daily files are still too large, we are happy to advise on the best solution forward.
My Data Source is not updating - how do I fix it?
In the event of an erroneous or unexpected file being sent to a Data Source, it will stop working until someone from the Data Integration Engineering team either clears out the file or reconfigures the Data Source to accept the new file format and data structure.
You may see an error message in the Data source details page with information about the issue and what action is needed. Users are typically prompted to reach out to Funnel support or to submit a new Data request in order to get the issue resolved.
Headers in report file has been changed.
If one or more expected headers have been omitted from a report file, the file will not be able to ingest and the import will break.
On the other hand, if you add in more headers in a report file than the data source has been configured to import, the extra fields will be omitted from the import completely.
If changes needs to be made to an existing setup, please submit a Data Request.
IMPORTANT: You cannot fix a broken source by sending the correct file to it, a member of our Support team or a Data Integration Engineer needs to manually get it back to normal working condition for it to import data as expected.
If you suspect an erroneous file has been sent to a source, please reach out to Funnel support and we will happily help you!
Automated data pulls
Integrations using the below methods are all set up and maintained by the Data Integration Engineering team.
All changes or issues with a source utilising these methods needs to go through the Data Integration Engineers. If you want to report a data mismatch or other issues with the integration, please reach out to Funnel support. If changes to the integration are needed, please submit a Data Request.
*While sources using external SFTPs technically are using the pull method to import data, the source imports files, so the push tips and tricks may be applicable in these cases as well.