Datapipe Weekly #24

Catching bad API data and an Arab proverb

Hello friend! This is a newsletter for builders.

What do you like to build?

I hope the ideas in this week’s newsletter can help you get it done.

In this weeks newsletter

  • 💻 Builders: Catching bad API data with Python

  • 📜 Philosophers: Quote of the week

Catching bad API data with Python

I have a script that pulls reporting data from the Facebook Marketing API. It runs daily jobs on my airflow cluster and updates the production data sources.

Since we want daily granularity for reporting, each record needs to have metrics which correspond only to one day. By default though, Facebook will return metrics over the entire date range of your request. Making one API call per day is not a good solution, since it’s more prone to error and takes longer to run.

The solution is to set the request parameter time_increment=1, i.e. return 1 day of data (views, clicks, spend, etc..) per record.

In the API response, this period can actually be verified because the record includes a start and end date. In my code, I verify these dates are the same in order to help ensure that my assumptions are correct.

When iterating over the response records, I do a check like this:

for r in insights_data:
    # Each field should span only a single day
    if r.get('date_start', '') != r.get('date_stop', ''):
        raise UnexpectedRecordValueError(
            'Fields date_start ({}) and date_end ({}) in API '
            'response field must match.'.format(
                record.get('date_start', ''),
                record.get('date_stop', ''),

I raise an exception which is easily understood by it’s name. This can be defined in a few lines:

class UnexpectedRecordValueError(ValueError):
    """Record has unexpected value."""

Quote of the week

“Everything that happens once can never happen again, but everything that happens twice will surely happen a third time.”
- Paulo Coelho, The Alchemist, Arab Proverb

The first two people I quoted this to said the same thing - that it’s illogical.

An understandable initial reaction. But you must bend your mind to see through that -

Over the full timespan of the universe it’s completely illogical, in fact, that anything which happens once would happen again. This is like saying that 1 = n where n >= 2.

Besides, it’s the other half of the proverb that caught my attention. What’s something that happened to you recently for a 2nd time…


Thank you for reading Datapipe 👋

Subscribe the Datapipe weekly newsletter ⬇️