Datapipe Weekly #23

JSONL + Surrendering to the will of the world

Hello friend! This is a newsletter for builders and philosophers.

What do you like to build?

I hope the ideas in this week’s newsletter can help you get it done.

In this weeks newsletter

  • 💻 Builders: Using JSON Lines in your Workflow

  • 📜 Philosophers: Quote of the week


Tip: Using JSON Lines in your Workflow

This week I’m dusting off an old blog post that does pretty well on Google search now. It was inspired by a filestore data type I read about in the book Agile Data Science 2.0 - JSON Lines, suffixed with the file extension jsonl.

Check out the full post here:

https://galea.medium.com/how-to-love-jsonl-using-json-line-format-in-your-workflow-b6884f65175b

I pulled out my jsonl read/write functions recently when building a scraper. It takes in a list of URLs and hits them with HTTP requests, then pulls out some data from the response page.

Since HTTP requests are valuable, I don’t want to lose any data along the way, so each time I parsed a response I appended the record onto my JSON Lines file on disk. The pseudocode goes something like this:

for url in urls:
    resp = requests.get(url)
    data = parse_resp(resp)
    with open("filestore.jsonl", "a+") as f:
        f.write(json.dumps(data, ensure_ascii=False)+"\n") 

This results in an on-disk filestore (as opposed to in memory) which can act as reliable source of truth.


Quote of the week

"Aligning one's will with the natural forces unfolding around us leads to some surprisingly powerful results"
- Michael A. Singer, The Surrender Experiment

If you look at life as a river descending a costal mountain, you can imagine the path from the top down to the point where the river meets the sea.

The water flows unperturbed by rocks.

Something unexpected? No worries. The water doesn’t fight its inevitable dissolution into the ocean and the world beyond it.

I think we can learn a lot from nature. Everything is designed to follow the path of least resistance. Or in physics terms - only energetically favorable things occur (except in the small time & energy scales of quantum mechanics).

That is to say - the water never flows up the mountain. Water never flows over the rock when it can go around.

To a large extent we shape our future by the way we act and the decisions we make. But I think most of us overestimate our ability to shape our path in the world. Rather - we must focus on the detailed things.

Breath. No really. Breath deeply and focus on that alone. Then put attention where it matters now.

Work hard. No really. Seek out large chunks of deep work in your day.

Dream big. No really. What do you want out of life?

From there? Well.. maybe it is just best to surrender to that which is beyond you control. Surely you weren’t in control when you entered this world and not when you leave it either. So why should we think the in between is so much different.

-Alex


Thank you for reading Datapipe 👋


Subscribe the Datapipe weekly newsletter ⬇️