Datapipe Weekly #7
Data Transfer from AWS Redshift to BigQuery
Hello friend! This is a newsletter for builders.
What do you like to build?
I hope the ideas in this week’s newsletter can help you get it done.
In this weeks newsletter
✍️ Blog post: Data Transfer from AWS Redshift to BigQuery
💻 Trick: Pandas DataFrame transpose
📜 Quote of the week
Blog post: Data Transfer from AWS Redshift to BigQuery
Have you ever wanted to move 100 million rows of data from AWS to Google Cloud…
… every day?
I wrote about my strategy for synchronizing tables between Redshift and BigQuery using SQL and bash commands.
Check it out here:
https://galea.medium.com/data-transfer-from-aws-redshift-to-bigquery-4ed65a38a3e8
Trick: Pandas DataFrame transpose
The first time I saw a transpose may have been a first year linear algebra class in university.
A very simple concept (which physics of course had to make a bit harder with imaginary numbers). Examples of both are shown below.
But I digress. What brought this to mind today was when I used the pd.DataFrame.transpose function to inspect records easier.
Check it out here
Quote of the week
“If one asked permission there was a 50% chance it would be refused. If one just got on with it, 19 times out of 20 nothing was said.”
- Trevor Kletz
I try to approach things with a “shoot from the hip” mentality. Make a decision and take action - right away.
Maybe not the optimized solution, but probably it doesn’t matter.
Maybe not something I even should be doing
- but, as Kletz would say - better sometimes to ask for forgiveness rather than request permission.
-Alex
Thank you for reading Datapipe 👋
Subscribe the Datapipe weekly newsletter ⬇️