Datapipe Weekly #14
Hello friend! This is a newsletter for builders.
What do you like to build?
I hope the ideas in this week’s newsletter can help you get it done.
In this weeks newsletter
💻 Tip: Opening remote CSVs with Python
📜 Quote of the week
Tip: Opening remote CSVs with Python
Today I needed to rearrange and add columns to a CSV file on an SFTP server.
I ended up using paramiko
and a convenient method called load_system_host_keys
, which handles the authentication using RSA keys (see this tutorial for more on RSA authentication).
SSH_HOST = "" | |
SSH_USER = "" | |
SSH_FILE_PATH = "" | |
COMPRESSION = "zip" | |
from paramiko import SSHClient | |
client = SSHClient() | |
client.load_system_host_keys() | |
client.connect(SSH_HOST, username=SSH_USER) | |
sftp_client = client.open_sftp() | |
f = sftp_client.open(SSH_FILE_PATH) | |
df_chunks = pd.read_csv(f, chunksize=100000, compression=COMPRESSION) | |
for i, df in enumerate(df_chunks): | |
display(df.head()) | |
print(i) |
I load a zipfile in chunks of size 100k, then iterate over the chunks. Of course, you can remove the chunksize
arg and just load the DataFrame itself instead of an iterable as done above.
Quote of the week
“The only real test of intelligence is if you get what you want out of life”
- Naval
Today I wanted to open a remote zipfile on my local machine with Python, and you know what? I got it.
But seriously - does Naval’s statement resonate with you?
Looking back, I realize that what I want from life is a moving target. For a long time I wanted to own things - like a big house and a nice car.
I never got these things - the house and car were long term goals, always out of reach. And now that I’m in a position to purchase them - they are no longer things I want. So maybe Naval was only half right. The other real test of intelligence is knowing what you want out of life.
-Alex
Thank you for reading Datapipe 👋
Subscribe the Datapipe weekly newsletter ⬇️