This newsletter provides 2-5 minutes of easily digestible content for people who code.
In this weeks newsletter
My code style philosophy
Building your coding portfolio
💻Blog post: Airflow Dynamic DAGs - Python Globals
Quote of the week
My code philosophy
Code should be beautiful.
Beautiful code is dependable,
maintainable,
and easy to learn.
Building your code portfolio
This advice is from The Pragmatic Programmer.
Invest regularly in your knowledge portfolio
Diversify what technologies you learn about
Manage risk - don’t put all you eggs in one technology
Buy low, sell high by learning emerging technologies early
Review and rebalance your skills
Blog post: Airflow Dynamic DAGs — Python Globals
In order to dynamically create DAGs with Airflow, we need two things to happen:
Run a function that instantiates an
airflow.DAG
object.Pass that object back to the global namespace of the DAGfile.
Here’s how you define a DAG in Airflow
from airflow import DAG
dag = DAG(
"my_dag",
default_args={"start_date": "2020-01-01"},
schedule_interval="0 0 * * *",
)
This could be done dynamically as follows:
def create_dag(name, cron_interval):
dag = DAG(
"my_dag_{}".format(name),
default_args={"start_date": "2020-01-01"},
schedule_interval=cron_interval,
)
return dag
dag = create_dag("alex", "24 7 */2 * *")
Here’s where it gets tricky.
We need to make sure we pass the DAG object instance to the global namesapce.
This is why we return it from the create_dag function.
However, when using loops we would fail to pass it back to the global namespace
for args in (
("daily", "0 0 * * *"),
("weekly", "0 0 * * 0"),
):
dag = create_dag(*args)
The fix is to use globals() e.g. like this:
for i, args in enumerate(
("daily", "0 0 * * *"),
("weekly", "0 0 * * 0"),
):
dag = create_dag(*args)
globals()[f"dag_{i}"] = dag
You can continue reading the full post at the link below:
https://medium.com/@galea/airflow-dynamic-dags-python-globals-4f40905d314a
Quote of the week
“What if the only things you had in your life tomorrow were the things you expressed gratitude for today?”
-Jim Kwik
Gratitude for something comes about naturally by imagining your life without the thing.
For example, I make sure to drink water through out the day. I don’t really enjoy doing this but I just do it to stay hydrated.
But.. what if the water was not in my life tomorrow? I would get thirsty obviously, have headaches, trouble focusing, etc..
What if I didn’t have access to clean water at all?
Thinking along those lines, it’s easy to enjoy the periodic glass of water I reach for. I don’t have to drink it. I get to drink it.
-Alex
Subscribe the Datapipe weekly newsletter ⬇️
Share this article ⬇️
Your water analogy is a helpful tip for our attitude on anything beneficial in life .. when we’ve come to see it as a “chore” ... taking away our ability to do it - helps to see it in a new light :)