SLIDE 1
Data Orchestration with Apache Airflow Data driven empower the - - PowerPoint PPT Presentation
Data Orchestration with Apache Airflow Data driven empower the - - PowerPoint PPT Presentation
Data Orchestration with Apache Airflow Data driven empower the organization to seek more understanding, through data analytics, about their business processes Apache Airflow Airflow is a platform to programmatically author, schedule
SLIDE 2
SLIDE 3
Data driven
“empower the organization to seek more understanding, through data analytics, about their business processes”
SLIDE 4
SLIDE 5
Apache Airflow
“Airflow is a platform to programmatically author, schedule and monitor workflows.”
SLIDE 6
SLIDE 7
SLIDE 8
SLIDE 9
SLIDE 10
SLIDE 11
SLIDE 12
SLIDE 13
SLIDE 14
SLIDE 15
SLIDE 16
SLIDE 17
SLIDE 18
Cloud Pub/Sub Cloud Dataflow
Batch Data Sources
Cloud Storage BigQuery Friendly Data Lake
External Data Sink
BigQuery Data Warehouse
Stream Data Sources
SLIDE 19
SLIDE 20
Cloud Composer
SLIDE 21
Important ETL principles
- Processes are idempotent and deterministic
SLIDE 22
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
SLIDE 23
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
- Data partitioning (usually by date)
SLIDE 24
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
- Data partitioning (usually by date)
- Dealing with alerts and SLA’s
SLIDE 25
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
- Data partitioning (usually by date)
- Dealing with alerts and SLA’s
- Globally consistent paths to data
SLIDE 26
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
- Data partitioning (usually by date)
- Dealing with alerts and SLA’s
- Globally consistent paths to data
- Data at rest is immutable
SLIDE 27
Important ETL principles
- Processes are idempotent and deterministic
- Reusable, parameterisable components
- Data partitioning (usually by date)
- Dealing with alerts and SLA’s
- Globally consistent paths to data
- Data at rest is immutable
“Data at rest to data at rest”
SLIDE 28
extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
SLIDE 29
extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
SLIDE 30
extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
SLIDE 31
extract_customer = PostgresToPostgresOperator( src_postgres_conn_id='postgres_oltp', dest_postgress_conn_id='postgres_dwh', sql='select_customer.sql', pg_table='staging.customer', parameters={"window_start_date":"{{ ds }}", "window_end_date":"{{ tomorrow_ds }}"}, pool='postgres_dwh')
SLIDE 32
SLIDE 33
https://github.com/gtoonstra/airflow-deployments https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home https://airflow.incubator.apache.org/ https://gtoonstra.github.io/etl-with-airflow/
References
https://medium.freecodecamp.org/the-rise-of-the-data-engineer-91be18f1e603 https://medium.com/@maximebeauchemin/the-downfall-of-the-data-engineer-5bfb701e5d6b
SLIDE 34