Lucas FonsecainDev GeniusEmpowering Data Analysis with PySpark: Benefits and Usage GuideIn this article I will cover a little more about PySpark, how to install and how to run your first code, I hope you like it, and don’t…May 2May 2
Lucas FonsecainDev GeniusEnsuring Data Integrity in Kafka with Python: Strategies for Connection LossIn the world of data streaming, Kafka has emerged as a prominent platform for handling large-scale real-time data feeds efficiently…Feb 19Feb 19
Lucas FonsecainDev GeniusStreamlining Data Pipelines: Integrating Kafka with AirflowIn this article I want to show a simple way to use Apache Kafka together with Airflow, even though we know that there are better frameworks…Dec 15, 20232Dec 15, 20232
Lucas FonsecainDev GeniusExploring Airflow for Data Pipelines: A Dive into the Airflow PoolIn this article, we will cover a little more about Airflow’s Pool feature, a valuable tool for managing shared resources and controlling…Oct 28, 2023Oct 28, 2023
Lucas FonsecainDev GeniusBalancing Data Integration: ETL vs. ELT — A Comprehensive GuideWith the increasing dissemination of the Data Engineering area, we often find ourselves in some scenarios where we need to decide which…Oct 22, 2023Oct 22, 2023
Lucas FonsecainDev GeniusPySpark in Apache Airflow: A Comprehensive Guide with Code ExamplesIn this article I want to cover a little more about the use of Pyspark with Airflow, I hope you like it! 😍Sep 24, 2023Sep 24, 2023
Lucas FonsecainDev GeniusCelery executor vs Kubernetes executor on AirflowAs we mentioned in previous articles, Airflow is an open-source platform that allows us to create, schedule and monitor workflows. And it…Mar 8, 20231Mar 8, 20231
Lucas FonsecainDev GeniusCelery Executor on AirflowContinuing our series of articles talking about Airflow, in this one, we will cover a little more about Celery executor, I hope you like…Feb 26, 2023Feb 26, 2023
Lucas FonsecainDev GeniusLet’s talk about FlinkIn this article, I want to present a little more about Flink, one of the tools that I have been increasingly passionate about working with…Feb 24, 2023Feb 24, 2023
Lucas FonsecainDev GeniusKubernetes Executor on AirflowIn this new series, we’re going to talk a little more about Airflow, its benefits, and use cases. In this first article, I will discuss…Feb 22, 2023Feb 22, 2023
