Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (2023)

Airflow is a community-built platform for programmatic creation, planning and process monitoring. It is essentially a more capable version of cron, as it can not only schedule tasks, but also run processes in parallel, manage/monitor individual tasks, and communicate with other platforms/tools such as Google Cloud and StatsD.

It is rapidly gaining ground in data engineering and ETL workflow coordination. Basically, it helps automate scripts to complete tasks. Docker is a containerization technology that packages your application and all its dependencies into a docker container so that your program runs smoothly in any environment.

Running Airflow on Docker is much easier than running on Windows without Docker. This is because Docker saves the time needed to install the necessary dependencies to run data pipelines.

In this how-to article, you can understand the process of running Airflow on Docker with a detailed explanation. Before diving into the process, you need to understand Airflow and Docker separately.

Content

• What is Docker?
  • Main advantages of Docker
• What is Apache Airflow?
  • How is it different from Cron?
• Start Airflow in Docker
  • DAG driven acyclic graph
  • Operators
  • Docker configuration
  • Create your DAY
• Conclusion

What is Docker?

Docker is a popular open source platform that allows running software programs in a portable and unified environment. Docker uses containers to create separate user space environments that share files and system resources at the operating system level. Containerization uses a fraction of the resources of a typical server or virtual machine.

Main advantages of Docker

• Portability:With Docker, you can ensure that the functionality of your applications can run in any environment. This advantage comes from the fact that all programs and their dependencies are stored in the Docker runtime container.
• Fast implementation: Docker can reduce setup time to seconds. This is because it creates a container for each process and does not start the operating system.
• Scalability: Docker scales faster and more reliably than virtual machines (as well as traditional servers, which lack scalability of any kind). Docker's scalability is critical if you're a business that wants to serve tens or hundreds of thousands of users with your applications.
• Insulation:All the supporting software required by your application is also located in the Docker container that hosts one of your applications. It won't be a problem if other Docker containers contain applications that require different versions of the same supporting software, because Docker containers are completely self-contained.
• Performance:Containers allow for more efficient allocation of the limited resources of the main server. Indirectly, this translates into better performance for containerized programs, especially as server demand grows and resource allocation optimization becomes more important.

What is Apache Airflow?

Apache Airflow is a data pipeline management system developed by Airbnb. In 2014, Airbnb released it as an open source project to help manage the company's aggregate data pipelines.

Since then, its popularity has grown as one of the most widely used open source tools for managing data engineering workflows. Since Apache Airflow is developed in Python, it has a lot of flexibility and reliability. Workflow management tasks such as job tracking and platform configuration are made easier thanks to an intuitive and powerful user interface. Users can write any code they want to run at any stage of the process because it relies on code to build workflows.

Airflow can be used for virtually any bulk data pipeline and there are numerous documented use cases, the most popular being Big Data projects. In the Airflow Github repository, some of the most common use cases are:

• Create a Data Studio dashboard with Airflow and Google BigQuery.
• Airflow is used to help build and manage a data lake on AWS.
• Airflow is used to improve production while reducing downtime.

How is it different from Cron?

Due to several factors, Apache Airflow has replaced Cron:

• Building relationships between tasks in cron is a difficult task, but it's as easy as writing Python code in Airflow.
• Cron needs external help to record, monitor and resolve tasks. Airflow provides a user interface for monitoring and monitoring workflow execution.
• Cron jobs cannot be repeated unless specifically stated. Airflow tracks all completed tasks.
• Another difference is that Airflow is easily extensible while Cron is not.

Start Airflow in Docker

Running Airflow on Docker requires some knowledge of Airflow concepts to complete this tutorial. Here are some key terms used in Airflow.

• DAG driven acyclic graph
• Operators
• Docker configuration
• Create your DAY

DAG driven acyclic graph

Workflow is represented by targeted acyclic graphs, which are essentially tasks and their dependencies to be executed. Tasks are represented by vertices, while dependencies are represented by edges. The reason it's called acyclic is because you have to break the workflow. Airflow has a Python class to create DAGs, you just need to instantiate an object from airflow.models.dag.DAG.

Operators

You've seen the DAGs so far that show the workflow for running Airflow on Docker. And assignments? This is where operators come to the rescue, so basically operators define the tasks to be performed. There is a wide variety of mounts available through Airflow including

• Python operator
• e-mail operator
• Jdbc-operator
• oracle operator

Airflow has a custom operator if you need one so you can easily create, schedule and monitor those jobs.

Now that you know the basics of Airflow and you can start using Airflow on Docker.

Docker configuration

Installing Docker must be done carefully to run Airflow on Docker. Firstdocker composestallDock workereDocker composition. In this article, you use the puckel/docker-airflow repository for automated Docker builds. Once you've automated the Docker build, it's easier to run Airflow on Docker. If you want more information about this repository, please visithunchbacked. So you will use this ready container to run Airflow on Docker DAGs. To get this Docker image, you need to run the following command:

docker pull puckel/docker-luchtstroom

Since you used a puckel-ready container, you don't need to create a docker build file yourself. Basically, Docker Compose helps you run multiple containers and you need a YAML file to configure your application services with Docker Compose to run Airflow on Docker. in this case for exampledocker-compose-CeleryExecutor.ymlThe file contains settings for the web server, scheduler, worker, etc. Now you can start the container with the following command:

docker run -d -p 8080:8080 puckel/docker-luchtstroomservidorweb

By default, Puckel uses SequentialExecutor if you do not specify an executor type. You must use other build files for other executables, for example:

docker-compose -f docker-compose-CeleryExecutor.yml gore -d

You can also run Docker Compose with some sample DAGs:

docker run -d -p 8080:8080 -e LOAD_EX=y puckel/docker-luchtstroom

Once the container is up and running, Airflow will run on your localhost. You can check this by simply going to http://localhost:8080/admin/:

Create your DAY

You've already seen examples of DAGs on your localhost. DAGs are very important for running Airflow on Docker. A DAG must be defined in a Python file that contains several components, including the DAG definition, operators, and their relationships. After you create the file, you need to add it to the DAG folder in the Airflow folder. If you can't find the DAG map, you can report itair flow.cfg file in the AirflowHome folder. You can create a simple DAN that schedules a task to print sample text every day at 8:00 AM.

van datetime import datetimefrom luchtstroom import DAGfrom airflow.operators.python_operator import PythonOperatordef print_firstdag(): return 'Meu primeiro DAG de HevoData!'dag = DAG('first_dag', description='HevoData Dag', Schedule_interval='0 8 * * * ', start_date=datetime(2022, 2, 24), catchup=False)print_operator = PythonOperator(task_id='first_task', python_callable=print_firstdag, dag=dag)print_operator

Once you've created a DAG, you just need to run it in a Docker container. For that you need to connect your local machine to the container. You specify the path of the DAGs in your directory. For simplicity, write the default path home/user/airflow/dags to run Airflow on Docker.

docker run -d -p 8080:8080 -v /home/user/airflow/dags:/usr/local/airflow/dags puckel/docker-airflow web poslužitelj

Now you have connected your container and local machine, but you still don't know the name of the container to run. Don't worry, it's just a simple command:

ps port worker

Once you have the name, replace it in the command below:

doker execute-tistriking

Basically, the command line is run in your docker container:

As you can see, by default it is created in a paused state and you activate it so that it can be started. This time you will use the UI because it is more convenient to run Airflow on Docker. (shown by an arrow):

That's all you programmed. You can also run it by activating it from the user interface. This completes the steps involved in running Airflow on Docker seamlessly.

Conclusion

In short, all you can say is that running Airflow on Docker frees you from the burden of managing, maintaining and deploying all Airflow dependencies. To run Airflow on Docker, you need to download Docker and Docker Compose and run your container. Then you can create your own DAY and schedule or activate tasks. Now you can create your own DAGs and run them on Docker.

Extracting complex data from different data sources to perform in-depth analysis can be challenging, and it is trueHevoSave the day!Hevoprovides a faster way to move data fromMore than 100 data sourcesincluding databases or SaaS applications to a destination of your choice or a data warehouse for visualization in a BI tool. Hevo is fully automated and therefore requires no coding.

VISIT OUR WEBSITE TO DISCOVER HEVO

Dice konj,code-free data pipeline provides a consistent and reliable solution for managing the transfer of data between different sources and a wide range of desired destinations, with just a few clicks. Hevo Data with its tight integration withMore than 100 fonts(including 40+ free fonts) not only can you export data from your desired data sources and load it to the destination of your choice, but also transform and enrich your data to be ready for analysis so you can focus on your core needs and execute analysis criteria using BI tools.

Would you like to go for a ride with Heva?To registerforFree 14-day trial periodand experience the versatile Hevo package first hand. You can also check out unbeatable prices to help you choose the right plan for your business needs.