Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (2023)

Airflow is a community-built platform for programmatic creation, planning and process monitoring. It is essentially a more capable version of cron, as it can not only schedule tasks, but also run processes in parallel, manage/monitor individual tasks, and communicate with other platforms/tools such as Google Cloud and StatsD.


It is rapidly gaining ground in data engineering and ETL workflow coordination. Basically, it helps automate scripts to complete tasks. Docker is a containerization technology that packages your application and all its dependencies into a docker container so that your program runs smoothly in any environment.

Running Airflow on Docker is much easier than running on Windows without Docker. This is because Docker saves the time needed to install the necessary dependencies to run data pipelines.

In this how-to article, you can understand the process of running Airflow on Docker with a detailed explanation. Before diving into the process, you need to understand Airflow and Docker separately.


  • What is Docker?
    • Main advantages of Docker
  • What is Apache Airflow?
    • How is it different from Cron?
  • Start Airflow in Docker
    • DAG driven acyclic graph
    • Operators
    • Docker configuration
    • Create your DAY
  • Conclusion

What is Docker?

Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (1)

Docker is a popular open source platform that allows running software programs in a portable and unified environment. Docker uses containers to create separate user space environments that share files and system resources at the operating system level. Containerization uses a fraction of the resources of a typical server or virtual machine.

(Video) Running Airflow 2.0 with Docker in 5 mins

Main advantages of Docker

  • Portability:With Docker, you can ensure that the functionality of your applications can run in any environment. This advantage comes from the fact that all programs and their dependencies are stored in the Docker runtime container.
  • Fast implementation: Docker can reduce setup time to seconds. This is because it creates a container for each process and does not start the operating system.
  • Scalability: Docker scales faster and more reliably than virtual machines (as well as traditional servers, which lack scalability of any kind). Docker's scalability is critical if you're a business that wants to serve tens or hundreds of thousands of users with your applications.
  • Insulation:All the supporting software required by your application is also located in the Docker container that hosts one of your applications. It won't be a problem if other Docker containers contain applications that require different versions of the same supporting software, because Docker containers are completely self-contained.
  • Performance:Containers allow for more efficient allocation of the limited resources of the main server. Indirectly, this translates into better performance for containerized programs, especially as server demand grows and resource allocation optimization becomes more important.

Simplify data analysis with Hevo's code-free data pipeline

fully managedThere is no platform for pipelined code cubes like Hevohelps you integrate data fromMore than 100data sources(including over 40 free data sources) in real time and effortlessly to the destination of your choice. With its minimal learning curve, Hevo can be set up in just minutes, allowing users to load data without sacrificing performance. Tight integration with countless sources gives users the flexibility to seamlessly enter different types of data without having to code a single line.


Check out some of Hevo's cool features:

  • Fully automated:The Hevo platform can be set up in just a few minutes and requires minimal maintenance.
  • Real-time data transfer:Hevo enables real-time data migration, so you always have data ready for analysis.
  • Transformations: Hevo provides preloading transformations via Python code. It also allows you to run transformation code on any event in the configured data pipelines.
  • connectors: Hevo supports over 100 font integrations for SaaS platforms, files, databases, analytics and BI tools. Supports multiple targets including Amazon Redshift, Firebolt, Snowflake Data Warehouses; Databricks, Amazon S3 Data Lakes, SQL Server, TokuDB, DynamoDB are just some of them.
  • 100% complete and accurate data transfer:Hevo's robust infrastructure ensures reliable data transmission without data loss.
  • Scalable infrastructure:Hevo has built-in integrations for more than 100 sources, allowing you to scale your data infrastructure as needed.
  • Live support 24 hours a day, 7 days a week:The Hevo team is available 24 hours a day to provide you with exceptional support via live chat, email and support calls.
  • Schedule management:Hevo eliminates the tedious task of schedule management and automatically detects the schedule of incoming data and assigns it to the target schedule.
  • Live tracking:With Hevo, you can track the flow of data, so you can see where your data is at all times.

Simplify your data analysis with Heva today!


What is Apache Airflow?

Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (2)

Apache Airflow is a data pipeline management system developed by Airbnb. In 2014, Airbnb released it as an open source project to help manage the company's aggregate data pipelines.

(Video) Airflow Tutorial for Beginners - Full Course in 2 Hours 2022

Since then, its popularity has grown as one of the most widely used open source tools for managing data engineering workflows. Since Apache Airflow is developed in Python, it has a lot of flexibility and reliability. Workflow management tasks such as job tracking and platform configuration are made easier thanks to an intuitive and powerful user interface. Users can write any code they want to run at any stage of the process because it relies on code to build workflows.

Airflow can be used for virtually any bulk data pipeline and there are numerous documented use cases, the most popular being Big Data projects. In the Airflow Github repository, some of the most common use cases are:

  • Create a Data Studio dashboard with Airflow and Google BigQuery.
  • Airflow is used to help build and manage a data lake on AWS.
  • Airflow is used to improve production while reducing downtime.

How is it different from Cron?

Due to several factors, Apache Airflow has replaced Cron:

  • Building relationships between tasks in cron is a difficult task, but it's as easy as writing Python code in Airflow.
  • Cron needs external help to record, monitor and resolve tasks. Airflow provides a user interface for monitoring and monitoring workflow execution.
  • Cron jobs cannot be repeated unless specifically stated. Airflow tracks all completed tasks.
  • Another difference is that Airflow is easily extensible while Cron is not.

Start Airflow in Docker

Running Airflow on Docker requires some knowledge of Airflow concepts to complete this tutorial. Here are some key terms used in Airflow.

  • DAG driven acyclic graph
  • Operators
  • Docker configuration
  • Create your DAY

DAG driven acyclic graph

Workflow is represented by targeted acyclic graphs, which are essentially tasks and their dependencies to be executed. Tasks are represented by vertices, while dependencies are represented by edges. The reason it's called acyclic is because you have to break the workflow. Airflow has a Python class to create DAGs, you just need to instantiate an object from airflow.models.dag.DAG.


You've seen the DAGs so far that show the workflow for running Airflow on Docker. And assignments? This is where operators come to the rescue, so basically operators define the tasks to be performed. There is a wide variety of mounts available through Airflow including

  • Python operator
  • e-mail operator
  • Jdbc-operator
  • oracle operator

Airflow has a custom operator if you need one so you can easily create, schedule and monitor those jobs.

Now that you know the basics of Airflow and you can start using Airflow on Docker.

(Video) Airflow DockerOperator: The Basics (and more 🤫)

Docker configuration

Installing Docker must be done carefully to run Airflow on Docker. Firstdocker composestallDock workereDocker composition. In this article, you use the puckel/docker-airflow repository for automated Docker builds. Once you've automated the Docker build, it's easier to run Airflow on Docker. If you want more information about this repository, please visithunchbacked. So you will use this ready container to run Airflow on Docker DAGs. To get this Docker image, you need to run the following command:

docker pull puckel/docker-luchtstroom
Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (3)

Since you used a puckel-ready container, you don't need to create a docker build file yourself. Basically, Docker Compose helps you run multiple containers and you need a YAML file to configure your application services with Docker Compose to run Airflow on Docker. in this case for exampledocker-compose-CeleryExecutor.ymlThe file contains settings for the web server, scheduler, worker, etc. Now you can start the container with the following command:

docker run -d -p 8080:8080 puckel/docker-luchtstroomservidorweb

By default, Puckel uses SequentialExecutor if you do not specify an executor type. You must use other build files for other executables, for example:

docker-compose -f docker-compose-CeleryExecutor.yml gore -d

You can also run Docker Compose with some sample DAGs:

docker run -d -p 8080:8080 -e LOAD_EX=y puckel/docker-luchtstroom

Once the container is up and running, Airflow will run on your localhost. You can check this by simply going to http://localhost:8080/admin/:

Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (4)

Create your DAY

You've already seen examples of DAGs on your localhost. DAGs are very important for running Airflow on Docker. A DAG must be defined in a Python file that contains several components, including the DAG definition, operators, and their relationships. After you create the file, you need to add it to the DAG folder in the Airflow folder. If you can't find the DAG map, you can report itair flow.cfg file in the AirflowHome folder. You can create a simple DAN that schedules a task to print sample text every day at 8:00 AM.

van datetime import datetimefrom luchtstroom import DAGfrom airflow.operators.python_operator import PythonOperatordef print_firstdag(): return 'Meu primeiro DAG de HevoData!'dag = DAG('first_dag', description='HevoData Dag', Schedule_interval='0 8 * * * ', start_date=datetime(2022, 2, 24), catchup=False)print_operator = PythonOperator(task_id='first_task', python_callable=print_firstdag, dag=dag)print_operator

Once you've created a DAG, you just need to run it in a Docker container. For that you need to connect your local machine to the container. You specify the path of the DAGs in your directory. For simplicity, write the default path home/user/airflow/dags to run Airflow on Docker.

docker run -d -p 8080:8080 -v /home/user/airflow/dags:/usr/local/airflow/dags puckel/docker-airflow web poslužitelj

Now you have connected your container and local machine, but you still don't know the name of the container to run. Don't worry, it's just a simple command:

(Video) Run Airflow 2.0 via Docker | Minimal Setup | Apache airflow for beginners

ps port worker

Once you have the name, replace it in the command below:

doker execute-tistriking

Basically, the command line is run in your docker container:

Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (5)

As you can see, by default it is created in a paused state and you activate it so that it can be started. This time you will use the UI because it is more convenient to run Airflow on Docker. (shown by an arrow):

Getting Started with Airflow on Docker Simplified 101 Guide - Learn | hello (6)

That's all you programmed. You can also run it by activating it from the user interface. This completes the steps involved in running Airflow on Docker seamlessly.


In short, all you can say is that running Airflow on Docker frees you from the burden of managing, maintaining and deploying all Airflow dependencies. To run Airflow on Docker, you need to download Docker and Docker Compose and run your container. Then you can create your own DAY and schedule or activate tasks. Now you can create your own DAGs and run them on Docker.

Extracting complex data from different data sources to perform in-depth analysis can be challenging, and it is trueHevoSave the day!Hevoprovides a faster way to move data fromMore than 100 data sourcesincluding databases or SaaS applications to a destination of your choice or a data warehouse for visualization in a BI tool. Hevo is fully automated and therefore requires no coding.


Dice konj,code-free data pipeline provides a consistent and reliable solution for managing the transfer of data between different sources and a wide range of desired destinations, with just a few clicks. Hevo Data with its tight integration withMore than 100 fonts(including 40+ free fonts) not only can you export data from your desired data sources and load it to the destination of your choice, but also transform and enrich your data to be ready for analysis so you can focus on your core needs and execute analysis criteria using BI tools.

(Video) What Is Docker - Docker Intro And Tutorial On Setting Up Airflow

Would you like to go for a ride with Heva?To registerforFree 14-day trial periodand experience the versatile Hevo package first hand. You can also check out unbeatable prices to help you choose the right plan for your business needs.


1. Airflow introduction and installation: Airflow Tutorial P1
2. Airflow Docker: run Airflow 2.0 in docker container
3. Airflow 101: Essential Tips For Beginners
4. Learn Apache Airflow with Python in 1 hour | Apache Airflow 101 | Apache Airflow Zero to Hero
5. Airflow 101: Essential Tips For Beginners
6. Dockerfile >Docker Image > Docker Container | Beginners Hands-On | Step by Step
(Automation Step by Step)


Top Articles
Latest Posts
Article information

Author: Kelle Weber

Last Updated: 22/10/2023

Views: 6730

Rating: 4.2 / 5 (53 voted)

Reviews: 84% of readers found this page helpful

Author information

Name: Kelle Weber

Birthday: 2000-08-05

Address: 6796 Juan Square, Markfort, MN 58988

Phone: +8215934114615

Job: Hospitality Director

Hobby: tabletop games, Foreign language learning, Leather crafting, Horseback riding, Swimming, Knapping, Handball

Introduction: My name is Kelle Weber, I am a magnificent, enchanting, fair, joyous, light, determined, joyous person who loves writing and wants to share my knowledge and understanding with you.