Problem connecting to container airflow docker daemon (2023)

Table of Contents
Provedores Apache Airflow Apache Airflow Provider Versions Apache Airflow version work system Implantation Implementation details What happened What did you expect? how to reproduce One more thing Ready to send PR? Code of conduct FAQs Videos

Provedores Apache Airflow

stevedore

Apache Airflow Provider Versions

apache-airflow-providers-amazon==2.3.0
apache-airflow-providers-ipo==2.1.0
apache-airflow-providers-cncf-kubernetes==2.0.3
apache-airflow-providers-docker==2.2.0
apache-airflow-proveedores-elasticsearch==2.0.3
apache-airflow-proveedores-ftp==2.0.1
apache-airflow-providers-google==6.0.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-anbieter-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-proveedores-microsoft-azure==3.2.0
apache-airflow-proveedores-mysql==2.1.1
apache-airflow-proveedores-odbc==2.0.1
apache-airflow-proveedores-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-proveedores-sftp==2.1.1
apache-airflow-providers-slack==4.1.0
apache-airflow-proveedores-sqlite==2.0.1
apache-airflow-proveedores-ssh==2.2.0

Apache Airflow version

2.2.1 (latest version)

work system

PRETTY_NAME="Debian GNU/Linux 10 (Buster)"

(Video) Airflow Docker Set Up Guide

Implantation

Docker-Compose

See Also
「Apache Airflow のインストール パート 1」

Implementation details

execution:"3.9"Services:postgraduate:start again:forever image:postgres: neueste container name:dpg_postgres neighborhood: -POSTGRES_USER=Airflow-POSTGRES_PASSWORD=Airflow-POSTGRES_DB=Airflow doors: -"5434:5432" web server:container name:dpg_airflow build up:context:. argument:PYTHON_BASE_IMAGE:"python: 3.7-slim-buster" PYTHON_MAJOR_MINOR_VERSION:3.7 AIR FLOW_INSTALLATION_METHOD:"Apache-Airflow" VERSION_AIR FLOW:"2.2.1" AIRFLOW_INSTALL_VERSION:"==2.2.1" AIRFLOW_CONSTRAINTS_REFERENCE:"Constraints-2-0" AIR FLOW_SOURCES_OF:"office hour" AIR FLOW_SOURCES_TO:"/office hour" EXTRA_ADDITIONAL_AIR_FLOW:"apache-airflow-proveedor-amazon \apache-airflow-anbieter-http \apache-airflow-proveedor-hashicorp \apache-airflow-proveedor-docker \apache-airflow-proveedor-ftp \apache-airflow-anbieter-postgres \apache-airflow-proveedor-sftp \apache-airflow-proveedor-ssh" DEV_ADDITIONAL_APT_DEPS:"gcc g++" neighborhood:key name:group_project:host name:web server start again:forever depends on: -postgraduate file_env: -.env volumes: -./dags:/opt/airflow/dags-./plugins:/opt/airflow/plugins-Volume of data:/data/project_meta doors: -"8090:8080" entry point:./dags/scripts/airflow-punto de entrada.sh health check:Test:["CMD-SHELL", "[ -f ./opt/airflow/airflow-webserver.pid ]"] interruption:30er Time is up:30er repeat:32volumes:data volume:

What happened

Hi, I was trying to connect to the docker daemon from a containerized airflow setup. I saw this problem:#16803Looking at this I got tired of migrating my Airflow version to 2.2.1 and setting mount_tmp_dir=False

I've tried running this using two approaches.

(Video) [Solved] Docker Image Pull Permission Denied While Trying To Connect To The Docker Daemon Error

  1. If the TCP Docker URL is not specified
  2. Define a Docker TCP URL

On first approach I got this error:

***readinglocalFile: /opt/airflow/logs/hello_world/echo_name/2021-11-10T04:14:45.866031+00:00/1.log[2021-11-10, 04:14:47 UTC] {taskinstance.py:1035 } INFO - All dependencies were metthrough <TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [en archivo]>[2021-11-10, 04:14:47 UTC] {taskinstance.py:1035} INFO - All dependencies satisfiedthrough <TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [en archivo]>[2021-11-10, 04:14:47 UTC] {taskingstance.en:1241} INFO - ---------------------- ----- -- ---------------------------------------------------------------- -------- ------- ----------------------------------- ------ -- -[10.11.2021, 04:14:47 UTC] {taskinstance.py:1242} INFORMATION - First attempt 1 of 2[2021-11-10, 04:14:47 UTC] { taskinstance.py:1242} 1243} INFORMATION - ------- --------------------------------- . ------ ----------- ----- ---------------------------------- -[11/11/2021, 04:14:47 UTC] { task status .py: 1262} INFO - In execution<Task (DockerOperator): echo_name>on 2021-10-11 04:14:45.866031+00:00[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:52} INFO - Process started 5836 to run task[2021-11- 10 , 04:14:47 UTC] {standard_task_runner.py:76} INFO - Running: ['***','Chores','according to','Hello World','echo_name','Manual__2021-11-10T04:14:45.866031+00:00','--Work ID','2','--cru','--subdirectory','CARPETA_DAGS/dag_creator.py','--ruta cfg','/tmp/tmp39jdfg29','--error file','/tmp/tmp2tmtb5n4'][10/10/2021 04:14:47 UTC] {standard_task_runner.en:77} INFO - Job 2: Subtask echo_name[10/10/2021 04:14:47 UTC] {logging_mixin.en:109} INFO - Running<TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [running]>on web server host[2021-11-10, 04:14:47 UTC] {taskinstance.py:1429} INFO - Exporting the following environment variables: AIRFLOW_CTX_DAG_OWNER=***Airflow_ctx_dag_id = hello_worldairflow_ctx_task_id = echo_nameairflow_ctx_execution_date = 2021-11-10t04: 14: 45.866031+00: 00airflow_ctxx_dag_run_id = manual__2021-11-110t0444: 141111111106011. py:1703} ERROR: la tarea falló con el rastreo de excepción (última última llamada): archivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43,Insideconnect sock.connect(self.unix_socket)FileNotFoundError: [Errno 2] No such file or directory Another exception occurred during exception handling above: trace(last last call): file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449,Insideshipping lead time = Lead Time-Date"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756,InsideMétodo urlopen, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] Arquivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532,Insideincrease increase six.reupload(type(error), error, _stacktrace) Datei"/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769,Insidereupload increase value.with_traceback(tb) File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43,Insideconectar sock.connect(self.unix_socket)urllib3.exceptions.ProtocolError: ('Lost connection.', File not found error (2,'The file or directory does not exist'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214,Inside_retrieve_server_versionReturnsself.version(api_version=False)["ApiVersion"] office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181,InsideexecutionReturnsself._result(self._get(url), json=True) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46,InsideinternalReturnsf(myself,*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237,Inside_Take itReturnsself.get (URL,**self._set_request_timeout(kwargs)) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555,Insidetake itReturnsauto.request('TAKE IT', URL,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542,Insidesolicitud o = self.send(prep,**send_kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655,Insidesend r = adapter. send request,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498,Insideenviar aumentar ConnectionError(err, request=request)requests.exceptions.ConnectionError: ('Lost connection.', File not found error (2,'The file or directory does not exist'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332,Inside_run_raw_task self._execute_task_with_callbacks(contexto) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458,Inside_execute_task_with_callbacks resultado = self._execute_task(contexto, self.tarea) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514,Inside_run_task result = run_callable(context=context) File"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367,Insideself.cli = executeself._get_cli() office hour"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397,Inside_get_cliReturnsAPIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config) Fechai"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197,Inside__init__ self._version =self._retrieve_server_version() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222,Inside_retrieve_server_version'Error getting server api version: {0}'.format(e)docker.errors.DockerException: ErroresWeatherGet server API version :('Lost connection.', File not found error (2,'The file or directory does not exist'))[2021-11-10, 04:14:47 UTC] {taskinstance.py:1280} INFO - Marcar tarea como UP_FOR_RETRY. dag_id=hello_world, task_id=echo_name, deployment_date=20211110T041445, start_date=20211110T041447, end_date=20211110T041447[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:91} ERROR: error al ejecutar el trabajo 2throughecho_nameTraceback task (last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43,Insideconnect sock.connect(self.unix_socket)FileNotFoundError: [Errno 2] No such file or directory Another exception occurred during exception handling above: trace(last last call): file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449,Insideshipping lead time = Lead Time-Date"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756,InsideMétodo urlopen, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] Arquivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532,Insideincrease increase six.reupload(type(error), error, _stacktrace) Datei"/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769,Insidereupload increase value.with_traceback(tb) File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43,Insideconectar sock.connect(self.unix_socket)urllib3.exceptions.ProtocolError: ('Lost connection.', File not found error (2,'The file or directory does not exist'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214,Inside_retrieve_server_versionReturnsself.version(api_version=False)["ApiVersion"] office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181,InsideexecutionReturnsself._result(self._get(url), json=True) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46,InsideinternalReturnsf(myself,*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237,Inside_Take itReturnsself.get (URL,**self._set_request_timeout(kwargs)) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555,Insidetake itReturnsauto.request('TAKE IT', URL,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542,Insidesolicitud o = self.send(prep,**send_kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655,Insidesend r = adapter. send request,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498,Insideenviar aumentar ConnectionError(err, request=request)requests.exceptions.ConnectionError: ('Lost connection.', File not found error (2,'The file or directory does not exist'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85,Inside_start_by_fork args .func ( args , day= self .day ) Datei"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48,Inside domain ReturnsOccupation(*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92,InsidePackageReturnsF(*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292,Insidetask_run _run_task_by_s selected_method (argumentos, día, hora) Fechai"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107,Inside_run_task_by_s selected_method _run_raw_task(args, ti) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184,Inside_run_raw_task error_file=args.error_file, Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70,InsidePackageReturnsOccupation(*arguments, session=session,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332,Inside_run_raw_task self._execute_task_with_callbacks(contexto) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458,Inside_execute_task_with_callbacks resultado = self._execute_task(contexto, self.tarea) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514,Inside_run_task result = run_callable(context=context) File"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367,Insideself.cli = executeself._get_cli() office hour"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397,Inside_get_cliReturnsAPIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config) Fechai"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197,Inside__init__ self._version =self._retrieve_server_version() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222,Inside_retrieve_server_version'Error getting server api version: {0}'.format(e)docker.errors.DockerException: ErroresWeatherGet server API version :('Lost connection.', File not found error (2,'The file or directory does not exist'))[2021-11-10, 04:14:47 UTC] {local_task_job.py:154} INFO: Job ended withReturnsCode 1[2021-11-10, 04:14:47 UTC] {local_task_job.py:264} INFO: 0 post scheduled tasks as of post schedule check

In the second approach I got this error:

***readinglocalFile: /opt/airflow/logs/hello_world/echo_name/2021-11-10T04:39:06.388937+00:00/1.log[2021-11-10, 04:39:07 UTC] {taskinstance.py:1035 } INFO - All dependencies were metthrough <TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [en archivo]>[2021-11-10, 04:39:07 UTC] {taskinstance.py:1035} INFO - All dependencies satisfiedthrough <TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [en archivo]>[2021-11-10, 04:39:07 UTC] {taskingstance.en:1241} INFO - ---------------------- ----- -- ---------------------------------------------------------------- -------- ------- ----------------------------------- ------ -- -[10.11.2021, 04:39:07 UTC] {taskinstance.py:1242} INFORMATION - First attempt 1 of 2[2021-11-10, 04:39:07 UTC] { taskinstance.py:1242} 1243} INFORMATION - ------- --------------------------------- . ------ ----------- ----- ---------------------------------- -[11/11/2021, 04:39:07 UTC] { task status .py: 1262} INFO - In execution<Task (DockerOperator): echo_name>on 10/11/2021 04:39:06.388937+00:00[2021-11-10, 04:39:07 UTC] {standard_task_runner.py:52} INFO: process started 13507 to run task[2021-11 -10 , 04:39:07 UTC] {standard_task_runner.py:76} INFO - Running: ['***','Chores','according to','Hello World','echo_name','Manual__2021-11-10T04:39:06.388937+00:00','--Work ID','2','--cru','--subdirectory','CARPETA_DAGS/dag_creator.py','--ruta cfg','/tmp/tmphw9dkz6v','--error file','/tmp/tmp5glruzwd'][10/10/2021 04:39:07 UTC] {standard_task_runner.en:77} INFO - Task 2: Subtask echo_name[10/10/2021 04:39:08 UTC] {logging_mixin.en:109} INFO - Running<TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [running]>on web server host[2021-11-10, 04:39:08 UTC] {taskinstance.py:1429} INFO - Exporting the following environment variables: AIRFLOW_CTX_DAG_OWNER=***AIRFLOW_CTX_DAG_ID=hello_worldAIRFLOW_CTX_TASK_ID=echo_nameAIRFLOW_CTX_EXECUTION_DATE=2021-11-10T04:39:06.388937+00:00AIRFLOW_CTX_DAG_RUN_ID=manual__2021-11-10T04:39:06.388937+00:00[2021-11-10, 04:39:08 UTC] {taskinstance. py : 1703 } ERROR : the task failed with exception tracing ( last last call ): file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 175,Inside_new_conn (self._dns_host, self.port), self.timeout,**extra_kw file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96,Insidecreate_connection Raise-Fehlerdatei"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86,Insidecreate_connection sock.connect(sa)ConnectionRefusedError: [Errno 111] Connection Refused Another exception occurred during exception handling above: Trace(last last call): file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 239,InsideAnfrage super(HTTPConnection, self).request(method, url, body=body, headers=headers) Datei"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 205,Insideconnect with =self._new_conn() office hour"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 187,Inside_new_conn automatically,"Failed to establish new connection: %s"% eurllib3.exceptions.Reconnection failed:<Objeto urllib3.connection.HTTPConnection em 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused While handling the above exception, another exception occurred: Rollback (last last call): file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449,Insideshipping lead time = Lead Time-Date"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756,InsideMétodo urlopen, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] Arquivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574,Insideincrease increase MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214,Inside_retrieve_server_versionReturnsself.version(api_version=False)["ApiVersion"] office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181,InsideexecutionReturnsself._result(self._get(url), json=True) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46,InsideinternalReturnsf(myself,*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237,Inside_Take itReturnsself.get (URL,**self._set_request_timeout(kwargs)) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555,Insidetake itReturnsauto.request('TAKE IT', URL,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542,Insidesolicitud o = self.send(prep,**send_kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655,Insidesend r = adapter. send request,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 516,Insideenviar aumentar ConnectionError(e, request=request)requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332,Inside_run_raw_task self._execute_task_with_callbacks(contexto) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458,Inside_execute_task_with_callbacks resultado = self._execute_task(contexto, self.tarea) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514,Inside_run_task result = run_callable(context=context) File"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367,Insideself.cli = executeself._get_cli() office hour"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397,Inside_get_cliReturnsAPIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config) Fechai"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197,Inside__init__ self._version =self._retrieve_server_version() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222,Inside_retrieve_server_version'Error getting server api version: {0}'.format(e)docker.errors.DockerException: ErroresWeatherGet the API version of the server: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))[2021-11-10, 04:39:08 UTC] {taskinstance.py:1280} INFO - Marcando tarefa como UP_FOR_RETRY. dag_id=hello_world, task_id=echo_name, deployment_date=20211110T043906, start_date=20211110T043907, end_date=20211110T043908[2021-11-10, 04:39:08 UTC] {standard_task_runner.py:91} ERROR - Falcutha 2 ua excuthathroughecho_nameTraceback task (last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 175,Inside_new_conn (self._dns_host, self.port), self.timeout,**extra_kw file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96,Insidecreate_connection Raise-Fehlerdatei"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86,Insidecreate_connection sock.connect(sa)ConnectionRefusedError: [Errno 111] Connection Refused Another exception occurred during exception handling above: Trace(last last call): file"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706,Insideurlopen fragmented = fragmented, File"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394,Inside_make_request conn.request(Método, URL,**httplib_request_kw) Archivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 239,InsideAnfrage super(HTTPConnection, self).request(method, url, body=body, headers=headers) Datei"/usr/local/lib/python3.7/http/client.py", line 1281,InsideAnfrage self._send_request(method, url, body, headers, encode_chunked) File"/usr/local/lib/python3.7/http/client.py", line 1327,Inside_send_request self.endheaders(cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1276,Insideendheaders self._send_output(mensaje_cuerpo, encode_chunked=encode_chunked) Arquivo"/usr/local/lib/python3.7/http/client.py", line 1036,Inside_send_output self.send(mensaje) Arquivo"/usr/local/lib/python3.7/http/client.py", line 976,Insideof youauto.connect() office hour"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 205,Insideconnect with =self._new_conn() office hour"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 187,Inside_new_conn automatically,"Failed to establish new connection: %s"% eurllib3.exceptions.Reconnection failed:<Objeto urllib3.connection.HTTPConnection em 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused While handling the above exception, another exception occurred: Rollback (last last call): file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449,Insideshipping lead time = Lead Time-Date"/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756,InsideMétodo urlopen, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] Arquivo"/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574,Insideincrease increase MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214,Inside_retrieve_server_versionReturnsself.version(api_version=False)["ApiVersion"] office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181,InsideexecutionReturnsself._result(self._get(url), json=True) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46,InsideinternalReturnsf(myself,*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237,Inside_Take itReturnsself.get (URL,**self._set_request_timeout(kwargs)) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555,Insidetake itReturnsauto.request('TAKE IT', URL,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542,Insidesolicitud o = self.send(prep,**send_kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655,Insidesend r = adapter. send request,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 516,Insideenviar aumentar ConnectionError(e, request=request)requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))Another exception occurred while handling the above exception: trace(last most recent call): file"/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85,Inside_start_by_fork args .func ( args , day= self .day ) Datei"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48,Inside domain ReturnsOccupation(*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92,InsidePackageReturnsF(*arguments,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292,Insidetask_run _run_task_by_s selected_method (argumentos, día, hora) Fechai"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107,Inside_run_task_by_s selected_method _run_raw_task(args, ti) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184,Inside_run_raw_task error_file=args.error_file, Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70,InsidePackageReturnsOccupation(*arguments, session=session,**kwargs) file"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332,Inside_run_raw_task self._execute_task_with_callbacks(contexto) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458,Inside_execute_task_with_callbacks resultado = self._execute_task(contexto, self.tarea) Arquivo"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514,Inside_run_task result = run_callable(context=context) File"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367,Insideself.cli = executeself._get_cli() office hour"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397,Inside_get_cliReturnsAPIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config) Fechai"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197,Inside__init__ self._version =self._retrieve_server_version() office hour"/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222,Inside_retrieve_server_version'Error getting server api version: {0}'.format(e)docker.errors.DockerException: ErroresWeatherGet the API version of the server: HTTPConnectionPool(host='127.0.0.1', porta=2376): Maximum attempts exceeded com url: /version (caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Error establishing a new connection: [Errno 111] Connection refused'))[2021-11-10, 04:39:08 UTC] {local_task_job.py:154} INFO: Job ended withReturnsCode 1[2021-11-10, 04:39:08 UTC] {local_task_job.py:264} INFO: 0 post scheduled tasks as of post schedule check
(Video) Docker - Tutorial 2 - Images, Containers, and Docker Server (Daemon)

In both cases, the similar error is Failed to get server API version: HTTPConnectionPool.

I can no longer debug. If you could tell me how to fix this problem, that would be a great help!

Thank you very much.

What did you expect?

Unanswered

(Video) Airflow docker install python packages: Airflow Tutorial P11

how to reproduce

Set up an Ubuntu operating system on an EC2 instance. Create an Airflow container with Airflow version = 2.2.1
Configure docker and docker-compose
Configure TCP access to the Docker daemon with the following commands:

sudo mkdir -p /etc/systemd/system/docker.service.dsudo touch /etc/systemd/system/docker.service.d/options.confecological-mi"""[Service]HomeExecutive=ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375""" >>/etc/systemd/system/docker.service.d/options.confsudo systemctl daemon-reloadsudo systemctl reinicio docker

Create and run a simple Docker operator in a dag.

One more thing

Unanswered

(Video) 5 MUST KNOW Airflow debug tips and tricks | Airflow Tutorial Tips 1

Ready to send PR?

  • Yes, I'm ready to send a PR!

Code of conduct

FAQs

How do I fix this error during connect this error may indicate that the Docker daemon is not running? ›

How to Fix the Docker Daemon is not Running Error on Windows?
  • Step 1: Open Windows PowerShell. ...
  • Step 2: Check Docker Version. ...
  • Step 2: Check the Docker. ...
  • Step 3: Check Docker Desktop Service. ...
  • Step 4: Download and Install Update WSL. ...
  • Step 5: Verify the Error is resolved.

Read The Full Story ›
Why is my Docker daemon not running? ›

The reason for this problem could either be that the command is not actually installed on the machine or that it is installed but not configured correctly. Let's first understand the Docker daemon (dockerd). It is a program that manages all the Docker objects, including images, containers, volumes and many more.

Learn More Now ›
How do I connect to Docker daemon? ›

How to Connect to a Remote Docker Daemon
  1. sudo mkdir -p /etc/systemd/system/docker.service.d.
  2. sudo nano /etc/systemd/system/docker.service.d/options.conf.
  3. [Service] ExecStart= ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375.
  4. # Reload the systemd daemon.

Read On ›
How do I fix Docker got permission denied while trying to connect to the Docker daemon socket? ›

Fix 1: Run all the docker commands with sudo

If you have sudo access on your system, you may run each docker command with sudo and you won't see this 'Got permission denied while trying to connect to the Docker daemon socket' anymore.

See Details ›
How do I know if Docker daemon is running? ›

The operating-system independent way to check whether Docker is running is to ask Docker, using the docker info command. You can also use operating system utilities, such as sudo systemctl is-active docker or sudo status docker or sudo service docker status , or checking the service status using Windows utilities.

Read On ›
What is daemon mode in Docker? ›

The Docker daemon ( dockerd ) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services.

Get More Info ›
How do I start Docker daemon or service? ›

HTTP/HTTPS proxy
  1. Create a systemd drop-in directory for the docker service: $ sudo mkdir -p /etc/systemd/system/docker.service.d.
  2. Flush changes and restart Docker. $ sudo systemctl daemon-reload $ sudo systemctl restart docker.

Read On ›
Is Docker Engine and Docker daemon same? ›

Docker Engine is the core product of Docker, including its daemon (dockerd) as well as its CLI (docker). Docker Daemon is simply a part of Docker Engine. Quoting the Docker engine overview page: Docker Engine is an open source containerization technology for building and containerizing your applications.

View More ›
How do I bring up Docker daemon in Linux? ›

Install Docker
  1. Log into your system as a user with sudo privileges.
  2. Update your system: sudo yum update -y .
  3. Install Docker: sudo yum install docker-engine -y.
  4. Start Docker: sudo service docker start.
  5. Verify Docker: sudo docker run hello-world.

Learn More Now ›
How do I fix Docker engine failed to start? ›

Analysis
  1. Restart Docker Desktop that helps most of the time.
  2. Windows reboot, welcome to the world of Microsoft.
  3. Next uninstall, Reboot and Re-Install Docker Desktop (This cleared all the image cache)
Aug 26, 2021

Tell Me More ›

How do I connect to Docker daemon Windows? ›

To start Docker in daemon mode, choose Application > Start "Docker Daemon". The state should transition to "Running" after a few seconds and Docker Daemon should be accessible over the remote bridge. That's it! Next time your computer boots, Docker Daemon will start up immediately, before anyone logs on.

View More ›

Videos

1. Running Airflow 2.0 with Docker in 5 mins
(Data with Marc)
2. How to REMOVE airflow example DAGs locally and in docker | Airflow Tutorial Tips 2
(coder2j)
3. Running Airflow 2.0 in Docker: Airflow Tutorial P2
(coder2j)
4. Airflow DockerOperator: The Basics (and more 🤫)
(Data with Marc)
5. Run Airflow 2.0 via Docker | Minimal Setup | Apache airflow for beginners
(MaxcoTec Learning)
6. Airflow Dag Package management | Why use KubernetesPodOperator & DockerOperator ?
(MaxcoTec Learning)
Top Articles
Latest Posts
Article information

Author: Ms. Lucile Johns

Last Updated: 03/25/2023

Views: 5556

Rating: 4 / 5 (61 voted)

Reviews: 92% of readers found this page helpful

Author information

Name: Ms. Lucile Johns

Birthday: 1999-11-16

Address: Suite 237 56046 Walsh Coves, West Enid, VT 46557

Phone: +59115435987187

Job: Education Supervisor

Hobby: Genealogy, Stone skipping, Skydiving, Nordic skating, Couponing, Coloring, Gardening

Introduction: My name is Ms. Lucile Johns, I am a successful, friendly, friendly, homely, adventurous, handsome, delightful person who loves writing and wants to share my knowledge and understanding with you.