Production Guide — Helmet Card Documentation (2023)

Data bank¶

It is recommended to set up an external database for the Airflow metastore. The default Helm chart implements a Postgres database running in a container. For production use, a database running on a dedicated computer or using a database service from a cloud provider such as AWS RDS is recommended. Supported databases and versions can be found atConfigure the database backend.

First, disable Postgres so that the diagram doesn't implement its own Postgres container:

postgresql: capable: vals

To provide database credentials to Airflow, you have two options: in your values ​​file or in the Kubernetes secret.

the facts: metapodaciVerbinding: user:  spend:  protocol: postgresql host:  porta: 5432 database: 

kubectl to make secret In general my database --verbatim=connection=postgresql://user:pass@host:5432/db

the facts: metadadosSecretName: my database

PgBouncer¶

If you use PostgreSQL as your database, you probably want to enable itPgBouncerAirflow can open many database connections due to its distributed nature, and using a connection pool can significantly reduce the number of open database connections.

pgbouncer: capable: WHERE

Depending on the size of your Airflow instance, you can also customize the following (default values ​​are shown):

pgbouncer: # Maximum number of connections to PgBouncer maxClientConn: 100 # Maximum number of server connections to the PgBouncer metadata database metadadosPoolSize: 10 # Maximum number of server connections to the PgBouncer backend results database resultadoBackendPoolSize: 5

The secret key of the web server¶

When deploying this image you must set a static secret for the web server as this will ensure that the Airflow components are only restarted when necessary.

First, generate a strong secret key:

python3 -C 'import secrets; print(secrets.token_hex(16))'

Now add the secret to your values ​​file:

webSecretKey service provider: 

You can also create and use a Kubernetes secretwebSecretKeySecretName service provider:

webSecretKeySecretName service provider: my-secret-server# where the random key in `webserver-secret-key` is in k8s Secret

An example for creating a Kubernetes secretkubectl:

kubectl to make secret In general my-secret-server --verbatim="webserver-secret-key=$(python3 -C 'import secrets; print(secrets.token_hex(16))')"

The web server key is also used to authorize requests to Celery employees when logs are retrieved. A token generated using a secret key has a short expiration time. Make sure the time is synchronized on ALL the machines you run the airflow components on (eg using ntpd) or you could get 'forbidden' errors when the logs are opened.

Expand and adjust the airflow image¶

The Apache Airflow community publishes Docker images that arereference Suchfor the Apache Airstream. However, Airflow has more than 60 community managed providers (which can be installed via plugins) and some of the pre-installed plugins/providers are not used by everyone, sometimes other plugins/providers are needed, sometimes (very often actually) you need to create your own extras / providers add dependencies, packages, or even custom providers, or add custom tools and binaries needed for your deployment.

In terms of Kubernetes and Docker, this means you need a different image with your specific requirements. So you have to learn how to build your own image.Dock worker(or ratherBracket) picture.

Typical scenarios where you want to use your custom image:

• To addanswerpackage

• To addPyPIpackage

• Adding the necessary binary resources for your deployment

• Adding the necessary custom tools to your deployment

verimage buildingfor more information on how to expand and adjust the airflow image.

Manage DAY files¶

verManaging DAG files.

ssh-keyscan -T RSA github.com > github_public_key

ssh-key -se github_public_key

time of day: gitSync: famous hosts: | github.com ssh-rsa AAAA...1/wsjk=

Access to the Airflow user interface¶

How you access the Airflow user interface depends on your environment; however, the chart supports several options:

web server: employ: tip: load balancer

log¶

Depending on your choice of executor, job logs may not work by default. You can find all registration options heremanage the administration.

Statistics¶

A graph can support sending statistics to an existing StatsD instance or provide a Prometheus endpoint.

stands: capable: valsinstitution: Statistics: # or 'planner' for airflow 1 on_state: WHERE host_statistics: ... port_state: ...

stands: capable: valsinstitution: Statistics: # or 'planner' for airflow 1 on_state: WHERE port_state: 8125extraEnv: |- - naziv: AIRFLOW__METRICS__STATSD_HOST the courage: campoRef: veldPath: status.hostIP

Background van aipo¶

ako korositieAipoExecutorfromAipoKubernetesExecutor, you can bring your own Celery wallpaper.

By default the graph is implemented by Redis. However, you can use any supported Celery backend:

again: capable: valsthe facts: brokerUrl: redis://redis-user:password@redis-host:6379/0

For more information on setting up the Celery broker, see the extensive documentCelery documentation on this topic.

Restrictions in the security context¶

ASecurity Context Limit(SCC) is an OpenShift construct that works as an RBAC rule; however, it focuses on pods rather than users. Defining an SCC allows you to control the actions and resources that a POD can perform or access during startup and execution.

SCCs are divided into different levels or categories slimitedSCC is the default value assigned to pods. By implementing Airflow on OpenShift you can use SCCs and allow pods to run containers usingyour momSCC.

To enable the use of SCCs, you must set the parameterrbac.createSCCRoleBindingforWHEREas shown below:

rbac: to make: WHERE criarSCCRoleBinding: WHERE

In this diagram, SCCs are connected to pods via RoleBindings, which means an optionrbac. to createmust also be defined asWHEREfully enable the use of SCC.

For more information on SCCs and what can be achieved with this build, seeManage restrictions in the security context.

security context¶

In Kubernetes onesecurity contextcan be used to define user IDs, group IDs, and resources, such as running containers in privileged mode.

When deploying an application to Kubernetes, it is recommended to grant least privilege to containers to limit access and secure the host on which the container is running.

In the Airflow Helm chart,security contextIt can be configured in several ways:

• UID(sets global uid or RunAsUser)

• feliz(stel globale gid of fsGroup in)

• security contexts(ButUIDbut it allows defining everythingContext options for Under SecurityeContext options for container security)

The same way you use globalsecurity contexts, you can also configure different values ​​for specific workloads by setting their local valuessecurity contextsas follows:

employees: security contexts: Fulani: runAsUser: 5.000 fs group: 0 containers: allow privilege escalation: vals

In the example above, the Task subsecurity contextswill be setrunAsUser: 5.000efs group: 0. The container is set toallow privilege escalation: vals.

As you can see, the local setting overrides the global setting when set. The priority rule is explained below forsecurity contextsoptions in this diagram:

UID: 40.000feliz: 0security contexts: Fulani: runAsUser: 50.000 fs group: 0employees: security contexts: Fulani: runAsUser: 1001 fs group: 0

This generates the following worker deployment:

tip: StatefulSetAPI version: applications/v1metapodaci: To do: air flow workerspecification: Service name: air flow worker model: specification: security context: # Since securityContexts is defined in ``workers'', its value will take precedence runAsUser: 1001 fs group: 0

If we delete bothsecurity contextseemployees, security contextsfrom the example above, the output will be:

UID: 40.000feliz: 0security contexts: {}employees: security contexts: {}

This generates the following worker deployment:

tip: StatefulSetAPI version: applications/v1metapodaci: To do: air flow workerspecification: Service name: air flow worker model: specification: security context: runAsUser: 40.000 # Since no security context is defined in ``workers'' or ``podSecurity'', the uid value will be used fs group: 0 # Since the security context is not defined in ``workers'' or ``podSecurity'', the gid value will be used initContainers: - To do: wait for airflow migration ... containers: - To do: worker ...

And finally if we definesecurity contextsbut noemployees, security contexts:

UID: 40.000feliz: 0security contexts: Fulani: runAsUser: 50.000 fs group: 0employees: security contexts: {}

This generates the following worker deployment:

tip: StatefulSetAPI version: applications/v1metapodaci: To do: air flow workerspecification: Service name: air flow worker model: specification: security context: # Since securityContexts is not defined in ``workers'', securityContexts values ​​will take precedence runAsUser: 50.000 fs group: 0 initContainers: - To do: wait for airflow migration ... containers: - To do: worker ...

Embedded secrets and environment variables¶

Helm Chart uses standard Kubernetes secrets to store the secrets needed by Airflow. By default, the contents of those secrets are converted into environment variables that Airflow reads (some environment variables have multiple variants to support older versions of Airflow).

By default, secrets are determined based on the version name used when the Helm diagram is deployed, but you can also use a different secret to define variables, or disable the use of secrets entirely and trust variables. use_CMDfrom__SECRETenvironment variable variant.

However, Airflow supports other variants of the secret configuration - you can issue a system command to automatically retrieve and rotate the secret (by setting a variable with_CMDsuffix) or to retrieve a variable from a supported secret (by defining the variable s_SECRETsuffix).

See Fr>is placed, takes precedence over_CMDe_SECRETvariant, so if you have one of_CMDfrom_SECRETvariants, you MUST disable built-in invariants fetched from Kubernetes secrets by setting.Values.enableBuiltInSecretEnvVars.it is false.

For example, if you want to use the command to retrieve a database connection, you must (in yourvalues.yamlfile) specify:

extraEnv: AIRFLOW_CONN_AIRFLOW_DB_CMD: "/usr/local/bin/retrieve_connection_url"enableBuiltInSecretEnvVars: AIRFLOW_CONN_AIRFLOW_DB: vals

Here is the full list of secrets that can be disabled and replaced_CMDe_SECRETvariants:

There are also several secrets, whose names are also determined based on the version name, that do not need to be disabled. It's because they don't follow the rules_CMDfrom_SECRETby default are variables that do not start withAIR FLOW__, or have no corresponding variable.

you i i a_AIR FLOW__*variable,AIRFLOW__CELERY__FLOWER_BASIC_AUTH, which does not need to be disabled even if you have_CMDe_SECRETvariant. This variable is not defined by default. It is defined only when.Values.flower.secretNameis set or when.values.flower.usere.values.flower.passwordare ready. So if you don't have any of the.Values.flower.*variables, you can freely configure basic flower authentication using_CMDfrom_SECRETvariant without deactivating the base variant.

You can read more about advanced ways to set configuration variables atSet configuration options.