Bitnami Docker Image for Apache Airflow
In order to unify the approaches followed for Bitnami containers and Bitnami Helm charts, we are moving the different bitnami/bitnami-docker-<container>
repositories to a single monorepo bitnami/containers
. Please follow bitnami/containers to keep you updated about the latest Bitnami images.
More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html
Apache Airflow is a tool to express and execute workflows as directed acyclic graphs (DAGs). It includes utilities to schedule tasks, monitor task progress and handle task dependencies.
Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.
$ curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-airflow/master/docker-compose.yml
$ docker-compose up
Warning: This quick setup is only intended for development environments. You are encouraged to change the insecure default credentials and check out the available configuration options in the Environment Variables section for a more secure deployment.
DOCKER_CONTENT_TRUST=1
to verify the integrity of the images.Dockerfile
linksLearn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags in our documentation page.
Subscribe to project updates by watching the bitnami/airflow GitHub repo.
To run this application you need Docker Engine >= 1.10.0
. Docker Compose is recommended with a version 1.6.0
or later.
Airflow requires access to a PostgreSQL database to store information. We will use our very own PostgreSQL image for the database requirements. Additionally, if you pretend to use the CeleryExecutor
, you will also need an Airflow Scheduler, one or more Airflow Workers and a Redis(R) server.
The main folder of this repository contains a functional docker-compose.yml
file. Run the application using it as shown below:
$ curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-airflow/master/docker-compose.yml > docker-compose.yml
$ docker-compose up -d
If you want to run the application manually instead of using docker-compose
, these are the basic steps you need to run:
$ docker network create airflow-tier
$ docker volume create --name postgresql_data
$ docker run -d --name postgresql \
-e POSTGRESQL_USERNAME=bn_airflow \
-e POSTGRESQL_PASSWORD=bitnami1 \
-e POSTGRESQL_DATABASE=bitnami_airflow \
--net airflow-tier \
--volume postgresql_data:/bitnami/postgresql \
bitnami/postgresql:latest
$ docker volume create --name redis_data
$ docker run -d --name redis \
-e ALLOW_EMPTY_PASSWORD=yes \
--net airflow-tier \
--volume redis_data:/bitnami \
bitnami/redis:latest
$ docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e [email protected] \
--net airflow-tier \
bitnami/airflow:latest
$ docker run -d --name airflow-scheduler \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_WEBSERVER_HOST=airflow \
--net airflow-tier \
bitnami/airflow-scheduler:latest
$ docker run -d --name airflow-worker \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_WEBSERVER_HOST=airflow \
--net airflow-tier \
bitnami/airflow-worker:latest
Access your application at http://your-ip:8080
The Bitnami Airflow container relies on the PostgreSQL database & Redis to persist the data. This means that Airflow does not persist anything. To avoid loss of data, you should mount volumes for persistence of PostgreSQL data and Redis(R) data
The above examples define docker volumes namely postgresql_data
, and redis_data
. The Airflow application state will persist as long as these volumes are not removed.
To avoid inadvertent removal of these volumes you can mount host directories as data volumes. Alternatively you can make use of volume plugins to host the volume data.
The following docker-compose.yml
template demonstrates the use of host directories as data volumes.
version: '2'
services:
postgresql:
image: 'bitnami/postgresql:latest'
environment:
- POSTGRESQL_DATABASE=bitnami_airflow
- POSTGRESQL_USERNAME=bn_airflow
- POSTGRESQL_PASSWORD=bitnami1
volumes:
- /path/to/postgresql-persistence:/bitnami/postgresql
redis:
image: 'bitnami/redis:latest'
environment:
- ALLOW_EMPTY_PASSWORD=yes
volumes:
- /path/to/redis-persistence:/bitnami
airflow-worker:
image: bitnami/airflow-worker:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_LOAD_EXAMPLES=yes
airflow-scheduler:
image: bitnami/airflow-scheduler:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_LOAD_EXAMPLES=yes
airflow:
image: bitnami/airflow:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_PASSWORD=bitnami123
- AIRFLOW_USERNAME=user
- [email protected]
ports:
- '8080:8080'
$ docker network create airflow-tier
$ docker run -d --name postgresql \
-e POSTGRESQL_USERNAME=bn_airflow \
-e POSTGRESQL_PASSWORD=bitnami1 \
-e POSTGRESQL_DATABASE=bitnami_airflow \
--net airflow-tier \
--volume /path/to/postgresql-persistence:/bitnami \
bitnami/postgresql:latest
$ docker run -d --name redis \
-e ALLOW_EMPTY_PASSWORD=yes \
--net airflow-tier \
--volume /path/to/redis-persistence:/bitnami \
bitnami/redis:latest
$ docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e [email protected] \
--net airflow-tier \
bitnami/airflow:latest
$ docker run -d --name airflow-scheduler \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_WEBSERVER_HOST=airflow \
--net airflow-tier \
bitnami/airflow-scheduler:latest
$ docker run -d --name airflow-worker \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_WEBSERVER_HOST=airflow \
--net airflow-tier \
bitnami/airflow-worker:latest
Custom DAG files can be mounted to /opt/bitnami/airflow/dags
.
This container supports the installation of additional python modules at start-up time. In order to do that, you can mount a requirements.txt
file with your specific needs under the path /bitnami/python/requirements.txt
.
The Airflow instance can be customized by specifying environment variables on the first run. The following environment values are provided to customize Airflow:
AIRFLOW_USERNAME
: Airflow application username. Default: user
AIRFLOW_PASSWORD
: Airflow application password. Default: bitnami
AIRFLOW_EMAIL
: Airflow application email. Default: [email protected]
AIRFLOW_EXECUTOR
: Airflow executor. Default: SequentialExecutor
AIRFLOW_FERNET_KEY
: Airflow Fernet key. No defaults.AIRFLOW_SECRET_KEY
: Airflow Secret key. No defaults.AIRFLOW_WEBSERVER_HOST
: Airflow webserver host. Default: 127.0.0.1
AIRFLOW_WEBSERVER_PORT_NUMBER
: Airflow webserver port. Default: 8080
AIRFLOW_LOAD_EXAMPLES
: To load example tasks into the application. Default: yes
AIRFLOW_BASE_URL
: Airflow webserver base URL. No defaults.AIRFLOW_HOSTNAME_CALLABLE
: Method to obtain the hostname. No defaults.AIRFLOW_POOL_NAME
: Pool name. No defaults.AIRFLOW_POOL_SIZE
: Pool size, required with AIRFLOW_POOL_NAME
. No defaults.AIRFLOW_POOL_DESC
: Pool description, required with AIRFLOW_POOL_NAME
. No defaults.AIRFLOW_DATABASE_HOST
: Hostname for PostgreSQL server. Default: postgresql
AIRFLOW_DATABASE_PORT_NUMBER
: Port used by PostgreSQL server. Default: 5432
AIRFLOW_DATABASE_NAME
: Database name that Airflow will use to connect with the database. Default: bitnami_airflow
AIRFLOW_DATABASE_USERNAME
: Database user that Airflow will use to connect with the database. Default: bn_airflow
AIRFLOW_DATABASE_PASSWORD
: Database password that Airflow will use to connect with the database. No defaults.AIRFLOW_DATABASE_USE_SSL
: Set to yes if the database is using SSL. Default: no
AIRFLOW_REDIS_USE_SSL
: Set to yes if Redis(R) uses SSL. Default: no
REDIS_HOST
: Hostname for Redis(R) server. Default: redis
REDIS_PORT_NUMBER
: Port used by Redis(R) server. Default: 6379
REDIS_USER
: User that Airflow will use to connect with Redis(R). No defaults.REDIS_PASSWORD
: Password that Airflow will use to connect with Redis(R). No defaults.REDIS_DATABASE
: Database number for Redis(R) server. Default: 1
AIRFLOW_LDAP_ENABLE
: Enable LDAP authentication. Default: no
AIRFLOW_LDAP_URI
: LDAP server URI. No defaults.AIRFLOW_LDAP_SEARCH
: LDAP search base. No defaults.AIRFLOW_LDAP_UID_FIELD
: LDAP field used for uid. No defaults.AIRFLOW_LDAP_BIND_USER
: LDAP user name. No defaults.AIRFLOW_LDAP_BIND_PASSWORD
: LDAP user password. No defaults.AIRFLOW_USER_REGISTRATION
: User self registration. Default: True
AIRFLOW_USER_REGISTRATION_ROLE
: Role for the created user. No defaults.AIRFLOW_LDAP_ROLES_MAPPING
: Mapping from LDAP DN to a list of Airflow roles. No defaults.AIRFLOW_LDAP_ROLES_SYNC_AT_LOGIN
: Replace ALL the user's roles each login, or only on registration. Default: True
AIRFLOW_LDAP_USE_TLS
: Use LDAP SSL. Defaults: False.AIRFLOW_LDAP_ALLOW_SELF_SIGNED
: Allow self signed certicates in LDAP ssl. Default: True
AIRFLOW_LDAP_TLS_CA_CERTIFICATE
: File that store the CA for LDAP ssl. No defaults.In addition to the previous environment variables, all the parameters from the configuration file can be overwritten by using environment variables with this format:
AIRFLOW__{SECTION}__{KEY}
. Note the double underscores.
version: '2'
services:
airflow:
image: bitnami/airflow:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_PASSWORD=bitnami123
- AIRFLOW_USERNAME=user
- [email protected]
$ docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e [email protected] \
bitnami/airflow:latest
To configure Airflow to send email using SMTP you can set the following environment variables:
AIRFLOW__SMTP__SMTP_HOST
: Host for outgoing SMTP email. Default: localhost
AIRFLOW__SMTP__SMTP_PORT
: Port for outgoing SMTP email. Default: 25
AIRFLOW__SMTP__SMTP_STARTTLS
: To use TLS communication. Default: True
AIRFLOW__SMTP__SMTP_SSL
: To use SSL communication. Default: False
AIRFLOW__SMTP__SMTP_USER
: User of SMTP used for authentication (likely email). No defaults.AIRFLOW__SMTP__SMTP_PASSWORD
: Password for SMTP. No defaults.AIRFLOW__SMTP__SMTP_MAIL_FROM
: To modify the "from email address". Default: [email protected]
This would be an example of SMTP configuration using a GMail account:
airflow:
image: bitnami/airflow:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_PASSWORD=bitnami
- AIRFLOW_USERNAME=user
- [email protected]
- [email protected]
- [email protected]
- AIRFLOW__SMTP__SMTP_PASSWORD=your_password
- AIRFLOW__SMTP__SMTP_PORT=587
ports:
- '8080:8080'
$ docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e [email protected] \
-e [email protected] \
-e [email protected] \
-e AIRFLOW__SMTP__SMTP_PASSWORD=your_password \
-e AIRFLOW__SMTP__SMTP_PORT=587 \
bitnami/airflow:latest
We'd love for you to contribute to this container. You can request new features by creating an issue, or submit a pull request with your contribution.
If you encountered a problem running this container, you can file an issue. For us to provide better support, be sure to include the following information in your issue:
$ docker version
)$ docker info
Copyright © 2022 Bitnami
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.