Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/
This is a repository for building Docker container of Apache Airflow (incubating).
Image | Pulls | Tags |
---|---|---|
abhioncbr/docker-airflow | tags |
XX.YY.ZZ
standalone
(simple container for exploration purpose, based on sqlite as airflow metadata db & SequentialExecutor ) or prod
(single node based, LocalExecutor amd mysql as airflow metadata db) and cluster
(for distributed production long run use-cases, container runs as either server
or worker
)entrypoint-s3.sh
)superuser
privilege.airflow-version
as a build-arg
. docker build -t abhioncbr/docker-airflow:$IMAGE_VERSION --build-arg AIRFLOW_VERSION=$AIRFLOW_VERSION
--build-arg AIRFLOW_PATCH_VERSION=$AIRFLOW_PATCH_VERSION -f ~/docker-airflow/docker-files/DockerFile .
Search abhioncbr/docker-airflow Image on docker-hub
General commands -
starting airflow image as a airflow-standalone
container in a standalone mode-
docker run --net=host -p 2222:2222 --name=airflow-standalone abhioncbr/airflow-XX.YY.ZZ -m=standalone &
Starting airflow image as a airflow-server
container in a cluster mode-
docker run --net=host -p 2222:2222 -p 6379:6379 --name=airflow-server \
abhioncbr/airflow-XX.YY.ZZ -m=cluster -t=server -d=mysql://user:password@host:3306/db-name &
Starting airflow image as a airflow-worker
container in a cluster mode-
docker run --net=host -p 5555:5555 -p 8739:8739 --name=airflow-worker \
abhioncbr/airflow-XX.YY.ZZ -m=cluster -t=worker -d=mysql://user:password@host:3306/db-name -r=redis://<airflow-server-host>:6379/0 &
In Mac using docker for mac -
Standalone Mode - starting airflow image in a standalone mode & mounting dags, code-artifacts & logs folder to host machine -
docker run -p 2222:2222 --name=airflow-standalone \
-v ~/airflow-data/code-artifacts:/code-artifacts \
-v ~/airflow-data/logs:/usr/local/airflow/logs \
-v ~/airflow-data/dags:/usr/local/airflow/dags \
abhioncbr/airflow-XX.YY.ZZ -m=standalone &
Cluster Mode
starting airflow image as a server container & mounting dags, code-artifacts & logs folder to host machine -
docker run -p 2222:2222 -p 6379:6379 --name=airflow-server \
-v ~/airflow-data/code-artifacts:/code-artifacts \
-v ~/airflow-data/logs:/usr/local/airflow/logs \
-v ~/airflow-data/dags:/usr/local/airflow/dags \
abhioncbr/airflow-XX.YY.ZZ \
-m=cluster -t=server -d=mysql://user:[email protected]:3306:3306/<airflow-db-name> &
starting airflow image as a worker container & mounting dags, code-artifacts & logs folder to host machine -
docker run -p 5555:5555 -p 8739:8739 --name=airflow-worker \
-v ~/airflow-data/code-artifacts:/code-artifacts \
-v ~/airflow-data/logs:/usr/local/airflow/logs \
-v ~/airflow-data/dags:/usr/local/airflow/dags \
abhioncbr/airflow-XX.YY.ZZ \
-m=cluster -t=worker -d=mysql://user:[email protected]:3306:3306/<airflow-db-name> -r=redis://host.docker.internal:6379/0 &
server
mode for serving the UI of the airflow, redis for celery task & scheduler.worker
mode for executing tasks using celery executor.