Yet another cron alternative with a Web UI, but with much more capabilities. It aims to solve greater problems.
Dagu is a powerful Cron alternative that comes with a Web UI. It allows you to define dependencies between commands as a Directed Acyclic Graph (DAG) in a declarative YAML format. Dagu simplifies the management and execution of complex workflows. It natively supports running Docker containers, making HTTP requests, and executing commands over SSH.
It shows the real-time status, logs, and workflow configurations. You can edit workflow configurations on a browser.
You can switch to the vertical graph with the button on the top right corner.
It shows all workflows and the real-time status.
It greps given text across all workflow definitions.
It shows past execution results and logs.
It shows the detail log and standard output of each execution and step.
You can install Dagu quickly using Homebrew or by downloading the latest binary from the Releases page on GitHub.
curl -L https://raw.githubusercontent.com/yohamta/dagu/main/scripts/downloader.sh | bash
Download the latest binary from the Releases page and place it in your $PATH
(e.g. /usr/local/bin
).
brew install yohamta/tap/dagu
Upgrade to the latest version:
brew upgrade yohamta/tap/dagu
docker run \
--rm \
-p 8080:8080 \
-v $HOME/.dagu/dags:/home/dagu/.dagu/dags \
-v $HOME/.dagu/data:/home/dagu/.dagu/data \
-v $HOME/.dagu/logs:/home/dagu/.dagu/logs \
ghcr.io/dagu-dev/dagu:latest dagu start-all
Start the server and scheduler with the command dagu start-all
and browse to http://127.0.0.1:8080
to explore the Web UI.
Navigate to the DAG List page by clicking the menu in the left panel of the Web UI. Then create a DAG by clicking the NEW
button at the top of the page. Enter example
in the dialog.
Note: DAG (YAML) files will be placed in ~/.dagu/dags
by default. See Configuration Options for more details.
Go to the SPEC
Tab and hit the Edit
button. Copy & Paste the following example and click the Save
button.
Example:
schedule: "* * * * *" # Run the DAG every minute
steps:
- name: s1
command: echo Hello Dagu
- name: s2
command: echo done!
depends:
- s1
You can execute the example by pressing the Start
button. You can see "Hello Dagu" in the log page in the Web UI.
# Runs the DAG
dagu start [--params=<params>] <file>
# Displays the current status of the DAG
dagu status <file>
# Re-runs the specified DAG run
dagu retry --req=<request-id> <file>
# Stops the DAG execution
dagu stop <file>
# Restarts the current running DAG
dagu restart <file>
# Dry-runs the DAG
dagu dry [--params=<params>] <file>
# Launches both the web UI server and scheduler process
dagu start-all [--host=<host>] [--port=<port>] [--dags=<path to directory>]
# Launches the Dagu web UI server
dagu server [--host=<host>] [--port=<port>] [--dags=<path to directory>]
# Starts the scheduler process
dagu scheduler [--dags=<path to directory>]
# Shows the current binary version
dagu version
The easiest way to make sure the process is always running on your system is to create the script below and execute it every minute using cron (you don't need root
account in this way):
#!/bin/bash
process="dagu start-all"
command="/usr/bin/dagu start-all"
if ps ax | grep -v grep | grep "$process" > /dev/null
then
exit
else
$command &
fi
exit
This example workflow showcases a data pipeline typically implemented in DevOps and Data Engineering scenarios. It demonstrates an end-to-end data processing cycle starting from data acquisition and cleansing to transformation, loading, analysis, reporting, and ultimately, cleanup.
The YAML code below represents this workflow:
# Environment variables used throughout the pipeline
env:
- DATA_DIR: /data
- SCRIPT_DIR: /scripts
- LOG_DIR: /log
# ... other variables can be added here
# Handlers to manage errors and cleanup after execution
handlerOn:
failure:
command: "echo error"
exit:
command: "echo clean up"
# The schedule for the workflow execution in cron format
# This schedule runs the workflow daily at 12:00 AM
schedule: "0 0 * * *"
steps:
# Step 1: Pull the latest data from a data source
- name: pull_data
command: "sh"
script: |
echo `date '+%Y-%m-%d'`
output: DATE
# Step 2: Cleanse and prepare the data
- name: cleanse_data
command: echo cleansing ${DATA_DIR}/${DATE}.csv
depends:
- pull_data
# Step 3: Transform the data
- name: transform_data
command: echo transforming ${DATA_DIR}/${DATE}_clean.csv
depends:
- cleanse_data
# Parallel Step 1: Load the data into a database
- name: load_data
command: echo loading ${DATA_DIR}/${DATE}_transformed.csv
depends:
- transform_data
# Parallel Step 2: Generate a statistical report
- name: generate_report
command: echo generating report ${DATA_DIR}/${DATE}_transformed.csv
depends:
- transform_data
# Step 4: Run some analytics
- name: run_analytics
command: echo running analytics ${DATA_DIR}/${DATE}_transformed.csv
depends:
- load_data
# Step 5: Send an email report
- name: send_report
command: echo sending email ${DATA_DIR}/${DATE}_analytics.csv
depends:
- run_analytics
- generate_report
# Step 6: Cleanup temporary files
- name: cleanup
command: echo removing ${DATE}*.csv
depends:
- send_report
In legacy systems, job dependencies are often complex and implicit, making it challenging to manage and maintain workflows. As the number of cron jobs on a server grows into the hundreds, keeping track of these dependencies and determining which jobs to rerun on failure becomes increasingly difficult. Additionally, viewing logs and manually rerunning shell scripts one by one via SSH can be a tedious and time-consuming process.
Dagu addresses these pain points by providing a user-friendly solution for explicitly defining and visualizing workflows. With its intuitive web UI, Dagu simplifies the management of workflows, enabling users to easily check dependencies, monitor execution status, view logs, and control job execution with just a few clicks.
While there are several existing workflow schedulers like Airflow, many of them require users to define workflows using a programming language such as Python. This can be problematic for legacy systems that have been in operation for an extended period and already have complex jobs written in languages like Perl or Shell Script.
Introducing another layer of abstraction and complexity on top of these existing codebases can hinder maintainability and increase the learning curve for team members. Dagu differentiates itself by being easy to use, self-contained, and requiring no coding. This makes Dagu particularly suitable for smaller projects or teams looking to introduce workflow orchestration without the overhead of a full-fledged scheduling system.
Dagu is designed as a standalone command-line tool that leverages the local file system for data storage, eliminating the need for a separate database management system or cloud service. This self-contained nature simplifies installation and setup, making it easy to get started with Dagu. By combining a user-friendly web interface, a declarative YAML format, and compatibility with existing programs, Dagu provides an efficient and accessible solution for managing and orchestrating workflows in a variety of scenarios.
Feel free to contribute in any way you want! Share ideas, questions, submit issues, and create pull requests. Check out our Contribution Guide for help getting started.
We welcome any and all contributions!
This project is licensed under the GNU GPLv3.
Join our Discord community to ask questions, request features, and share your ideas.