Dagu

Dagu is a lightweight, self‑hosted workflow engine that brings the clarity of directed acyclic graphs (DAGs) to legacy cron‑based pipelines. It exposes a single binary with no external dependencies, making it trivial to drop into an existing environment. Workflows are expressed in declarative YAML, which can reference shell scripts, SSH commands, or Docker images without any code change. The engine provides a built‑in web UI for visualizing the DAG, inspecting logs, and controlling execution flow, all while remaining fully operational without a database or cloud service.

Architecture

• Core Language: Written in Go, compiled to a single static binary.
• Execution Model: A master process orchestrates tasks and delegates them to worker processes via an embedded message queue. Workers can run locally or be distributed across multiple hosts for high‑availability (HA) mode.
• Storage: Uses a lightweight embedded key/value store (BoltDB or similar) for state persistence, eliminating the need for an external DBMS.
• Networking: Exposes a RESTful HTTP API and WebSocket endpoint for the UI, while worker communication occurs over local IPC or TCP.
• Container Support: Each step can be executed as a Docker container, enabling isolated environments and reproducible builds.

Core Capabilities

• Declarative YAML: Define steps, dependencies, and cron schedules in a single file.
• Step Types: Shell commands, SSH remote execution, Docker image runs, and nested workflows.
• Error Handling: Built‑in retry logic, failure notifications (email, Slack via webhooks), and automatic task cancellation.
• API Surface: Exposes endpoints for starting, stopping, querying status, and retrieving logs; can be consumed by external tooling or CI pipelines.
• Web UI: Interactive DAG graph, real‑time log streaming, and manual controls for rerunning or pausing jobs.

Deployment & Infrastructure

• Self‑Hosting: Requires only the binary and a writable directory for state.
• Containerization: Official Docker image available on ghcr.io; volumes can be mounted for persistence.
• Scalability: HA mode distributes tasks across multiple workers; the queue system can be backed by Redis or NATS for larger clusters.
• Resource Footprint: Minimal CPU/memory usage; suitable for edge or on‑premise deployments where a full Airflow stack would be overkill.

Integration & Extensibility

• Plugins: Supports custom executors via Go plugins, allowing integration with proprietary runtimes or cloud APIs.
• Webhooks: Trigger external services on task start/finish/failure; can be wired into monitoring or alerting systems.
• CLI Hooks: The command‑line interface can be scripted, enabling automated deployment of new DAGs or dynamic configuration changes.
• Nested Workflows: Reuse common sub‑DAGs across projects, promoting DRY principles without code duplication.

Developer Experience

• Configuration: YAML files are human‑readable; no templating or code generation required.
• Documentation: Comprehensive docs with examples, API reference, and troubleshooting guides; a growing community on Discord for real‑time support.
• Extensibility: The Go codebase is open source, making it straightforward to fork and add features such as custom logging backends or new executor types.
• Testing: Lightweight unit tests and integration examples make it easy to validate workflow logic locally before deployment.

Use Cases

• Legacy Cron Migration: Replace hundreds of crontab entries with a single DAG that visualizes dependencies and provides retry logic.
• Data Pipelines: Orchestrate ETL jobs that run shell scripts or Docker containers, with scheduled runs and failure notifications.
• DevOps Automation: Coordinate multi‑step deployment scripts, including remote SSH commands and containerized tasks.
• CI/CD Hooks: Trigger Dagu workflows from GitHub Actions or other CI systems via the REST API, enabling complex post‑build pipelines.

Advantages

• Zero Dependencies: One binary, no external database or message broker needed for small deployments.
• Performance & Simplicity: Go’s concurrency model delivers low latency task scheduling while keeping the codebase compact.
• Licensing: MIT‑licensed, allowing unrestricted use in commercial or internal projects without subscription costs.
• Flexibility: YAML definition keeps it language‑agnostic; developers can continue using existing shell scripts or Docker images without refactoring.
• Community & Support: Active GitHub repository, Discord channel, and frequent releases ensure quick issue resolution and feature updates.

In short, Dagu offers a developer‑friendly, lightweight alternative to heavyweight schedulers like Airflow, while still delivering the full benefits of DAG visualization, error handling, and web‑based control—all in a single, easily deployable binary.