Introducing Kontroler: A Kubernetes Scheduling Engine for DAGs

What is Kontroler?

Kontroler is an open-source Kubernetes scheduling engine designed to simplify the management of Directed Acyclic Graphs (DAGs). With Kontroler, you can define workflows in YAML, run tasks as containerized jobs, and even visualize them through an optional web-based UI. Whether you need to execute workflows on a schedule or respond to real-time events, Kontroler offers a flexible, Kubernetes-native solution.

That said, I must caveat that Kontroler is currently in its alpha state. It is ideal for experimentation and prototyping but may not yet be ready for production use. As a side project, Kontroler does not come with support, so use it in production at your own risk!

If you'd like to check out the GitHub repository, here it is: https://github.com/GreedyKomodoDragon/Kontroler

Why use Kontroler?

Managing workflows on Kubernetes can be challenging. While Kubernetes offers powerful primitives like CronJobs and Jobs, coordinating complex workflows with dependencies, retries, and dynamic triggers often requires custom scripts or external tools. Kontroler simplifies this process by providing a Kubernetes-native solution to define, schedule, and manage workflows as Directed Acyclic Graphs (DAGs).

Here’s what makes Kontroler stand out:

1. Gitops DAG Management with YAML

Kontroler enables you to define your entire workflow—including tasks, dependencies, retry logic, and conditional execution—using straightforward YAML files. This approach keeps your workflows declarative, version-controlled, and easy to share across teams.

Wrap them in a Helm chart and use an automated deployment system such as ArgoCD or FluxCD, and you’ve got your DAGs managed within a GitOps workflow.

2. Supports Event-Driven and Scheduled Workflows

Whether you need workflows to run on a regular schedule (like CronJobs) or react to real-time triggers (such as a message from a queue), Kontroler has you covered. It seamlessly integrates both scheduling paradigms, making it versatile for a wide range of use cases.

Kontroler operates using DagRuns (a Custom Resource Definition, or CRD, that Kontroler can monitor). To execute a DAG run, you simply create a DagRun, and Kontroler manages the execution. By exposing this functionality outside the controller, Kontroler allows seamless integration with external systems. See an example of a DagRun below:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DagRun
metadata:
  labels:
    app.kubernetes.io/name: dagrun
    app.kubernetes.io/instance: dagrun-sample
  name: dagrun-sample
spec:
  dagName: dag-sample
  parameters:
    - name: first
      fromSecret: secret-name-new
    - name: second
      value: value_new

You simply provide the name of the DAG, optionally including any parameters. Parameters will use their default values if none are specified.

3. Integrates Seamlessly with Containers

As a Kubernetes-native application, Kontroler leverages the platform’s advantages. From simple scripts to complex applications, Kontroler’s container-first design ensures that any workload can be easily integrated into a DAG.

To further reduce duplication of effort, we are adding support for Tasks as CRDs.

4. Offers Optional UI for Better Visibility

While YAML is excellent for infrastructure as code, visualizing workflows and tracking their execution can significantly simplify debugging and optimization. Kontroler offers an optional web-based UI for creating and monitoring DAGs, providing teams with an intuitive way to manage workflows.

In the screenshot above it shows you the:

• Connections between tasks
• In a DagRun it will show the status
• Pod Outcomes if you click on the network

Under the hood, this functionality is powered by vis-network, an open-source project for creating graphs. You can find their GitHub repository here. Initially, we developed our own graphing tool but found it difficult to scale and challenging to make complex graphs readable. As a result, we adopted this open-source solution.

5. Flexible Pod Templates for Advanced Configurations

Kontroler’s pod templates let you customize your tasks with advanced Kubernetes features like secrets, PVCs, node affinity, security contexts, and more. This gives you fine-grained control over how tasks are executed, ensuring compatibility with your cluster's specific requirements.

Flexibility of Datastores

One of the key challenges in managing workflows is ensuring that the underlying infrastructure can scale with your needs. Kontroler offers flexible database support and logging capabilities to make it easier for teams to manage DAGs and capture detailed execution logs. Whether you're working with a lightweight setup or a more robust production environment, Kontroler has options that can fit your needs.

SQLite (Default Small Scale)

For smaller setups or during development, SQLite is the default database used by Kontroler. It’s simple, lightweight, and doesn’t require much configuration. If you’re just getting started or running Kontroler for testing purposes, SQLite is a convenient choice as it works "out-of-the-box" with no additional database setup.

If you prefer to use SQLite, you don’t need to make any additional changes to the deployment. Just follow the installation instructions, and Kontroler will handle the rest.

PostgreSQL (For Production/Large Scale)

For larger, production-grade deployments, PostgreSQL is the recommended database for storing DAGs and DagRuns. Kontroler uses PostgreSQL to provide higher performance, scalability, and reliability needed for handling large volumes of workflow data. Kontroler is compatible with PostgreSQL 16 and above

Log Collection

For many Kubernetes users, centralized log storage is a crucial requirement for observability and troubleshooting. Kontroler integrates seamlessly with Amazon S3 for log storage, making it easy to collect, store, and analyze logs from DAG executions. This integration ensures that you can capture detailed logs, keep them long-term for auditing, and use them for post-mortem analysis when something goes wrong.

We are testing with MinIO to avoid costs, but under the hood, it uses the AWS S3 API and should be able to detect your IAM attributes to grant access to your S3 buckets.

Final Remarks

We hope you’ll check out Kontroler! You can find it here: https://github.com/GreedyKomodoDragon/Kontroler.

Although still in alpha, Kontroler has significant potential for teams looking to streamline their Kubernetes operations. As it evolves, Kontroler aims to become a powerful tool for orchestrating workflows in Kubernetes, delivering the reliability and scalability teams need to automate complex processes with ease.

Kontroler is constantly improving, so keep an eye out for future updates! If you’d like to contribute to the project, we welcome all forms of contributions.

The above is the detailed content of Introducing Kontroler: A Kubernetes Scheduling Engine for DAGs. For more information, please follow other related articles on the PHP Chinese website!