Skip to content

Commit

Permalink
add tekton cd integration task
Browse files Browse the repository at this point in the history
  • Loading branch information
kannon92 committed Dec 20, 2024
1 parent 12cc376 commit ff05459
Show file tree
Hide file tree
Showing 3 changed files with 190 additions and 168 deletions.
2 changes: 1 addition & 1 deletion site/content/en/docs/tasks/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ As a batch user, you can learn how to:
- [Run a Kueue managed plain Pod](run/plain_pods).
- [Run a Kueue managed JobSet](run/jobsets).
- [Submit jobs to MultiKueue](run/multikueue).
- [Run an argo workflow](run/argo_workload)
- [Run a tekton cd pipeline](run/tektoncd)

### Serving user

Expand Down
167 changes: 0 additions & 167 deletions site/content/en/docs/tasks/run/argo_workload.md

This file was deleted.

189 changes: 189 additions & 0 deletions site/content/en/docs/tasks/run/tektoncd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
---
title: "Run An Argo Workflow"
date: 2024-01-03
weight: 7
description: >
Integrate Kueue with TektonCD Pipelines.
---

This page shows how to leverage Kueue's scheduling and resource management capabilities when running [TektonCD pipelines](https://tekton.dev/docs/).

This guide is for [batch users](/docs/tasks#batch-user) that have a basic understanding of Kueue. For more information, see [Kueue's overview](/docs/overview).

As of the writing of this doc, Kueue doesn't support TektonCD Pipelines [Pipelines](https://github.com/tektoncd/pipeline/blob/main/docs/README.md) resources directly,
but you can take advantage of the ability for Kueue to [manage plain pods](/docs/tasks/run_plain_pods) to integrate them.

## Before you begin

1. By default, the integration for `v1/pod` is not enabled.
Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version)
and enable the `pod` integration.

The pod integration for TektonCD Pipelines could look like:

```yaml
apiVersion: config.kueue.x-k8s.io/v1beta1
kind: Configuration
integrations:
frameworks:
- "pod"
podOptions:
# You can change namespaceSelector to define in which
# namespaces kueue will manage the tektoncd pods.
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: [ kube-system, kueue-system ]
# Tekton pipelines uses the app.kubernetes.io/managed-by label to
# keep track of pods it manages. We will use that as a hint for Kueue
# to find Tekton pods.
podSelector:
matchExpressions:
- key: app.kubernetes.io/managed-by
operator: In
values: [ "tekton-pipelines" ]
```
2. Pods that belong to other API resources managed by Kueue are excluded from being queued by `pod` integration.
For example, pods managed by `batch/v1.Job` won't be managed by `pod` integration.

3. Check [Administer cluster quotas](/docs/tasks/administer_cluster_quotas) for details on the initial Kueue setup.

4. Your cluster has tekton pipelines [installed]()

## Motivation

Let's say you are providing a build platform for a software delivery service.
During a busy time of a year, you have many engineers creating build pipelines and your cluster is at capacity due to the number of builds created.
One can use Kueue to limit the number of pods that are created by using the pod integration.
Kueue provides quota limits on core resources so one could create a clusterqueue that limits the number of pods.

```yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: "cluster-queue"
spec:
namespaceSelector: {} # match all.
resourceGroups:
- coveredResources: ["pods"]
flavors:
- name: "default-flavor"
resources:
- name: "pods"
nominalQuota: 2
```

## Tekton Background

Tekton has the concept of [Pipelines](https://tekton.dev/vault/pipelines-v0.59.x-lts/pipelines/), [Tasks](https://tekton.dev/vault/pipelines-v0.59.x-lts/tasks/) and [PipelineRun](https://tekton.dev/vault/pipelines-v0.59.x-lts/pipelineruns/).

A pipeline consists of tasks. Tasks and pipelines must be created before running a pipeline.

A PipelineRun runs the pipeline.

A TaskRun runs a single task. PipelineRuns will reuse TaskRuns to run each task in a pipeline.

### Tekton Defintions

As a simple example, we will define two tasks named sleep and hello:

Tasks:

```yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: sleep
spec:
steps:
- name: echo
image: alpine
script: |
#!/bin/sh
sleep 100
```

```yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: hello
spec:
params:
- name: username
type: string
steps:
- name: hello
image: ubuntu
script: |
#!/bin/bash
echo "Hello $(params.username)!"
```

A pipeline composes these tasks.

```yaml
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: kueue-test
spec:
params:
- name: username
type: string
tasks:
- name: sleep
taskRef:
name: sleep
- name: hello
runAfter:
- sleep
taskRef:
name: hello
params:
- name: username
value: $(params.username)
```

## a. Targeting a single LocalQueue

If you want every task to target a single [local queue](/docs/concepts/local_queue),
it should be specified in the `metadata.label` section of the PipelineRun configuration.

```yaml
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: kueue-test
labels:
kueue.x-k8s.io/queue-name: my-local-queue
spec:
pipelineRef:
name: kueue-test
params:
- name: username
value: "Tekton"
```

This will inject the kueue label on every pod of the pipeline.

## c. How it works

Tekton pipelines manages its own pods, and only creates those pods once task nodes are
ready to execute. When configured, Kueue attaches an admission webhook that monitors for
pods created by Tekton pipelines. When it finds a newly created pod, it will add an
entry to the `spec.schedulingGates` parameter of the pod, preventing the Kubernetes scheduler
from assigning a node to the pod. It also creates a corresponding `Workload` resource to
track the resource requirements. Once the Workload meets all admission criteria,
Kueue will remove the scheduling gate and allow the pod to proceed.

Once the pod is scheduled and runs successfully, Tekton will register the task complete and continue with it's processing.

## d. Limitations

- Kueue will only manage pods created by Tekton.
- Each pod in a Workflow will create a new Workload resource and must wait for admission by Kueue.
- There is no way to ensure that a Workflow will complete before it is started. If one step of a multi-step Workflow does not have
available quota, Tekton pipelines will run all previous steps and then wait for quota to become available.

0 comments on commit ff05459

Please sign in to comment.