Skip to content

Commit

Permalink
Support KubevirtIpamController plugin (kubevirt#1808)
Browse files Browse the repository at this point in the history
* Support KubevirtIpamController plugin

Signed-off-by: Or Shoval <oshoval@redhat.com>

* cluster: Add memory

Without it, we don't have memory to create some virtual machines
(saw it for example on KSD lane).

Signed-off-by: Or Shoval <oshoval@redhat.com>

* components: Refactor script to allow robust usage

Since we introduce a non kubevirtci cluster,
refactor the script to allow code reuse while using
both types of clusters.

All current lanes are agnostic for the change.

Signed-off-by: Or Shoval <oshoval@redhat.com>

* git actions: Add kubevirt ipam controller e2e lane

Signed-off-by: Or Shoval <oshoval@redhat.com>

---------

Signed-off-by: Or Shoval <oshoval@redhat.com>
  • Loading branch information
oshoval authored Jun 24, 2024
1 parent 0634b4f commit 587dfb3
Show file tree
Hide file tree
Showing 19 changed files with 918 additions and 69 deletions.
23 changes: 23 additions & 0 deletions .github/workflows/kubevirt-ipam-controller.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Kubevirt IPAM controller Tests
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]

jobs:
e2e:
runs-on: ubuntu-latest
steps:

- uses: actions/checkout@v3

- name: Set up Go
uses: actions/setup-go@v3
with:
go-version-file: 'go.mod'

- name: Run e2e tests
env:
KIND_ALLOW_SYSTEM_WRITES: true
run: automation/check-patch.e2e-kubevirt-ipam-controller-functests.sh
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ prom-rules-verify:

cluster-up:
./cluster/up.sh
./cluster/cert-manager-install.sh

cluster-down:
./cluster/down.sh
Expand Down Expand Up @@ -160,6 +161,7 @@ gen-manifests: manifest-templator
MACVTAP_CNI_IMAGE=$(MACVTAP_CNI_IMAGE) \
MULTUS_DYNAMIC_NETWORKS_CONTROLLER_IMAGE=$(MULTUS_DYNAMIC_NETWORKS_CONTROLLER_IMAGE) \
KUBE_SECONDARY_DNS_IMAGE=$(KUBE_SECONDARY_DNS_IMAGE) \
KUBEVIRT_IPAM_CONTROLLER_IMAGE=$(KUBEVIRT_IPAM_CONTROLLER_IMAGE) \
CORE_DNS_IMAGE=$(CORE_DNS_IMAGE) \
KUBE_RBAC_PROXY_IMAGE=$(KUBE_RBAC_PROXY_IMAGE) \
./hack/generate-manifests.sh
Expand Down
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ spec:
ovs: {}
macvtap: {}
kubeSecondaryDNS: {}
kubevirtIpamController: {}
imagePullPolicy: Always
```
Expand Down Expand Up @@ -180,6 +181,29 @@ Additionally, container image used to deliver this plugin can be set using
`KUBE_SECONDARY_DNS_IMAGE` environment variable in operator
deployment manifest.

## kubevirtIpamController

[This controller](https://github.com/maiqueb/kubevirt-ipam-claims)
allows to support IPAM for user defined networks.

```yaml
apiVersion: networkaddonsoperator.network.kubevirt.io/v1
kind: NetworkAddonsConfig
metadata:
name: cluster
spec:
multus: {}
kubevirtIpamController: {}
```

Additionally, container image used to deliver this plugin can be set using
`KUBEVIRT_IPAM_CONTROLLER_IMAGE` environment variable in operator
deployment manifest.

Note: This component requires certificates mounted on the controller pods for the webhook to work.
On non OpenShift clusters, the user should manually install a certificate library (e.g. [cert-manager](https://github.com/cert-manager/cert-manager)).
It is done for convenience as part of the helper scripts.

## Image Pull Policy

Administrator can specify [image pull policy](https://kubernetes.io/docs/concepts/containers/images/)
Expand Down
55 changes: 55 additions & 0 deletions automation/check-patch.e2e-kubevirt-ipam-controller-functests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/usr/bin/env bash

set -xeuE

# automation/check-patch.e2e-kubevirt-ipam-controller-functests.sh

GITHUB_ACTIONS=${GITHUB_ACTIONS:-false}

teardown() {
cd ${TMP_COMPONENT_PATH}
make cluster-down || true
rm -rf "${TMP_COMPONENT_PATH}"
}

main() {
if [ "$GITHUB_ACTIONS" == "true" ]; then
ARCH="amd64"
OS_TYPE="linux"
kubevirt_version="$(curl -L https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)"
kubevirt_release_url="https://github.com/kubevirt/kubevirt/releases/download/${kubevirt_version}"
cli_name="virtctl-${kubevirt_version}-${OS_TYPE}-${ARCH}"
curl -LO "${kubevirt_release_url}/${cli_name}"
mv ${cli_name} virtctl
chmod +x virtctl
mv virtctl /usr/local/bin
fi

# Setup CNAO and artifacts temp directory
source automation/check-patch.setup.sh
cd ${TMP_PROJECT_PATH}

export USE_KUBEVIRTCI=false
COMPONENT="kubevirt-ipam-controller" source automation/components-functests.setup.sh

cd ${TMP_COMPONENT_PATH}
export KIND_ARGS="-ic -i6 -mne"
make cluster-up
export KUBECONFIG=${TMP_COMPONENT_PATH}/.output/kubeconfig

trap teardown EXIT

cd ${TMP_PROJECT_PATH}
export KUBEVIRT_PROVIDER=external
export DEV_IMAGE_REGISTRY=localhost:5000
./cluster/cert-manager-install.sh
deploy_cnao
deploy_cnao_cr
./hack/deploy-kubevirt.sh

cd ${TMP_COMPONENT_PATH}
echo "Run kubevirt-ipam-controller functional tests"
make test-e2e
}

[[ "${BASH_SOURCE[0]}" == "$0" ]] && main "$@"
61 changes: 43 additions & 18 deletions automation/components-functests.setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,27 @@ source hack/components/git-utils.sh
source hack/components/yaml-utils.sh
source cluster/cluster.sh

# Spin up Kubernetes cluster
make cluster-down cluster-up
USE_KUBEVIRTCI=${USE_KUBEVIRTCI:-"true"}

# Export .kubeconfig full path, so it will be possible
# to use 'kubectl' directly from the component directory path
export KUBECONFIG=$(cluster::kubeconfig)
export KUBECONFIG=${KUBECONFIG:-$(cluster::kubeconfig)}

# Deploy CNAO latest changes
make cluster-operator-push
make cluster-operator-install
function deploy_cluster {
# Spin up Kubernetes cluster
export KUBEVIRT_MEMORY_SIZE=9216M
make cluster-down cluster-up
}

# Test kubemacpool with restricted
if [ "$COMPONENT" == "kubemacpool" ]; then
function deploy_cnao {
# Deploy CNAO latest changes
make cluster-operator-push
make cluster-operator-install
}

function patch_restricted_namespace {
# Test kubemacpool with restricted
if [ "$COMPONENT" == "kubemacpool" ]; then
cluster/kubectl.sh apply -f - <<EOF
apiVersion: v1
kind: Namespace
Expand All @@ -42,32 +50,42 @@ metadata:
labels:
pod-security.kubernetes.io/enforce: restricted
EOF
fi
fi
}

# Deploy all network addons components with CNAO
cat <<EOF | cluster/kubectl.sh apply -f -
function deploy_cnao_cr {
# Deploy all network addons components with CNAO

cat <<EOF > cr.yaml
apiVersion: networkaddonsoperator.network.kubevirt.io/v1
kind: NetworkAddonsConfig
metadata:
name: cluster
spec:
multus: {}
multusDynamicNetworks: {}
linuxBridge: {}
kubeMacPool:
rangeStart: "02:00:00:00:00:00"
rangeEnd: "02:00:00:00:00:0F"
ovs: {}
macvtap: {}
kubeSecondaryDNS: {}
kubevirtIpamController: {}
imagePullPolicy: Always
EOF

if [[ ! $(cluster/kubectl.sh wait networkaddonsconfig cluster --for condition=Available --timeout=13m) ]]; then
echo "Failed to wait for CNAO CR to be ready"
cluster/kubectl.sh get networkaddonsconfig -o custom-columns="":.status.conditions[*].message
exit 1
fi
if [[ $USE_KUBEVIRTCI == true ]]; then
echo " multus: {}" >> cr.yaml
echo " multusDynamicNetworks: {}" >> cr.yaml
fi

cluster/kubectl.sh apply -f cr.yaml

if [[ ! $(cluster/kubectl.sh wait networkaddonsconfig cluster --for condition=Available --timeout=13m) ]]; then
echo "Failed to wait for CNAO CR to be ready"
cluster/kubectl.sh get networkaddonsconfig -o custom-columns="":.status.conditions[*].message
exit 1
fi
}

# Clone component repository
component_url=$(yaml-utils::get_component_url ${COMPONENT})
Expand All @@ -80,3 +98,10 @@ component_path=${component_temp_dir}/${component_repo}
git-utils::fetch_component ${component_path} ${component_url} ${component_commit}

export TMP_COMPONENT_PATH=${component_path}

if [[ $USE_KUBEVIRTCI == true ]]; then
deploy_cluster
deploy_cnao
patch_restricted_namespace
deploy_cnao_cr
fi
26 changes: 26 additions & 0 deletions cluster/cert-manager-install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash
#
# Copyright 2024 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -ex

export DEPLOY_CERT_MANAGER=${DEPLOY_CERT_MANAGER:-true}

if [[ $DEPLOY_CERT_MANAGER == true ]]; then
CERT_MANAGER_VERSION="v1.14.4"
echo "Installing cert-manager..."
manifest="https://github.com/cert-manager/cert-manager/releases/download/${CERT_MANAGER_VERSION}/cert-manager.yaml"
./cluster/kubectl.sh apply -f "$manifest"
fi
1 change: 1 addition & 0 deletions cluster/operator-install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,4 @@ if [[ ! $(./cluster/kubectl.sh -n cluster-network-addons wait deployment cluster
./cluster/kubectl.sh describe deployment cluster-network-addons-operator -n cluster-network-addons
exit 1
fi

6 changes: 6 additions & 0 deletions components.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ components:
branch: main
update-policy: tagged
metadata: v0.43.0
kubevirt-ipam-controller:
url: https://github.com/maiqueb/kubevirt-ipam-claims
commit: c01d8855783da6b2a1773ec1f82694cd0658cd10
branch: main
update-policy: tagged
metadata: v0.1.1-alpha
linux-bridge:
url: https://github.com/containernetworking/plugins
commit: 14bdce598f9d332303c375c35719c4a158f1e7db
Expand Down
65 changes: 65 additions & 0 deletions data/kubevirt-ipam-controller/000-crd.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{{ if not .IsOpenshift }}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.13.0
name: ipamclaims.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
names:
kind: IPAMClaim
listKind: IPAMClaimList
plural: ipamclaims
singular: ipamclaim
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: IPAMClaim is the Schema for the IPAMClaim API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
interface:
description: The pod interface name for which this allocation was
created
type: string
network:
description: The network name for which this persistent allocation
was created
type: string
required:
- interface
- network
type: object
status:
properties:
ips:
description: The list of IP addresses (v4, v6) that were allocated
for the pod interface
items:
type: string
type: array
required:
- ips
type: object
type: object
served: true
storage: true
subresources:
status: {}
{{ end }}
Loading

0 comments on commit 587dfb3

Please sign in to comment.