Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kubevirt cert rotate doc #73

Merged
merged 1 commit into from
Dec 11, 2024

Conversation

brandboat
Copy link
Member

address comment harvester/harvester#5798 (comment), add a doc explain the KubeVirt certificate rotation.

Copy link

netlify bot commented Nov 28, 2024

Deploy Preview for harvester-home-preview ready!

Name Link
🔨 Latest commit 500faac
🔍 Latest deploy log https://app.netlify.com/sites/harvester-home-preview/deploys/67568db1211ecc0008be3262
😎 Deploy Preview https://deploy-preview-73--harvester-home-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link
Member

@w13915984028 w13915984028 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the full context of issue 5798, please double confirm if the cert rotation has some side effects on a short time-window. thanks.


- Harvester: [Issue 5798](https://github.com/harvester/harvester/issues/5798)

In Harvester's embedded Rancher UI, you may see some warnings on the secrets page indicating that the KubeVirt certicates are about to expire. While there is no need to worry about it as KubeVirt handles automatic certificate rotation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are also logs from related PODs, better to add some examples.

- `.server.renewBefore`: refers to how much time before the server certificates expire a new certificate should be issued. The default value is 1 day * 0.2.

## Triggers for Certificate Rotation
The following conditions can trigger certificate rotation in KubeVirt, note that this list highlights key triggers and is not exhaustive.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when kubevirt rotates the certs, are there a short time-window ? that will cause new VMs are failed to be created?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. From what I can see, there is an overlap period where the old and new certs are merged and served, before the old one expired: https://github.com/kubevirt/kubevirt/blob/ec514a29d5552cf5442f262a140681c099e124c8/pkg/virt-operator/resource/apply/core.go#L533-L551

Copy link
Member Author

@brandboat brandboat Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you both !

when kubevirt rotates the certs, are there a short time-window ? that will cause new VMs are failed to be created?

I tried to test this by setting KubeVirt CR certificateRotateStrategy to very small period

certificateRotateStrategy:
  selfSigned:
    ca:
      duration: 60s
      renewBefore: 10s
    server:
      duration: 30s
      renewBefore: 5s
configuration:

Then I tried to pause/unpause vm multiple times, and it turns out during the cert rotation, there might have a chance that KubeVirt may be unable to handle requests.
image

unpause virtual machine default/test failed, the server is currently unable to handle the request (put virtualmachineinstances.subresources.kubevirt.io test)
pause virtual machine default/test failed, Internal error occurred: Put "https://10.52.0.67:8186/v1/namespaces/default/virtualmachineinstances/test/pause": could not verify peer certificate: x509: certificate has expired or is not yet valid: current time 2024-12-06T08:37:58Z is after 2024-12-06T08:37:56Z

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we need to warn users about the comment above. I’d appreciate your thoughts on this. Thanks! c.c. @bk201

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the side effect is trully existing, we may consider:
(1) Add this formally in the document;
(2) Evaluate a proper default value on Harvester, is current KubeVirt's default value fitting for Harvester? Expose it to Harvester setting so to let user have an option to adjust.
(3) Have a place on the UI to show how long it will expire, user can monitor it, trigger the rotation actively e.g. on a cluster maintainence period.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@w13915984028 The PR should be used to clarify the current kubevirt cert rotate behavior, for any improvement it's out of scope and please file an issue to track.

This page explains how KubeVirt manages self-signed certificates, the configuration, and the triggers for certificate rotation.

# KubeVirt Certificate Rotation Strategy
KubeVirt applies self-signed certificate mechamism, where both CA and certifcates are rotated on a defined recurring interval. You can check the corresponding setting `certificateRotateStrategy` by using cmd
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
KubeVirt applies self-signed certificate mechamism, where both CA and certifcates are rotated on a defined recurring interval. You can check the corresponding setting `certificateRotateStrategy` by using cmd
KubeVirt applies self-signed certificate mechanism, where both CA and certifcates are rotated on a defined recurring interval. You can check the corresponding setting `certificateRotateStrategy` by using cmd


- Harvester: [Issue 5798](https://github.com/harvester/harvester/issues/5798)

In Harvester's embedded Rancher UI, you may see some warnings on the secrets page indicating that the KubeVirt certicates are about to expire. While there is no need to worry about it as KubeVirt handles automatic certificate rotation.
Copy link

@ihcsim ihcsim Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In Harvester's embedded Rancher UI, you may see some warnings on the secrets page indicating that the KubeVirt certicates are about to expire. While there is no need to worry about it as KubeVirt handles automatic certificate rotation.
In Harvester's embedded Rancher UI, you may see some warnings on the secrets page indicating that the KubeVirt certicates are about to expire, like the one in the screenshot below.
These warnings will be auto-resolved by KubeVirt before the certificates expired.

- `.server.renewBefore`: refers to how much time before the server certificates expire a new certificate should be issued. The default value is 1 day * 0.2.

## Triggers for Certificate Rotation
The following conditions can trigger certificate rotation in KubeVirt, note that this list highlights key triggers and is not exhaustive.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. From what I can see, there is an overlap period where the old and new certs are merged and served, before the old one expired: https://github.com/kubevirt/kubevirt/blob/ec514a29d5552cf5442f262a140681c099e124c8/pkg/virt-operator/resource/apply/core.go#L533-L551

Copy link
Contributor

@jillian-maroket jillian-maroket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I restructured parts of the content. Ping me if anything's unclear.

kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
@brandboat brandboat force-pushed the add-kubevirt-cert-rotate branch from 4af2acd to 3dec6de Compare December 6, 2024 09:49
Copy link
Contributor

@jillian-maroket jillian-maroket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying the suggested changes. I reviewed the new content.

kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
kb/2024-11-28/kubevirt_cert_rotation.md Outdated Show resolved Hide resolved
Copy link
Contributor

@innobead innobead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, LGTM.

Signed-off-by: Cooper Tseng <cooper.tseng@suse.com>
@brandboat brandboat force-pushed the add-kubevirt-cert-rotate branch from 3dec6de to 500faac Compare December 9, 2024 06:26
Copy link
Contributor

@jillian-maroket jillian-maroket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@bk201 bk201 merged commit 389f474 into harvester:main Dec 11, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants