From 3e6a1c992669e8d9dd9de24fc6ce0d1952804080 Mon Sep 17 00:00:00 2001 From: Jian Wang Date: Mon, 22 Jul 2024 19:48:51 +0200 Subject: [PATCH] Add KB about how to shutdown a Harvester cluster Signed-off-by: Jian Wang --- .../harvester_cluster_shutdown_and_restart.md | 572 ++++++++++++++++++ kb/2024-07-22/imgs/harvester_503_error.png | Bin 0 -> 33184 bytes .../imgs/harvester_unavailable_on_rancher.png | Bin 0 -> 93895 bytes .../imgs/rancher_manager_harvester.png | Bin 0 -> 76466 bytes 4 files changed, 572 insertions(+) create mode 100644 kb/2024-07-22/harvester_cluster_shutdown_and_restart.md create mode 100644 kb/2024-07-22/imgs/harvester_503_error.png create mode 100644 kb/2024-07-22/imgs/harvester_unavailable_on_rancher.png create mode 100644 kb/2024-07-22/imgs/rancher_manager_harvester.png diff --git a/kb/2024-07-22/harvester_cluster_shutdown_and_restart.md b/kb/2024-07-22/harvester_cluster_shutdown_and_restart.md new file mode 100644 index 00000000..0e186093 --- /dev/null +++ b/kb/2024-07-22/harvester_cluster_shutdown_and_restart.md @@ -0,0 +1,572 @@ +--- +title: Shutdown and Restart a Harvester Cluster +description: Details steps about how to shutdown a Harvester cluster and restart it. +slug: shutdown_and_restart_a_harvester_cluster +authors: + - name: Jian Wang + title: Staff Software Engineer + url: https://github.com/w13915984028 + image_url: https://github.com/w13915984028.png +tags: [harvester, cluster, shutdown, poweroff, replacing, rancher] +hide_table_of_contents: false +--- + +Scenarios: + +1. The Harvester cluster is installed with 3+ nodes. + +1. The **Rancher manager/server** is deployed independently. (Hereafter it is mentioned as `Rancher manager`) + +1. The Harvester cluster is imported to this Rancher manager and works as a node driver. + +1. The Rancher manager deploys a couple of downstream k8s clusters, the machines/nodes of those clusters are backed by Harvester VMs. + +1. There are also some traditional VMs deployed on the Harvester cluster, which have no direct connection with the Rancher manager. + +You plan to move those Harvester nodes geographically, or to power off the whole cluster for some time, it is essential to shutdown the Harvester cluster and restart later. + +:::note + +2 3 4 are optional if your Harvester cluster is mainly running as an IaaS component. This instruction covers all the above scenarios. + +::: + +## General Principle + +To safely shutdown a Harvester cluster, you need to follow the roughly reverse order of the cluster installation and the workload deployments. + +Those facts need to be taken into account particularly: + +1. The common methodology of kubernetes operator/controller is to try things continuously until they meet expectations. When the cluster is shutting down node by node, if you don't stop those workloads in advance, they will try hard until the last node is off. It causes the last few nodes to have heavy CPU/memory/network/storage usage and increases the chance of data corruption. + +1. Each Harvester node has limited capacity of CPU/memory/network/storage and the max-pod-number, when all workloads are crowded on the last few nodes, the unexpected pod eviction, scheduling failure and other phenomena may happen. + +1. Harvester has embedded Longhorn as the default CSI driver, each PV can have 3 or more replicas, when replicas are rescheduled to other nodes, Longhorn will copy data from source node and rebuild the replica. Undoubtedly, stop the PVs as much as possible before the cluster shutdown to avoid the data moving. + +1. Unlike normal kubernetes deployments which have no PVs and are more flexible & agile to deploy anywhere on the cluster, the VMs are backed by massive sized PVs, slowly to move/migrate or even pinned on certain nodes to take the advantage of pci-passthrough/vgpu/... and much more sensitive to data consistency. + +Needless to say, it is a bad practice to brutally power off the nodes on production environments. + +## 1. Precondition + +### 1.1 Generate a Support-bundle File + +For trouble-shooting purpose, it is essential to follow [this instruction](https://docs.harvesterhci.io/v1.3/troubleshooting/harvester#generate-a-support-bundle) to generate a support-bundle file before taking any actions. And make sure the [workload namespaces are added](https://docs.harvesterhci.io/v1.3/advanced/index#support-bundle-namespaces). + +### 1.2 Keep Network Stability + +:::info important + +Harvester cluster is built on top of Kubernetes, a general requirement is that the Node/Host IP and the cluster VIP should keep stable in the whole lifecycle, if IP changes the cluster will fail to work as expected. + +If your VMs on Harvester are used as Rancher downstream cluster machines/nodes, and their IPs are allocated from DHCP server, also make sure those VMs will still get the same IPs after the Harvester cluster is rebooted and VMs are restarted. + +::: + +A good practice is to have detailed documents about the infrastructure related settings. + +- The bare metal server nic slot/port connections with the remote (ToR) Switches. + +- The VLAN for the management network. + +- (Optional) The DHCP Server, ip-pools and ip-mac bindings for the Harvester cluster if DHCP server is used. If there is no fixed ip binding, when the server restarts after some days it may get a different IP from the DHCP server. + +- The [VLANs for the VM networks](https://docs.harvesterhci.io/v1.3/networking/harvester-network#vlan-network), the CIDRs, default gateways and optional DHCP servers. + +- [NTP servers](https://docs.harvesterhci.io/v1.3/advanced/index#ntp-servers). + +- DNS servers. + +- (Optional) The [http proxy](https://docs.harvesterhci.io/v1.3/advanced/index#http-proxy). + +- (Optional) The private [containerd-registry](https://docs.harvesterhci.io/v1.3/advanced/index#containerd-registry). + +- (Optional) The firewall configurations. + +See the [Harvester ISO Installation](https://docs.harvesterhci.io/v1.3/install/index) to review the infrastructure related settings for the Harvester cluster. + +Before the Harvester cluster is restarted later, check and test those settings again to make sure the infrastructure is ready. + +## 2. Backup + +### (Optional) Backup VMs if Possible + +It is always a good practice to backup things before a whole cluster shutdown. + +### (Optional) Backup Downstream k8s Clusters if Possible + +Harvester doesn't touch the (Rancher manager managed) downstream k8s clusters' workload, when they are not able to be migrated to other node drivers, suggests to backup those clusters. + +### (Optional) Stop or Migrate Downstream k8s Clusters if Possible + +Harvester doesn't touch the downstream k8s clusters' workload, but suggests to stop or migrate the downstream clusters to avoid your service interruption. + +## 3. Shutdown Workloads + +### 3.1 Shutdown Traditional VMs + +1. Shutdown VM from the VM shell (e.g. Linux `shutdown` command), the OS itself will save data to disks. + +2. Check the VM status from [Harvester UI - VM page](https://docs.harvesterhci.io/v1.3/troubleshooting/vm#vm-general-operations), when it is not `Off`, then click the `Stop` command. + +### 3.2 Shutdown Rancher Downstream Cluster Machines(VMs) + +Suppose your Harvester cluster was [imported to Rancher](https://docs.harvesterhci.io/v1.3/rancher/virtualization-management) as a [node driver](https://docs.harvesterhci.io/v1.3/rancher/rancher-integration#creating-kubernetes-clusters-using-the-harvester-node-driver) before. + +When Rancher deploys a downstream cluster on node driver Harvester, it creates a couple of VMs on Harvester automatically. Directly stopping those VMs on Harvester is not a good practice when Rancher is still managing the downstream cluster. For example, Rancher may create new VMs if you stop them from Harvester. + +:::note + +This depends on the [auto-replace](https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration#auto-replace) and/or other options on `Rancher Manager`. + +::: + +If you have got a solution to **shutdown** those downstream clusters, and check those VMs are `Off`; or there is no downstream clusters, then jump to the step [disable some addons](#33-disable-some-addons). + +Unless you have already deleted all the downstream clusters which are deploy on this Harvester, **DO NOT** [remove this imported Harvester from the Rancher manager](https://docs.harvesterhci.io/v1.3/rancher/virtualization-management#delete-imported-harvester-cluster). Harvester will get a different driver-id when it is imported later, but those aforementioned downstream clusters are connected to driver-id. + +To safely shutdown those VMs but still keep the Rancher manager managed downstream cluster `alive`, please follow the steps below: + +#### Disconnect Harvester from the Rancher Manager + +![Rancher and Harvester relationship](./imgs/rancher_manager_harvester.png) + +:::note + +Harvester has an `embedded rancher` deployment which is used to help the lifecycle management of Harvester itself, it is different from the independently deployed **Rancher manager** for multi-cluster management and more. + +::: + +The `cattle-cluster-agent-***` pod is the [direct connection between Rancher manager and Harvester cluster](https://docs.harvesterhci.io/v1.3/rancher/virtualization-management#importing-harvester-cluster), and this pod is monitored and managed by the `embedded rancher` in Harvester, scaling down this pod does not work. The `embedded rancher` will scale it up automatically. + +Run steps below to suspend the connection. + +All following CLI commands are executed upon **Harvester** cluster. + +1. Set the `management.cattle.io/scale-available` of `deployment rancher` to be `""` instead of `"3"` or other values. + +This change will stop the auto-scaling. + +``` +harvester$ kubectl edit deployment -n cattle-system rancher +apiVersion: apps/v1 +kind: Deployment +metadata: + annotations: +... + management.cattle.io/scale-available: "3" // record this value, and change it to "" +... + generation: 16 + labels: + app: rancher + app.kubernetes.io/managed-by: Helm +... + name: rancher + namespace: cattle-system +``` + +2. Scale down the `rancher` deployment. + +``` +harvester$ kubectl scale deployment -n cattle-system rancher --replicas=0 +deployment.apps/rancher scaled + + +harvester$ get deployment -n cattle-system rancher +NAME READY UP-TO-DATE AVAILABLE AGE +rancher 0/0 0 0 33d +``` + +3. Make sure the rancher pods are gone. + +Check the `rancher-*` pods on `cattle-system` are gone, if any of them is stucking at `Terminating`, use `kubectl delete pod -n cattle-system rancher-pod-name --force` to delete it. + +``` +harvester$ kubectl get pods -n cattle-system +NAME READY STATUS RESTARTS AGE +.. +rancher-856f674f7d-5dqb6 0/1 Terminating 0 3d22h +rancher-856f674f7d-h4vsw 1/1 Running 23 (68m ago) 33d +rancher-856f674f7d-m6s4r 0/1 Pending 0 3d19h +... +``` + +4. Scale down the `cattle-cluster-agent` deployment. + +``` +harvester$ kubectl scale deployment -n cattle-system cattle-cluster-agent --replicas=0 +deployment.apps/cattle-cluster-agent scaled + + +harvester$ kubectl get deployment -n cattle-system +NAME READY UP-TO-DATE AVAILABLE AGE +cattle-cluster-agent 0/0 0 0 23d +``` + +Please note: + +1. From now on, this Harvester is `Unavailable` on the Rancher manager. + +![Unavailable](./imgs/harvester_unavailable_on_rancher.png) + +2. The Harvester WebUI returns `503 Service Temporarily Unavailable`, all operations below can be done via `kubectl`. + +![503 Service Temporarily Unavailable](./imgs/harvester_503_error.png) + +#### Shutdown Rancher Downstream Cluster Machines(VMs) + +1. Shutdown VM from the VM shell (e.g. Linux `shutdown` command). + +2. Check the `vmi` instances, if any is still `Running`, stop it. + +``` +harvester$ kubectl get vmi +NAMESPACE NAME AGE PHASE IP NODENAME READY +default vm1 5m6s Running 10.52.0.214 harv41 True + + +harvester$ virtctl stop vm1 --namespace default +VM vm1 was scheduled to stop + +harvester$ kubectl get vmi -A +NAMESPACE NAME AGE PHASE IP NODENAME READY +default vm1 5m6s Running 10.52.0.214 harv41 False + + +harvester$ kubectl get vmi -A +No resources found + +harvester$ kubectl get vm -A +NAMESPACE NAME AGE STATUS READY +default vm1 7d Stopped False +``` + +### 3.3 Disable Some Addons + +Harvester has some addons which are backed by PVCs, it is necessary to disable them. + +The `rancher-monitoring` addon should be disabled. + +The experimental [Rancher Manager addon](https://docs.harvesterhci.io/v1.3/advanced/addons/rancher-vcluster) should be disabled. + +For other addons, please follow the Harvester document to keep or disable them. + +From Harvester UI [addon page](https://docs.harvesterhci.io/v1.3/advanced/addons), write down those none-Disabled addons, click `Disable` menu to disable them, wait until the state becomes `Disabled` + +From CLI: + +``` +$ kubectl get addons.harvesterhci.io -A + +NAMESPACE NAME HELMREPO CHARTNAME ENABLED +cattle-logging-system rancher-logging http://harvester-cluster-repo.cattle-system.svc/charts rancher-logging false +cattle-monitoring-system rancher-monitoring http://harvester-cluster-repo.cattle-system.svc/charts rancher-monitoring true +harvester-system harvester-seeder http://harvester-cluster-repo.cattle-system.svc/charts harvester-seeder false +harvester-system nvidia-driver-toolkit http://harvester-cluster-repo.cattle-system.svc/charts nvidia-driver-runtime false +harvester-system pcidevices-controller http://harvester-cluster-repo.cattle-system.svc/charts harvester-pcidevices-controller false +harvester-system vm-import-controller http://harvester-cluster-repo.cattle-system.svc/charts harvester-vm-import-controller false + +Example: disable rancher-monitoring + +$ kubectl edit addons.harvesterhci.io -n cattle-monitoring-system rancher-monitoring + +... +spec: + chart: rancher-monitoring + enabled: false // set this field to be false +... + +``` + +:::note + +When an addon is disabled, the configuration data is stored to reuse when the addon is enabled again. + +::: + +### 3.4 (Optional) Disable other Workloads + +If you have deployed some customized workloads on the Harvester cluster directly, it is better to disable/remove them. + +### 3.5 Check Longhorn Volumes + +The volumes should be in state `detached`, check the related workload if some volumes are still in state `attached`. + +``` +harvester$ kubectl get volume -A +NAMESPACE NAME DATA ENGINE STATE ROBUSTNESS SCHEDULED SIZE NODE AGE +longhorn-system pvc-3323944c-00d9-4b35-ae38-a00b1e8a8841 v1 detached unknown 5368709120 13d +longhorn-system pvc-394713a4-d08c-4a45-bf7a-d44343f29dea v1 attached healthy 6442450944 harv41 8d // still attached and in use +longhorn-system pvc-5cf00ae2-e85e-413e-a4f1-8bc4242d4584 v1 detached unknown 2147483648 13d +longhorn-system pvc-620358ca-94b3-4bd4-b008-5c144fd815c9 v1 attached healthy 2147483648 harv41 8d // still attached and in use +longhorn-system pvc-8174f05c-919b-4a8b-b1ad-4fc110c5e2bf v1 detached unknown 10737418240 13d +``` + +## 4. Shutdown Nodes + +Get all nodes from Harvester WebUI [Host Management](https://docs.harvesterhci.io/v1.3/host/). + +From CLI: + +``` +harvester$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +harv2 Ready 24d v1.27.10+rke2r1 // worker node +harv41 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 // control-plane node +harv42 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 // control-plane node +harv43 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 // control-plane node + +``` + +### 4.1 Shutdown the Worker Nodes + +1. Ssh to the Harvester `worker nodes`. + +2. Run command `sudo -i shutdown`. + +``` +$ sudo -i shutdown + +Shutdown scheduled for Mon 2024-07-22 06:58:56 UTC, use 'shutdown -c' to cancel. +``` + +3. Wait until all those nodes are downs + +### 4.2 Shutdown Control-plane Nodes and Witness Node + +To now, there are generally three control-plane nodes left, and three `etcd-*` pods are running in `kube-system` namespaces. + +The first step is to find which one of the `etcd-*` pod is running as the leader. + +Run below command upon any of the `etcd-*` pod, note the `IS LEADER` column. + +``` +$ kubectl exec -n kube-system etcd-harv41 -- env ETCDCTL_API=3 etcdctl endpoint status --cluster -w table --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key + ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| https://192.168.122.141:2379 | c70780b7862269c9 | 3.5.9 | 34 MB | true | false | 45 | 6538756 | 6538756 | | +| https://192.168.122.142:2379 | db04095b49eb5352 | 3.5.9 | 34 MB | false | true | 45 | 6538756 | 6538756 | | +| https://192.168.122.143:2379 | c27585769b2ce977 | 3.5.9 | 34 MB | false | true | 45 | 6538756 | 6538756 | | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +``` + +#### Witness Node + +If your cluster has one [Witness Node](https://docs.harvesterhci.io/v1.3/advanced/witness#creating-a-harvester-cluster-with-a-witness-node) and the `etcd` leader happens to be on this node. + +``` +harvester$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +harv2 Ready 25d v1.27.10+rke2r1 // worker node +harv41 Ready control-plane,etcd,master 55d v1.27.10+rke2r1 // control-plane node +harv42 Ready control-plane,etcd,master 55d v1.27.10+rke2r1 // control-plane node +harv43 Ready etcd 1d v1.27.10+rke2r1 // witness node + ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| https://192.168.122.141:2379 | c70780b7862269c9 | 3.5.9 | 34 MB | false | true | 46 | 6538829 | 6538829 | | +| https://192.168.122.142:2379 | db04095b49eb5352 | 3.5.9 | 34 MB | false | true | 46 | 6538829 | 6538829 | | +| https://192.168.122.143:2379 | a21534d02463b347 | 3.5.9 | 34 MB | true | false | 46 | 6538829 | 6538829 | | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +``` + +Run `kubectl delete pod -n kube-system etcd-name` command to delete the etcd pod on the `witness node` to trigger the pod replacement and leader re-election so that the `etcd` leader will be located on one of the `control-plane` nodes. Check the `etcd` leader again to make sure. + +``` ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| https://192.168.122.141:2379 | c70780b7862269c9 | 3.5.9 | 34 MB | true | false | 47 | 6538833 | 6538833 | | +| https://192.168.122.142:2379 | db04095b49eb5352 | 3.5.9 | 34 MB | false | true | 47 | 6538833 | 6538833 | | +| https://192.168.122.143:2379 | a21534d02463b347 | 3.5.9 | 34 MB | false | true | 47 | 6538833 | 6538833 | | ++------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +``` + +To now, the `etcd` has three running instances and the leader is located on the `control-plane` node. + +:::info important + +Write down the information of those nodes like name, ip, and the leader. Ideally give them a sequence like 1, 2, 3. + +::: + +Shutdown the two `IS LEADER == false` nodes one by one. + +``` +harvester-node-shell$ sudo -i shutdown +``` + +### 4.3 Shutdown the Last Control-plane Node + +Shutdown the last `IS LEADER == true` node. Remember its physical information for restarting it first in the steps below. + +``` +harvester-last-node-shell$ sudo -i shutdown +``` + +## 5. Restart + +If the Harvester cluster has been moved to a new location, or has been off for days, or your infrastructure has changes, check and test the [network stability](#12-keep-network-stability). + +### 5.1 Restart the Control-plane Nodes + +#### Restart the Leader Control-plane Node + +Start the [last shutdown node](#43-shutdown-the-last-control-plane-node) first. + +Wait until the node is `Ready`. + +From CLI: + +``` +harvester$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +harv41 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 // control-plane node +``` + +:::note + +If the `embedded rancher` was not scaled down before, this step can also be: + +Check the Harvester UI is accessible and this [node on Harvester UI](https://docs.harvesterhci.io/v1.3/host/) is `Active`. + +This also applies to the following steps. + +::: + +#### Restart the Rest of Control-plane Nodes and the Witness Node + +Wait until all the three control-plane nodes or possibly two control-plane and one witness nodes are `Ready`. + +From CLI: + +``` +harvester$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +harv41 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 +harv42 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 +harv43 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 + +``` + +The `etcd` forms a quorum and can tolerant the failure of one node. + +#### Check the VIP + +The following `EXTERNAL-IP` should be the same as the VIP of the Harvester cluster. + +``` +harvester$ kubectl get service -n kube-system ingress-expose +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +ingress-expose LoadBalancer 10.53.50.107 192.168.122.144 443:32701/TCP,80:31480/TCP 34d +``` + +### 5.2 Restart the Worker Nodes + +Wait until all nodes are `Ready`. + +From CLI: + +``` +harvester$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +harv2 Ready 24d v1.27.10+rke2r1 // worker node +harv41 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 +harv42 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 +harv43 Ready control-plane,etcd,master 54d v1.27.10+rke2r1 + +``` + +### 5.3 Enable Addons + +Enable those previously disabled addons, wait until they are `DepoloySuccessful`. + +### 5.4 Restore the Connection to the Rancher Manager + +Run following 1, 2 commands on the Harvester cluster. + +1. Set the `management.cattle.io/scale-available` of `rancher deployment` to be the value recorded on the above steps. + +This change will enable the auto-scaling. + +``` +harvester$ kubectl edit deployment -n cattle-system rancher +apiVersion: apps/v1 +kind: Deployment +metadata: + annotations: +... + management.cattle.io/scale-available: "3" // recorded on the above steps +... + generation: 16 + labels: + app: rancher + app.kubernetes.io/managed-by: Helm +... + name: rancher + namespace: cattle-system +``` + +2. Scale up the `rancher` deployment on Harvester cluster. + +``` +harvester$ kubectl scale deployment -n cattle-system rancher --replicas=3 +deployment.apps/rancher scaled + +harvester$ get deployment -n cattle-system rancher +NAME READY UP-TO-DATE AVAILABLE AGE +rancher 0/0 0 0 33d + + +... + +harvester$ kubectl get deployment -n cattle-system +NAME READY UP-TO-DATE AVAILABLE AGE +cattle-cluster-agent 2/2 2 2 23d +rancher 1/2 2 1 33d + +``` + +:::note + +After the `rancher` deployment is ready, it will automatically scale up the `cattle-cluster-agent` deployment quickly. + +::: + + +3. Check the virtualization management on the Rancher manager. + +The Harvester cluster continues to be `active` on the [Rancher Virtualization Management](https://docs.harvesterhci.io/v1.3/rancher/virtualization-management) . + +4. Check the Harvester cluster WebUI. + +You should be able to access the Harvester WebUI again. + +### 5.5 Start VMs + +#### 5.1 Start Traditional VMs + +When there are many VMs deployed on the cluster, don't start all the VMs at the same time, suggest starting them group by group. + +Wait until they are `Running`. + +#### 5.2 Rancher Downstream Cluster Machines(VMs) + +After the Harvester cluster is re-connected to the `Rancher manager` successfully, the `Rancher manager` will handle the downstream k8s clusters' machines(vms) automatically. Wait until all the downstream clusters are ready. + +If `Rancher manager` does not restart the machines(vms) automatically, you can start those VMs from the **Vitrual Machines** page on Harvester UI. + +:::note + +This depends on the [auto-replace](https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration#auto-replace) and/or other options on `Rancher manager`. + +::: + +### 5.6 Generate a new Support-bundle File + +Generate a new support-bundle file on the Harvester cluster. + +Together with the [previously generated support-bundle file](#11-generate-a-support-bundle-file), the two files record the cluster settings, configurations and status before shutting down and after rebooting. It is helpful for troubleshooting. diff --git a/kb/2024-07-22/imgs/harvester_503_error.png b/kb/2024-07-22/imgs/harvester_503_error.png new file mode 100644 index 0000000000000000000000000000000000000000..304d84576b4d2d2c50f655f2dc690058b14023eb GIT binary patch literal 33184 zcmeFZWmuKl8a9fqWr2zTf=H;CARr=LiXw{AC=DXr-DP2rDxe@GEiKZGARvv>EueHa zQs;ix+Iycr-}!%jd|cPQ_F68PbG~Cd<9Y5H^QE+;*gjGkQW6r9eYb92mn9(~%_1S$ z_Hg%3{K)`W$y5B_-&P{GA#AFNJx&8+`4{M-acfa$4-%Ac$sU)DTZwKi|rowsVFJEKFUl8E~U&Srro$~ z*i5bB@m>E;^QqRark~1BaYeGy|@eAF|nWZNM+)nH?x z_Q`4}sO%y?e@z&#uec1wj_d!YpPlz=8}}SI_<#7>cER}c$^XL-LeY@G-(Gg)WSTQp@ z%WkcVv$S)5?)Pn4KI`4@Sk*+7m1&q1QoL*@JHLGSa&<5d2Wn+?I6Sk{bo56IyX(4r za_i}nKeF4~760D3bD%zy{~`y+wNO5r#?Lyp1A~K&RF_v~2ix=QQtx${G=A8YXFHkD zlJ{(>MM2{PBjdG1o8pxLGDT(OF&@S`E|bou<87%%nWn|A8`Al4B*XUb*%_$6r4n6*}IBt|_LQYPe?D_J=iv>{`irQdqa;u@*s}CML$kO>; z^R5yv^}Wc&DO1aNYhyW3z~S?~uI2gpG7n1T4CS`QI}#F^tj6Bn-tL8kg+3fVlB|a7 z+tc-Y?~R*weBQCKw%lfMgZSj4*VwO}y+ZsGoLVq|p+kpeM&rUi zM=>3wJJJ3~&}r!>US>Brd4o+pr(yGzU@nu!d^@wt`i)oOcuB6H z0z|G|vsj(e>CUnEOH)(Re(u+vFad|hdJSPGPM_WpA0Pj%w)SamE^ktDa{tK4i%UAy zA1r@Ye^g38GuD>s>$o`fg3h85x&Dgx%;-Nja_?e5QRx-XA}n`}+0kabDh| zc$tgr?2lq%PG4YQ(KIq5)6>)A;pP3((Q%ZL`QG&arn}NH;*|0U^0)M>Z#we3%vGJu zv7e3X42+Wse^Byp8;{GM%koTjx9Qw4>rB@pKgr0rFZ7ut9&DGbt?e_(Coj$^*iHAA ze|ybnRoFjhbNlUut6fB+JFrmZEk|TKd4_$B)fdW;DC=?HU@-8)R7i7N_Mg zKR#|odE|)hh(B>eJ^l`e92GwFZ|Pj;`)u5C$?F(*%9X6&Sl_@v>QFCADyrwH#hWyV z_mV}NoeP@tylFT~x%-@-yk$`tVt(#P#r6Q7Z8s@t>FWIO>-X>9r_Pd;^!C!@XqJA` zENe`-o9OJTAWb3qrO0LN=k&C1q0{n7R#wTggZR6V*t40XrDxgM+<%s*$rTh7LcJs^ zCOgvg%q=YsP31Vx{QTl99Cyo!MLDD7HKW*Gth8#r?LO2sw>8c6>$&fFEKVL_Q9eb{ zm88VcZC9S7)5TknH@s<9`kymIdn#}H`K!uL zza@ckeSLlIt{2YOy$gz;NKl=8=M8r}9Bv3FwpYI=kCQ0lC^6qLZB>`fT&uH&tw}GJ zd!5LOw>IQ*#;Sha$78Cr+NMPW|xV zc)wyto9xNR(sM#W>Ny`Nf3EE$K3nAaMNaGO|Jm#c!^N?dm60glS7BjcdR~Qvf`%;# z?)i4p#NTYlxtn;eAKhtXJhj*>vg}*1h=@pIq$t^)_m}tJ!(K~!P+`M1F+beE?!01- z7FpI9->5QzjdS!qd-5ccm$v%!kEW)BR8+h0BdB}*ubq1zK6&yNaln_S)I#s5q{znq z92(k*?r3jsf3r^UqW#Q$`6R`I*`__E$tu~=rhHZw7FUy%Gk=bb?&09zFp)jWWS*9m zCOhlg_fM;f7Ce%5=-;;U`$5fHmYMnaCz0nID)!P`-pzb3>B{Ut`Rkydm#9juKMSJ{ z6%4|zv}$?QSMT3Ho|Kei{nn`J&1v=rUw2zibsxo{7s0Yx8ngYv$UzJZ?aSrKzVw13`L>hhi(_{)EvBZXMp~1PjkP9wj&wGDuIo9Hq}xU&rSq%}3M=Hkq<{K!<>jH%_tAWqnVB(_N}HQ0 zbai#5lxv%9j*2ERv%l@2N^DMiS}h?8vc>oIveURbb>j*DYtVcCZcc2itjL{|1q zXB#;-`GfW7kKJTshdnsCxu1uIhK6xO;;cOQ_3bsu!(YF&RSO*{6;f2cWEeCZIC}KF zfaLT2bQg`=FJj$4V9S+@xsM7uUcTW=AG$d2cUJ1NhbkJbHwr=5Go0Iuo){9HnMP6C zRd19{{Jy=SrK5|GjenXtzf$ywXdyS9{s&tf%gQ~e*tNeM$Yi+3sozLaygo;E=gu7x zl5!8qGQ9l^X3+(etP2f`K6L!g(zGfcOm=3j);In5afrcfojq16JS>VVDIkEd`Yns+ zP+jn#pOnzma3L4-vF7s;7TLxfkMVgmhiZdX>)DG31_!@KT-%i?nvD7l6m}Ehvz-Xk zyVH_zcN>YI;{pkZfc*>!x>jj}@YaPMo9Dek-p9C~1qB`b@#DwS0hLU{m|?G*H{xYu zh;vV@h4bp1xrN0ejN8V%!)NZ443vAiXWxyJr2BiZT$=7< zD_}W!@|sa^0k34Z5DTr*1#VGF%6s?jeN;%LM?E~pGsz@;YH4eua{06H-9ZvRXlsvw zBGxxHtSGWvXTI#hJAPlaeM#z(naSzw>})<%do;qn$m}?A2!GIT)%<6wFTBS{+e^!H zfK8)F{usC2RCiKjYn#i)V%w+8%)9wh=qU#d9C(!~ugu4Z46 z%gn+81-(;!2yc$Epte`=Q8oA@IE%q%lJknU01?+;ky{ouM*_rPpUR2NHGPFV)%WO zk3Zz>;=*qJ^UA_xS6idO!GpJsa+`*jpKXYjiyPxTcJk!lMV)Gj2p-JAmDSZz;!n_( zvZ=^a0Fa0u^>pV|%S?BF66v%Dq8X+=d~G`W_wOI+D=w~>yc5E65T&pDd&T391Nr!C5!asM$bM2KvscbEbO&zmXVd5<>3GeTl9KXF^6E{^D!&&aHW)iX zOA{SW!~>XqE9&a$J&(Elj%Ho=+{wttOY;rFgKN{pV(2@T7ZPP-NVos>7w2Z*!+(~& znd5(RLEgHft4j-`!SQHUX{m@~C_dukdNe;)nC-ip%F?EocQ)-+14o2z;eIXFxSYNk3m?tC_5KPNiU4F`!$ z#1KA)`Ejs*HUOjjR%%q8BNeOa#haSrvxAkd-@JJ_nimmoN8Rx^&5! z>kOqjHfiPO6Y{;GJNGbM*b#O5_S@^kbn^Gt9+7|XB02~_NT=qle#hsS2)Pp{FVSF% zjC_xJ9?^8}k~~vDgdEWgBDecnsdEzN1^;|<1l)o^r4>AKa!smn4_xFhDW7KL1C&JM za2u$!E9YA3#7m|wPxrMM@~_(Tm+r`{F)=o_&MQ20TD+vIOC?V0kjxiz8yik*(o#~n zlz)o3;%TTkaS%O@{I96DwHLYMONRsp2RA-H572&q#%y`2=ca@NrH4ysmU}YEeKRvQ z+G7k1sf)IApdEK6=dgR2QW9}p#=Ql(l9h~+k7X43vbMW3EBy1}wI1MnjZhGa|B)o^ zQ^3Ewu)757%SE`M%L#U~rd-8R6U#z{Vsw&Xp#s54=m0Z01&b6(7sNK<> zYm0~f0S+>+-@Sj&B(F+JQeR&$LX7j$E@A7D1MfMFV!2{51p#Q{W&+1fojF5V`Sya6 z?Ab2=fY+}h*oV>^gs!bEP1>|(#<;94<)CS`He7PMcJk6*Dk`c#0B62^n*SM#Z}bcd z4F2=H5C8YP9tC7ntjABE7Co1R=`CUrx#JVb0b1G%w25Tz^CF7sr9%%SCna63c>3&F zq}$de(aWr5yA3rpAE5#KcA~Y3jEXu4j!A%p{<7ycZ{ECuHK?tt1EcZ8frucGHHfyN zOxhWo#P&3;T}zW)&l?-12^xf!8Fj0(zGJPDPeY3Gm1aTyVj?eqfn55;`xbv7o6XZ81g z3c48`9bHmgP0Gc^g-I-mj{>~<63ldDCA+DRSs~^Acxy6`?F4mcSy{TNb)Lig84oHp z)1MPBNS{TX%fY`se)5Fiyu`M!vhF9=Ix32>ps?_JfrC|H@6RP@Bu}3|ClKL_moKaM z8VX!f*f}_!Bqy_=;mKqfE7?0bMgk_DyXSS-!g6a={mz}ecryZj=H%oUuU$HSeh*N? zQ*ZBm#>U1tu7KR+#P3{xd1%M`xF=7prT9L_8$do$nvPOeBUe;XBFHdK13}GveWfCt zre|Uo78a70z0ps^tgSD<@$-u~M+zAI^VhGZD4Vjbu46!90z>E?9~-qang7M*ln&se z6h3w1W&Ye%6p5;N*0*@LiqQq4`Lei-+ogRwe$`b~-8yGFTpz06ox{#$+Wmrt({O~B zc~Zc6F8P(cYfAS71Edg|2QMr|qO z;clBRp=(9hU?m!}OjLkr`U{tPpNPl2=}pP9PMh9W-qWL|m3BvyNAH4q0X3*;2Yt=A zZ&$!UvVdkEJ|y{CS(!*z*4atJ;JR``x9)vR+Qk8k90Fl(-@g3-J$(vSmSWn2k)C{R z-TIL83wg(#_B-@|n8~0?5(xJgmvIT!#eBS#CBo(|6D0w_z-snVU%bL5H~#F|vj@Mc z12B3^%AS+Ew}BW@({P)5xo-T?#7h1UJzg>A7+oL0tYD52(OuwhnBZTNUD+1c64lDX zGGHv$lbw_Ru!M>cLwf5itC|Ot9Q{l+Zf@=*^+IKnJLcvWh1MoGpyZuv`yCV%q|~R8 zXT675b*6XN&innP?#juD$ti021JJxy>Em#w@l2v&HQdJS4>9!b+81b+J&Q2wD_X5j zJhz4&8gYf8cU6UAbZpTVHFSaPmEG`Dr_hWwS20$O?X}}1( zB117ci}U5vnyeCqmmBFV%x}zfXo{5pXZGV79iCZi(}=qIWT&APvHrliWWKzBt3cKl z+jJ^EZK+o^xK6aCGCbQ)2l+71_Sd(A#G-JU^?oH5Wo^misk{5F4Y!{*;K!n=>V=KP zZf^Dt4x>C>AwXRLi!N9Ul=j2o%|EwSRM8n9;iG7QijDk;xgBx-53ti$pouh~$aO$6 z^z8%6sh9xxr2K@Eh>9HHy-f^CL79Ii;|tPn_*F_0B@GOzXOg#MC`6C)+ZtyTG`rM= z3v1Y`9zJ}y-vTTtp|WpgYYiCB-RcrIw*p!e*{M|SWW_WQ9w`zJAm@SM;kfQxtIFf6 zLuY{KX6NVSpfJX`zT+5jpEtK>G=Go1%L^dZ4dnFhqxt<)wanBS9b~JwYtINi-{8h2)Wv8 zI#}V2I!pOw6i!`@W)tJuhPoYNc~@#(-~LJSS}V`3TeqsEgg2rI?JY5lN&e#p2!V`Z zu9%O7|;o&)l@gfgN!advULaNx<$Xw=F9h;fibW$75L04B-WwS)%j?WH0YXjwE zIyf0LmG&E%ON|lNl!s4RxvGA;|7a!$$JfC}OH(t#uQ(_!F0RvAcx&A%ezs_1Nu}Q< zf>~W%U6!{V_$MwWCw6j>}PhpRU*NmS#RNwdm5=Mx3W=tYj!t#y(LQ ze*oG&zT?BgX9cF1|8$0v_@mz$E~*QhJ9pFm81$2SVA!#j!Sef79i5z*p*8(jEqY*L z!cvgTqLQUw7}}F>$BY`TYnx7xjrRw>VurY)JbV?WNc)`o;Js7-fHjn1p4G6AITY7q z^CXZi^9lT-yVkrGKzwN$)L`!R<_w&pcOFtwQiPl>lWBO@dc1WXy2UkdagVUiO%o^z zP|&aVkFPN#QqwS_xJt^({@PmW+cKZ(=1s1UfZl>5c3G%x&Wd%UIsPz2$ETp=$(1CV`TGspQs__x zUC!Qrkzjg!l#BCe9xvp>f(AmL#AxyXCMHYPLg78LM4!zt?l^?WMCI`=H_KVhbd7W+ zwZ;O1DdbT$&Ml4WG!>f`puH*5N;x(y%s?%faHy`zG=;h|OO zW59V&SNBY;MDR0E3jNF?AyHA$eju&uc<2^emev-lX!Kc;%FGF)gorTa$hB$8YlUZ7 z??Bev#>U1*D_}2e*2UwtDFDFx>>qJ738&EKdDc_#~zunQ=82R{=4GV!mCpt69psk63qBf#FpdnD9>9*xPyxmGn zS|OJ|`WLLBi5~s4G)bnSqB7ByEj37y&7oWS5Nd$X=E|VzyLU`?NoQwg2^Q4G%wkx`+Y8$iqYJv@}sAleAgSKYbsaX{0@kF(f2}gv4!Qk%|yDA+^&AIfD^T zy!s=kBj_;qHyyw2S7@T{14fWmp&~?KMtuABtpq*n01b_Y+tvmZM#O5hPPPAeMf8|X zYBY0(@KKLiP$`FRy?j6@EP z4bUth4buxco}x&t!r5fE8Iv#A+FT>5kX557$r z`4ogai=_#r3mQeVXQjd@X!ORK<6-WVK{;)yV1ZDMfl`K5OkZjPR&WO|@$R~L=g##T zH+DjK_)OoGXT#jyxqbG(S%977^cONVgMmDrg2MHKNfUmH?NoQrbQ`Tpp4*mCmT3<@ z>AjnjU*4UUJJZD}R`;H>r9vv0Yac*TLtZNnAc0S50#N*YYz%SS6YeD^v`k;y>VsNl zjt>hTg@mXY*IuTwKttPBb|f_Wu`v%Yz>+Utwvo)t^j{;|Ddx7wU`$MmX7fiGNZ8c8meKX|E%#() z4?&WB0VU?6cY1ny8ODGN%vJ%{b!y1bRpt>oVZ|a?asUn&*>F0Xn0}Xq*-8ezWW+^c zn&aif3>k`f7p>r<$tEjv_AlOk$C?b9X;eJFC57H(>E$x;b&vOW)q0B4@>D?ec_t?D zc-7p`jqO82L)B24leBephxpKzV?S$E?g#uAtjvQ4^*ykr)SZlTNIFT8{q*tU$GaRy zM@PjW#9vNPiI+Vcx=u&-4<_1Nx+G`fc`?@NhHGp`CZt`c?;ouv+GS=J7w>{)L|0;g zdFf#wNW*nV%rNUyH-Kz3@atFfT~|%O^!BF^H zm=+vv0MfIUH&GQD$%-37d&c>w_dK#XS>oL}b( zrG5VV!_ZI{Grr$Mz-j4%Q6V}qZ`<&jywsJI6@mFzylme>7^p5N9v4UfnXH(l7jpg> z`2IbqZwEB`epM^KF6(g!nl){KtG|E$KD}rcF6fliYvnlau=99>RSX}N;?18P^{vm@ z*$RYGYdz9<)u1IIU|?V%vV{j?#hd!Ix?x}O7+KWRM??UafRYT`dZ120EI1j`2Y!&i z9)6JV#?6iT6DLmmD4q`vC|B2<5wLuNq`@Kv-BfK7{i0 z@ERQ?8y%?QG~w?sISBabZk2d9@$DR#cSHzUY_v}ev~T8MZnJwJg_d;Fpq16o3Igyy zGG7CX;02*Eo7UFW(m}J1&!UDCfNn;#P37gHxeoKXS&R533Rs)o<3;Gjva3W`=%{Re zkleF>EBz+9x{V0MX^zk+oFVfheLV#ehzj>%s(}R-N*DP0k{7K_@fZGPG(=}yeIK@L zgD$O@uDg#I+c=Cr;cb?|xv{>q#7rF9uI$tb$(w{LLihlcXsTYHb!rmSmchPO7Mi5y z1N79@1<5fdo`P}iv1_gI6ptxA8}59PmVu7W8{p@cQ84Dx%5tAuy5P{@VAMOGtC8wO z&MyHnhel2YACr@lTSqv98om*%@Dz6F|FB%X%{qDT*iF%%kqS#*RR_P9MYxRa@saf=(4ksFjt9LNSNcStgN{e1YciUo_?O1 zn%WrX5n0mEKn~nj>OeGL2A74`6*-sz4?ym7!!&m8rGAK5gaIw%<;(JBmCej~u7Zb% zW|*XywjJ*J59)mA^LRxC{cg;k+E9M(;?ej;LpK+=gM^3rG>@XlL{&`Q1a+66haB$O8>;f1uZ{7pykL$*wLVt%5kjgAb;`qrrTEZ*9gwU=l zosCsUrv%C=`~F=3~UElBg8~NEFT}AkII=$|M7{y0geH- z_%Vssd5>6^rM|f1z3Bnls&}sO7e+ph%@{?|>d>*~B&EZ`8w-AojzvX9n9dL7Wm1ZG z`1uDt*@_xhb6|&kfl|;vHRW^S+OwY^8hl5WgUyN#zK@)Z@)ddvj^R^TS6fS*6UGD=&O}ZXkwYK3K6Y)QcCsmp`kTGS4PvO>NsQ53tqRG*|VePgMSFMJw9@ z;;B4B`R!QwtVfPhs9-+$Gb>0ky@Qc?QczG$OI3X>2?y$18j zIJOv7^gn@r-Jz%XY(H#dWP~E30_F~&Uxt3G_*r`oPV?KrWmYIOL^+ytW!>_2`mjmZ$Sf0y8 zt@u?sU|$M)E2y6c{80J0H6i|S;K-4;lWC~9-Biblsv6g(O`4KbxZ{`Cfk3V`gbBJ2 zVkhGzl=4OsZ=L1kryGW^L3w$a70_2|g940vT>9U9 zbIib_;1E|Zf}|7V z&u5r)9YH!w%+N5wdjDtlv_DM7byAN^(dALIz;gRgcV)G;dr`>tAn>#0R2siluCS-7 z>eU85`fJzzn|7s5ad22aprJS(<>%z2Bwh_@?t7|+Fj$Aec-jLktu*UV*@$*-e$Ap3 zXmKK1FKqh)W`U6s=P#>_vBnJcG3(EGqG40MFvQ)b(i&K7L7+(%OaMb$7ANBnlrS8K>~-t3o(tsEWsjkgY9b9F?wqq7?&bjRAayfVQQXGHQ;N+T6jSK8(kkVB$}Gb z{r_s%N4+A?txnw!2n!>MUJKIDN9l;|_fpkaIAbUL-@K0%Z05Qb9NDdKiGzcvCe>nB zI;k*$gJa)T#EEc$!H-i!00W2T(9{_PX<*MZqc&Fg#CsBEUD6ag+}3Az2?z>iZq7g( z$^@-xrdELFnv<6|lv@B*U!Pq=_9DDO1g zI+}h{8?6ejM*wZ+^qsc89f%JR8M4VVlT7KKTaNoZIjcRIXMP;k*H32IvO+&~6YT%RR6}UF-P#Ci+!T8Uw7n~TJLz+sw ze0bRAdhq<-hqQ9uiHw$E1}HE2=O}vxMuf_EHVHX+9U^ZBJ(0`6^og$A?xnX?$TBf8 zk&*ZPs;(Pk1o=O1EPsw$4Ss>_b@1@vs7m8G*eo@wo?on@JeQ52Hxfb8#w7xu?R^PH z@t+@UYi-Ybbq6?IH+*PcA`kQvR99sDo8-G>)#d1%3NAr4=0n2T1jP^$s9)9j;su<@I-E!tSDF z7#S|dG&eMCP`FN}0^7}W@%;cz50ckZVph~H26b%E?!JHb_gCmov+>`kyT*PP#6o4#%HQa#H!sCfl> zt2zsxx%qiNnkQaj$YDss*yy+KXnT!KjlzE=-U9(H#$bAqn|0AA9K(Vx8@!nbM4bQv zdz-H_8?`1?SwwjlXW!H6adYoT+~9@mk!-r@7us?s-i<`P4@ZIT^R~6lSouLay}vu;nLv8C+E#l_@lZA0qM!keLvQ+h%hV> z>q8JxWY(8K_z3G)YYPjR{BEp|J}n7vUdN~ij6|r$I|+J?EYdd_fdxK*FKBV{`rY8N z*T-@hyD}G{ii6slXH{Z>qAc9T{q8J}pwE#?jBm!0QZntPtlF`2Cuz8rwT+j0zO6*w zm4c7lqp{(Iz&G3lO3ey#O`0X?F?m2&XOi#xFmqy`6Qc z;fUDAGu2jILa{Lc(+}>0B@k z$wr7C(Yo(8=G)Ich+nRS-o)#n+0)bW9eAOion-Jo8}0JPI=S^-NBBdF2{|ZLa5YIQ z!+9)3U@T4|40FV%rDg3e6Vo%qJq#xO|K`*px6e=IPd@g@I2hglMG-{fykRt62fa`zVXNAf^x>qoRx7b24oSxipS(ZDG4N&cNP1s$@(6SD*UZ|#500SEaWBQ9m%Mw{@$?z8a}AI zZ7!JKj3Qhn3Li?D0(%Q$h@e&uxvJj@6HppyPrjE-lV?48GNcBXvPUyB=KU+s?tj~x zR@=J&$ovNk7;;dKFvBgVfQUipe$ax>v;_jnk^06ZYRSt`K*PIjpD$X~Wm$hf$H<78 zlI;&4a~HeZ(4tDXHeX#Bzn$b{=F?g${?z%h;O?Ft_euemKN;pHKJRtGY4--O-Qniu z)<2*FDht0%y7<=7Mrdm>KJE}|NXvVmr&d~}XcW7C%FcdQ?d#=vXP3B^16>sx$f6g6 zusY8sx-GoBVDt4px2@1YKG)hncIAd@5r2(Kp&k2d7z0oArXC$D+y;95+ADH3oSqoO zCr@754fG0;K|eDE;k^o9hH%c)5jRhiW71y)?1gNoI+=56;ck0^M0SM&46WBl4gh^? zyOdW{3_x_vfy_jtil7A>)CvOo!A4|x%cij(k29LWw5UcG43d#-BHg5r87(l;NAv%$K+)sRt?wFcFZxmf%rWm?lS? zc?NeNe)%vl9CswoYz+Gn^AhQy44C$}B|@{2??9|%C|7l<%sJ@6-JtgMBa}qg72yJ` z+ch`|-&551y=Zw45s6}CHgLF}Bk4v;xlIsf35r4qmVQ3KPY4OA2dI`Dr)AbGvpyfh z;zwdi44}<|zuh}#h2z;jdwC;N$90#41EoJ+;sHXU`fVwpojGtv#3_RTK4~EC&Zc^_a@gU%(<16cRc zv`cj7bbRz&4!a1R{P3XMBaqkX3{3geR?4jHC14v*wIHw00*@tnpb@=@bDJjRLiI;>1fvDWdQkPl(=xBWUtVV%- zgoQ^Z#FN|JcGx9>V%(r$5rClk#ylx45aZW;0)6unP^Y&A!I77&AbdB*NimFeTi=R( zC2u{ZZ&GMKyS*V?n99f3%S+zLtJ-?wf>)*f)t|gk`c^hJ0UmC1ziy@Y>H$;ylXd>4 zs`riJ3mj)Dp0^)ATsCUm3p=%R7PI}TrKM$94LQ9);9z&1X|jB~H=0QTe^%adb1L!& ze3f;f&x-oJzIAjcF%^9<+v(?Eb|2hrsgQ`W#`LbuZV(;iD8JnQuE?CK>F%b*bCp4o zbsxw^LBS08#dkOH$&)9|S&MhunVRTWgyB-`MTa1m)j3zBKlR%`od6V#AFiEC#aIsH zGNB=yH!Q>eP#zS6^@yyl&=+V+MDaP;3cd93I7AQnBdVC8FHo+%cg|ZhNE6% z;{Y6mxNh)d8hUC3{^l;^+ z+qZ9H`o>TGSl}aCkYdO7u5}2cY(&(E$Y^L81H=?<{xMod`R{lYG=(t7mzo+f3=2_o zB-hp7RLBMFz=!&wML2rU7oMi3nx)j}@I{WixpUPo4Q%Sm(>;d?I3KZsGouCK>~Q2k zaYu3s`Zb~jtMwekb4=H3yo=&=Ih$wqP}FN>9G?l1$xu>Myb>e3M~XLC;o!@O@t|7k z`P`n}{_jdvFE3zBBq^ouBsw?r(dgK1X9j_@H^`wFQMyV1IxD5PClg#LO-sxtZBejm zdm$ZCSwyY`!tr0v$r*0IeUFf<*8LHEtJFOi&Mjg%ion(>HIQLLqte}6^D}AIHkrMM zq@2xot0oxoYSFfgPp?>vSww z-l($)B!iaW>oTYf!ttxsQdGEjMo_758DF-O43=5#tP+@>&%1SFiKjYo+}?C<>`k(# zfY()6OAwj8FoA!<%659e$$a2m=N%|~VT zfsGIvAudM@0Limg8(o46j(Z8v#Ly0ic%4m0Fn7pZTtk85{hcqn^wSopAK(@dXz1S!deM5NmXd|35cTWFoY+w4UH&9Y+%O+$P|;1FMyt9TmNp z?jOOR(tCDxcKwQK`c1L>XlZGm@)dzn5dj5Q#>U&UV})^dK&k16>wFzIlzVz2tkSw~ z-@dxVS?DV8l}Ly0o0{%N!|%7Z(dKWUCPQqRh@3(GKkOlFHMAGnYa<@V*cU#-G_OzJ zI?y3y&~|${xs6RH$pvb+^6F~w1QJQQp#85!cQhRRn*^6mN=5fN^n7Jp>wcSi^yU!l z@utvTY9f6H^;SJBbv!o>msMap62>3$nA$dL8ri0Fo-$tDVhk=%fs)j77<<4)zvJ4Y zu`i*;ceuJj7wiB1J0_jhMs$eR^rrjihIRCv@?8wh4}~{ZuG!NCi0?oO5%TGN$Y@H# zK|m*iKn;$yM+k!=&27syq8(_1@Psj~bh0YTN=pw=QwL6_5rGXG38nTW4Nc;1n>8-Y zLB%|vUI|B%Koyuo_Ynlgxrdvj3AM?wR$f>;|IOG~r;7gsXDCdy`YaYJ4k7{k`oyRI z=8s8};r8yPihlO|`EYI$u7Z?y`Jcb%?df^gC!p#os-6;b^z8bHApkNm7tL{I>{TF# z-u})EgC}U{dnBUPkuy+bV1s-MSNf^&1=vOOXYK?Z518&68T1vd0Chg6dSH>5)#r>| z(wXIvhmuKE7YzL!$Z)AV;WLAF^_z|ROK`mryKI%?`f9avn`LP3+}IQo_Ga`ZlZvyN zBEd0`-=Sc&Oi;_`0w#9{7W$=3O;mdNrb2tt-&3EH6y=Ahd&H!`ORb??$F%LYC8%5wUtk-kUv2=cz_2AzdO$5&`PN zOX_#++BGsA&BTc{mX7_YtEa8hGhhiL1hW{pkJt%|?-FY}2#|xHzdog<9rW?`^=+_H zf{^Q8d2rJ5O-RTE0m0Om7@bE{S=%4NV>F+D=(8AjOOg<=@j|oft5$^)R^5@W9X3Av zHwz$2hEU1vkLSeHUVpxG;!%+&HJK+i`sc=SAC*rZC<`tC1}x5F0wH`I!cw7_m$}_C zlzmZ+;`G{=3%)WNgTN~_wYC0C=W524#j;*V#46s6O;@@~<@T!*%EnW&5G>4dpfNBl z`UzXrZ@_JmJY3)akm0}mSp{MtF0jCotG8^5A>_4ks_UBarX1G_CL7}L1Y}e?dHMN+ zQ0G=w27RqqsxRCOcbmr06?_5|ELZUs4VD_p+SKYb^j?+QbwN zZGv@F9^5#b~)VhlD=$^XJcp=jCEYO!#5Gi1Zw2 zae4Ub=g;E>g{53N=}^-~<4V&&(2=OpY;8!T3MF}%n8=LyOiX(BTq16b0XGkmntY=T zC;1P2IpdW6EIr*S#kVFzPdl4g&YaE3hut4wh}149JuB;3aj{zjQ)f&;rv7)4F=aQL zYJ1=nT&5tdE*sY%-(j5L9)y7*JNu_-^P9;rsG>LWx2*f&Z(>iLnXjYdr6WXkY@9x& zi5t60p&p{^SX?DKW%vs(KYsjZT%$u;t#Q6~tw9m;vbg;-Twq|R4dIn5-gES(h{$8} zB&hG6o}TpSRWGOa5#w*J;=djPQ-(HQSUj_&{cxz=(@Ln62R(?aAAOT_lWUFhr*!}+ zLEXq+Ozj1}qHnuW$Cd(WcWFTDDNQrWC&K60xYAj~#O(hpoX6};bC@48t`Yh(!U(^7 zj88N1M^%-EW{@%5s-~qe!g&I&|0G$QVE-(B!?bjpM?| zBO-Rr%gA)|lBlhWj0~Pw7(J8t^u{%CN3BEO*MB>*g8ZHon?u1HT)!Q39-f*?X)~cw zzJLG2j7?~8@G+p?q+zeGLBYYd1*M3Xz1!yME|O0mL&i05BAGcEtC`&$wvHwOT$5G$ z&~TD61&I50Q2W$ozQeCWKwAiuki0a#54Y^vGl|O`9u)t*O!8ViCN543PP>u@l26(7 z6+XVcR6cjPDI+~S$zW4mxLl7G^>jenqXqZP-Vc8UiqY+Iga1THuiyttzq^tqGHQYu zap1P`=iD%jnpTz-xalHrvW>4cb(s=SCk-nNn8QoBonOn#bp%Zf3|?a5k1+}+zYLQi z`3b7ySo^EkpjIcg=@)x~n}8mLiP@2A16f=lP-RevmzNcNaH*$SJB1wiKsH*Vo4CzX znucW9cdawpyJ7J7k69J~6Vl2hX6t`H*)SZ%j(E>od1oYmnk628wmeph^sHl782c;U z27u7c5s%$s-ZVQftov8sgKr}e)}W}L1ve2qG0NO1E!y&`gK`tSX*a`Fyi~PrhLk6t zW9=!s&)jDy7L54_aQjSKAo&8Zl`me5BuC4}&-{L;p}WxyVSz}8oYfltJOgL_2(%8L z3JW3umLV9#db2K=8$DEBUOsDM8Rimk`^?_hMmzTST?OhSGuYv{xpIjJl4P6U3Vi&= zS|Y!zVe`im1)<;!@ddrMa3S3R`T!9D=!u6`TOcy`@Pfn{Mh)s}YHssz;nqTbe?K5z zlZp`Tbr8_&=>rF^F@}btqQQ4G6Hyir-{({@CT;r9Khr2aapiE(_+LST9nvTgLR;&% zN3cXt_u8}j&>Y})X+b;@nZ+eK5fBVeB5#@)-)8Hf9=pB(4=G%)g?by)cGw2NA6nm* zR1FQbu1iPYR3Gx;te7V*G9z3@BqDd0Ga#mgFY(Xk5f9 z%7aVW^ptVqhez}rRccV??jxxHHWjflGBP64#2${tK3CkH_Ma zM{w|0=@tH!)p*pKo@)>pNt$860}%@VY)X$rB;6oq1`f}m>+7wp0{w~%db~P6hij5= z>a!BP@m&!kHqH_~ZY^}e(*^TBGq&139g!ljMn-7;tg1P$CfQB$MMmWi47C)nYk{im zPoLh;ZfLx+e6CYl-6{C8L8NVNZQ7~M4Q^Z{%3&-*^giMotA~%z+p#<6n|>pGgYj^) z?#hhgqt||ZhtVP;d^44#fF z{@+D+6(fNV0zr~TJ-kO&6!$xnQwNPY;{Zdetsqv_J&h|Z03U@iA7km%wK(0!0OqNj z+?8PW-qm~4eMYH;?^P+yvU_Yj(df=(rvCZ!#~$CKAva|q)(ZdcNNV1D zpTta8P|2!k*X8_=iHSzp>OPmSlGCU)Lxqh;*SX{L+Es6aV=6)W-zOb~ zKJi1-KOrF@dQ4VnM^a2&aBr*ZzEfGlvw=P>Z$F4)?-DysBs}nT}8ZMbtAAZFZ&3c=C1x)YLme7jhF!NkpTWxR)vz|=J zpX%)I4?yoOM}oGURjc7WXJXY_gTQ>Pu2j53tCHU7!eWh$F@;-gu1c!Tv*ovv0}FCf zoqsbMxG4YRsOMpXKK3So2qF!(6>r+)FaE#=MsF$QfztvF-N;`oz2S3IL(&au&R}wO zFV;7sNzxPDm{X|S0R2yqw+I&Wgt=dzqR>huake4UJ8W*q*5vKw`LaMAiS0}E?DjyVhFu02ge_`(4q@7q$vGwa}=M>f4ONU-w=eKRLfQT9-uZ9l(X46U9 ziDxQjFcK~FR?FLW?<%3ub>%XWJX|;qW>0b@e{DS>3TWRK_#@lC0A(00)f0-Z3>d8R zl}fN}{BZS_doev+^)u6MxTNma*#|-uoNVBGf4sn9e&Dqj{jI#sAGg`OF_P|0rf1@= z&Vl7_E2&`Ld{BWvAgFuO(|82g)bROwem3Z%LCr$M(upZ4DSc*fJ&zPN2W!Dhsee}P z!jIccj=xip@vR_top};Ho{Ee3(F*d$FXgwN=I*uENCEHPCw zVG3_AkuX)q-RFgZ)E$50;yb&T>L!S9Kna&O5gUgdDjugKsoUa`r4j#Z8jf0R5JUp0+pgvYR&pLm+*=`rlnt!)7sRXPOAk^RR+l`m~8xME2amP z({B;7i(b9GE53FZPW%AA5y_;|l&=Cqasb7LawsKZ^CAjrvio~0?CJBYxxd~-8ytv%8uGxIiU?kUsU-*34#cGU+csCWDtbHFzHx-9o(cUWMW9aK=Y5S3lKbLtQ~ zbZtc#XFptAkHTyc!Zj*?c@qSktHi8ScKG2oU-d&qCbK`%u_vjn>NjS1^HhWE{uCYuPt(gkU zem#84dohY(<6L~6^{dyf54z)SSex)d3L81}yd6l8mcIT{2G&u@P@S<-~8a*qQ3AL}tsu8|8akTF~|t@c?-pN53#3o6T{(gxjdrAo^6 z^FBVqP9Vjw1@U~h)f$a~YUWhqVIL9_zOehu_=3I3lr*=G0;|9LSZK!apS;kU&)jwT zdQj|CCezpuaq4xSgh1K0SRbG&)?X<4{Q2`rSKnsquO)zG55(C7N5h1K-5^rULbf?u zxDj^`D_Kcvb>{+t{qcW*W6K4XyZz=8q$Xn`B2J{_4!tK7W&wnv_^+_fy zO{6#Y^w@SKhw>X1hT8S=PUQ}VMgNr#|BaYu+4eRA*?q40Rcmguo*LJQ217HR>b7Bo zkm#vR9FN#;F&7t?^unKkvpPQ-8xOowvsHrmpViTc_E>I`-ShUx3SPiTwlZjIb9&2k zZjr-b?pO4^X5rxws{!ObCF3i7Y$wtx�}%32&T-Ak(5<2lhFD*rB0`iuFS@mGT`_ zs;}fH-Ko?ATNBI7>hKY}B1@W(sG$a%@FH3`s`tD0@ubh6FHIH9*Yhm3YiS__Bcraq z>HeN*FZjEG3#7lh^5uOv@RcSW7_eDY@6o&~p#61qW+|UyWMmvA|4D0JvQ0)_zIn`L z>gN2r{UCE8KN`kB-Y*~9Zqkr`J0lapK&~~Km+dRc#p5aYZ#GM=IT7KRZp#{$TWtt+ zaZ2L+9uZ{YEjfCU_rBLw$qpuZBrfi6V5?z8C*hpKKYontE zX&_O^(d$du#S(_$P*XiI&3I5RvQx{j6?|cH6jVJNgDX9tv9+i%_~FU^PHPnt6O;5f zW+XGc&^ehAV(_yT2C7&qB~8W<^C8YZYyQM%U8$gVH%|=|txb6%<n6L-NMRM!nR1d%S z?b~vEHWqO81aMp}h%=V?SXx3t!i$cd%M@pi2d_uucE1<9ZQ)x0^zbb~yh7_UW%ih} zS}@*_-}~_~Daq@dy5spH7ZeX*AJ%Laeap4gY+CNh$jbgOR#QnL%}l0sS@_Nlc(pQR zR~#(wVq;?y;xv_OQMQl!>-o5lL&(T@;qGO-mapJR2)kqBAmGeZsK*p(+o{iK|5qkWpAATNeB?5_QXEL>CJr z<-qyP+PvSLszS?b0*ytw3g41onNfHGqJ^uip@&^!Ekz0wrENde1L ze>lZ}4U1~f?Kqw+v9r()BV(m^xhQ*IzM)F^&JK{m_djfjhSPYB+(fxB4YJ#JToiA| zVLkc!=2yU6DnhwVPs{@g5ruI@>$=j<;<~phILe2Q3EB~DgW$5N>|%DWqfruW3qqnj zfbfeM2Gj|4K-d*{UiA1PoR>uXq5(d!opKz%FHE