-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Daniel Moessner edited this page Mar 1, 2020
·
5 revisions
-
This is to document the steps needed to install OCP 4.3 on AWS in a disconnected fashion using a squid proxy
-
Setup is as follows:
- VPC with 1 public facing network and 3 private networks
- host in the public network will be called bastion/proxy
- ssh accessible via internet
- squid installed on it
- OCP deployed in private networks
- no direct internet access
- need to use proxy on bastion/proxy server
-
basis is documentation and this bug
-
current flaws found
-
https proxy needs to be set although being similar to http proxy and no cert. Not setting https proxy will lead to an error when running
oc adm upgradewarning: Cannot display available updates: Reason: RemoteFailed Message: Unable to retrieve available updates: Get https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.3&id=efe7c267-48a0-42cf-a23c-89f2c31889f2&version=4.3.1: dial tcp 18.207.44.243:443: connect: connection timed out -
additional endpoints need to be created and added to the private subnets
-
com.amazonaws.< region >.elasticloadbalancing
-
com.amazonaws.< region >.ec2
at least the ec2 endpoint needs also be added to the security group bastion is in as otherwise this cannot communicate with AWS api and openshift-installer will stuck / aws cli will also not work
-
-
-
open topics
- strip down proxy to be more restrictive
- move from internet accessible bastion to vpn connected vpc
- some woarkload testing
- we set up aws cli locally on the client machine, in this case laptop, to create the infrastructure on AWS step by step
<laptop>$ sudo -i
<laptop># curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
<laptop># unzip awscli-bundle.zip
<laptop># ./awscli-bundle/install -i /usr/local/aws -b /bin/aws
<laptop># /bin/aws --version
<laptop># logout
<laptop>$
<laptop>$ mkdir $HOME/.aws
<laptop>$ export AWSKEY= <redacted>
<laptop>$ export AWSSECRETKEY= <redacted>
<laptop>$ export REGION=eu-west-1
<laptop>$ cat << EOF >> $HOME/.aws/credentials
[default]
aws_access_key_id = ${AWSKEY}
aws_secret_access_key = ${AWSSECRETKEY}
region = $REGION
EOF
<laptop>$
<laptop>$ aws sts get-caller-identity
- now we can start creating aws objects from the local machine
<laptop>$ aws ec2 create-vpc --cidr-block 192.168.0.0/16
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne-vpc --resources vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 modify-vpc-attribute --enable-dns-hostnames --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 modify-vpc-attribute --enable-dns-support --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 describe-vpcs --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 describe-dhcp-options --dhcp-options-ids dopt-7d4aee18
- first the public facing network
<laptop>$ aws ec2 create-subnet --availability-zone eu-west-1a --cidr-block 192.168.0.0/24 --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-public-1a --resources subnet-02ed1fe8249e9af75
- then the 3 private networks
<laptop>$ aws ec2 create-subnet --availability-zone eu-west-1a --cidr-block 192.168.10.0/24 --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-private-1a --resources subnet-0ea3ec602f2e0baee
<laptop>$ aws ec2 create-subnet --availability-zone eu-west-1b --cidr-block 192.168.11.0/24 --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-private-1b --resources subnet-0b032d4c5b631a6ea
<laptop>$ aws ec2 create-subnet --availability-zone eu-west-1c --cidr-block 192.168.12.0/24 --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-private-1c --resources subnet-0e6e892f5f01f081b
<laptop>$ aws ec2 create-internet-gateway
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-igw --resources igw-0545e4d6fb607074c
<laptop>$ aws ec2 attach-internet-gateway --internet-gateway-id igw-0545e4d6fb607074c --vpc-id vpc-0f0c0f81d7f588d51
- public routing table
<laptop>$ aws ec2 create-route-table --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-public-rtb --resources rtb-0e200bbeb86d915ec
- private routing table
<laptop>$ aws ec2 create-route-table --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-private-rtb --resources rtb-09ced81d48f9961c8
- link internet gateway to routing table to be associated with public subnet
<laptop>$ aws ec2 create-route --destination-cidr-block 0.0.0.0/0 --gateway-id igw-0545e4d6fb607074c --route-table-id rtb-0e200bbeb86d915ec
<laptop>$ aws ec2 describe-route-tables --route-table-id rtb-0e200bbeb86d915ec
<laptop>$ aws ec2 describe-route-tables --route-table-id rtb-09ced81d48f9961c8
<laptop>$ aws ec2 describe-subnets --filters --filters "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51" --output text
<laptop>$ aws ec2 describe-subnets --filters --filters "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51" --query 'Subnets[*].[AvailabilityZone,CidrBlock,SubnetId]' --output table
- associate private subnets
<laptop>$ aws ec2 associate-route-table --subnet-id subnet-02ed1fe8249e9af75 --route-table-id rtb-0e200bbeb86d915ec
- associate private subnets
<laptop>$ aws ec2 associate-route-table --route-table-id rtb-09ced81d48f9961c8 --subnet-id subnet-0ea3ec602f2e0baee
<laptop>$ aws ec2 associate-route-table --route-table-id rtb-09ced81d48f9961c8 --subnet-id subnet-0b032d4c5b631a6ea
<laptop>$ aws ec2 associate-route-table --route-table-id rtb-09ced81d48f9961c8 --subnet-id subnet-0e6e892f5f01f081b
<laptop>$ aws ec2 describe-route-tables --filter "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51" --output text
<laptop>$ aws ec2 describe-route-tables --filter "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51" --output table
<default-rtb-assoc> <public rtb>
<laptop>$ aws ec2 replace-route-table-association --association-id rtbassoc-0e8d05a2e6dc2ceab --route-table-id rtb-0e200bbeb86d915ec
<laptop>$ aws ec2 delete-route-table --route-table-id rtb-00dbbea46ac28b350
<laptop>$ aws ec2 describe-route-tables --filter "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51" --output table
- already there:
<laptop>$ aws ec2 describe-network-acls --filter "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51"
- we could make it even more restrictive for private nets, for now outbound is allowed all all
- possibly revisit later
- creating 2 security groups
-
public one for the bastion and which we need to assign to an endpoint as well to run aws cli and ocp installer from it (otherwise it fails)
-
private one which is there for the private nets too test proxy and is not needed for the actual setup
-
public-sg
-
<laptop>$ aws ec2 create-security-group --description dmoessne-public-sg --group-name dmoessne-public-sg --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 authorize-security-group-ingress --group-id sg-019fa2236a7bade4d --protocol tcp --port 22 --cidr 0.0.0.0/0
<laptop>$ aws ec2 authorize-security-group-ingress --group-id sg-019fa2236a7bade4d --protocol all --cidr 192.168.0.0/16
<laptop>$ aws ec2 create-tags --resources sg-019fa2236a7bade4d --tags Key=Name,Value=dmoessne2-sg-public
<laptop>$ aws ec2 describe-security-groups --group-id sg-019fa2236a7bade4d
- private sg
<laptop>$ aws ec2 create-security-group --description dmoessne-private-sg --group-name dmoessne-private-sg --vpc-id vpc-0f0c0f81d7f588d51
<laptop>$ aws ec2 create-tags --resources sg-042ae430e9ddc97c3 --tags Key=Name,Value=dmoessne2-sg-private
<laptop>$ aws ec2 authorize-security-group-ingress --group-id sg-042ae430e9ddc97c3 --protocol all --cidr 192.168.0.0/16
<laptop>$ aws ec2 authorize-security-group-egress --group-id sg-042ae430e9ddc97c3 --protocol all --cidr 192.168.0.0/16
<laptop>$ aws ec2 describe-security-groups --group-id sg-042ae430e9ddc97c3
<laptop>$ aws ec2 revoke-security-group-egress --group-id sg-042ae430e9ddc97c3 --protocol all --cidr 0.0.0.0/0
<laptop>$ aws ec2 describe-security-groups --group-id sg-042ae430e9ddc97c3
-
docs only mention as of yet s3 endpoint
-
ec2 and elasticloadbalancing are needed as otherwise worker nodes will not be created
- both need to be assigned to 3 private subnets
- if not specifically changed/configured default sg will be automatically assigned
- especially for ec2 endpoint it is important to specify the public security group as otherwise aws cli and openshift-installer will not be able to communicate with AWS api and nothing will happen at all
- elasticloadbalancing ep has no such requirement according to testing, however as I find the default sg too open I also assigne the public sg I created
-
all endpoints can be viewd by:
<laptop>$ aws ec2 describe-vpc-endpoint-services
- create s3 enpoint:
<laptop>$ aws ec2 create-vpc-endpoint --vpc-endpoint-type Gateway --vpc-id vpc-0f0c0f81d7f588d51 --service-name com.amazonaws.eu-west-1.s3 --route-table-ids rtb-09ced81d48f9961c8 --no-private-dns-enabled
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-EP-s3 --resources vpce-0e49c5e57d4485442
- create elasticloadbalancing endpoint
<laptop>$ aws ec2 create-vpc-endpoint --vpc-endpoint-type Interface --vpc-id vpc-0f0c0f81d7f588d51 --service-name com.amazonaws.eu-west-1.elasticloadbalancing --subnet-ids "subnet-0ea3ec602f2e0baee" "subnet-0b032d4c5b631a6ea" "subnet-0e6e892f5f01f081b" --private-dns-enabled
- we check it and replace the default security group by the public one created manually
- as at least one security group must be present we add first add the public one and then remove the default one
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-EP-elb --resources vpce-09c6bccf9ce2c5bf3
<laptop>$ aws ec2 describe-vpc-endpoints --filter "Name=vpc-id,Values=vpc-0f0c0f81d7f588d51"
- replace default sg by public one:
<laptop>$ aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-09c6bccf9ce2c5bf3 --add-security-group-ids sg-019fa2236a7bade4d
<laptop>$ aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-09c6bccf9ce2c5bf3 --remove-security-group-ids sg-0fb7986c54dd48071
- create ec2 endpoint
- this time we simply add the right security group at creation time
<laptop>$ aws ec2 create-vpc-endpoint --vpc-endpoint-type Interface --vpc-id vpc-0f0c0f81d7f588d51 --service-name com.amazonaws.eu-west-1.ec2 --security-group-ids sg-019fa2236a7bade4d --subnet-ids "subnet-0ea3ec602f2e0baee" "subnet-0b032d4c5b631a6ea" "subnet-0e6e892f5f01f081b" --private-dns-enabled
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-EP-ec2 --resources vpce-04c23c388f2fef2da
- bastion instance (RHEL8) which will be publicly accessible via ssh and we configure proxy on later
<laptop>$ aws ec2 run-instances --image-id ami-04facb3ed127a2eb6 --count 1 --instance-type t2.medium --key-name dmoessne-key --security-group-ids sg-019fa2236a7bade4d --subnet-id subnet-02ed1fe8249e9af75 --associate-public-ip-address
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-public-bastion --resources i-0def11d77f3d7ab4f
- instance in private network to validate network has no internet access and proxy is working
<laptop>$ aws ec2 run-instances --image-id ami-0e61341fa75fcaa18 --count 1 --instance-type t2.micro --key-name dmoessne-key --security-group-ids sg-042ae430e9ddc97c3 --subnet-id subnet-0e6e892f5f01f081b
<laptop>$ aws ec2 create-tags --tags Key=Name,Value=dmoessne2-private-proxy-test --resources i-03a7e5d488fcbb8c8
- we want to connect to the instance with the assigned dns name
<laptop>$ aws ec2 describe-instances --filters "Name=tag:Name,Values=dmoessne2*" --output text
-
set up aws cli, ocp related/needed tools to maintain and deploy ocp as well as a very simple proxy
-
login to bastion in public network
<laptop>$ ssh -i ~/.ssh/dmoessne-key.pem ec2-user@ec2-<....>.eu-west-1.compute.amazonaws.com
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ sudo -i
[root@ip-192-168-0-237 ~]# yum install -y firewalld squid vim wget unzip openssl python3 bind-utils
[root@ip-192-168-0-237 ~]# alternatives --set python /usr/bin/python3
[root@ip-192-168-0-237 ~]#
- enable FW and enable squid ports
[root@ip-192-168-0-237 ~]# systemctl enable firewalld --now
[root@ip-192-168-0-237 ~]# firewall-cmd --add-port=3128/tcp --permanent
success
[root@ip-192-168-0-237 ~]# firewall-cmd --add-port=3128/tcp
- set up squid (very simple)
[root@ip-192-168-0-237 ~]# cp /etc/squid/squid.conf /etc/squid/squid.conf.orig
[root@ip-192-168-0-237 ~]# vim /etc/squid/squid.conf
[root@ip-192-168-0-237 ~]# cat /etc/squid/squid.conf
acl SSL_ports port 443
# Ports where clients can connect to.
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
# if connection is not to any of this port, Sqiud rejects. otherwise check the next rule.
http_access deny !Safe_ports
# Squid cache manager app
http_access allow localhost manager
http_access deny manager
# localhost is allowed. if source is not localhost, squid checks the next rule
http_access allow localhost
# Simply allow everyone with everything - we are trusting everybody :D
http_access allow all
# IMPORTANT LINE: deny anything that's not allowed above
#http_access deny all
# listen on this port as a proxy
http_port 3128
# memory settings
cache_mem 512 MB
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 # refresh_pattern [-i] regex min percent max [options]
# here, . means 'any link'. Cache for at least 0, at most 20160 minutes, ot 50% of its age since 'last-modified' header.
refresh_pattern . 0 50% 20160
# delete x-forwarded-for header in requests (anonymize them)
forwarded_for delete
[root@ip-192-168-0-237 ~]#
[root@ip-192-168-0-237 ~]# systemctl enable squid --now
[root@ip-192-168-0-237 ~]#
- aws cli/ocp tools install and config
[root@ip-192-168-0-237 ~]# curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
[root@ip-192-168-0-237 ~]# unzip awscli-bundle.zip
[root@ip-192-168-0-237 ~]# ./awscli-bundle/install -i /usr/local/aws -b /bin/aws
[root@ip-192-168-0-237 ~]# /bin/aws --version
[root@ip-192-168-0-237 ~]# wget -qO - https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.3.1.tar.gz | tar xfz - -C /usr/bin/
[root@ip-192-168-0-237 ~]# wget -qO - https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.3.1.tar.gz | tar xfz - -C /usr/bin/
[root@ip-192-168-0-237 ~]#
[root@ip-192-168-0-237 ~]# oc completion bash >/etc/bash_completion.d/openshift
[root@ip-192-168-0-237 ~]# oc version
[root@ip-192-168-0-237 ~]# openshift-install version
[root@ip-192-168-0-237 ~]# logout
[ec2-user@ip-192-168-0-237 ~]$
- configure aws tools as well on bastion as ec2 user
[ec2-user@ip-192-168-0-237 ~]$ mkdir $HOME/.aws
[ec2-user@ip-192-168-0-237 ~]$ export AWSKEY=<redacted>
[ec2-user@ip-192-168-0-237 ~]$ export AWSSECRETKEY=<redacted>
[ec2-user@ip-192-168-0-237 ~]$ export REGION=eu-west-1
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ cat << EOF >> $HOME/.aws/credentials
[default]
aws_access_key_id = ${AWSKEY}
aws_secret_access_key = ${AWSSECRETKEY}
region = $REGION
EOF
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ aws sts get-caller-identity
[ec2-user@ip-192-168-0-237 ~]$
- create ssh key (basically follow docs)
[ec2-user@ip-192-168-0-237 ~]$ ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa
[ec2-user@ip-192-168-0-237 ~]$ cat .ssh/id_rsa.pub
- check ip for proxy
[ec2-user@ip-192-168-0-237 ~]$ ip a|grep 192
inet 192.168.0.237/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0
[ec2-user@ip-192-168-0-237 ~]$
- configure install-config.yaml
- mind the
publish: Internal - mind, even if docs seem to imply there is no need to configure a https proxy in case it is the same as the http proxy, tests showed updates are not working. So configure both proxies, even if they are the same!
- mind the
[ec2-user@ip-192-168-0-237 ~]$ vim install-config.yaml
[ec2-user@ip-192-168-0-237 ~]$ cat install-config.yaml
apiVersion: v1
baseDomain: dmoessne2.csa2-lab.org
proxy:
httpProxy: http://192.168.0.237:3128
httpsProxy: http://192.168.0.237:3128
noProxy: csa2-lab.org
controlPlane:
hyperthreading: Enabled
name: master
platform:
aws:
zones:
- eu-west-1a
- eu-west-1b
- eu-west-1c
rootVolume:
iops: 4000
size: 500
type: io1
type: m5.xlarge
replicas: 3
compute:
- hyperthreading: Enabled
name: worker
platform:
aws:
rootVolume:
iops: 2000
size: 500
type: io1
type: m5.xlarge
zones:
- eu-west-1a
- eu-west-1b
- eu-west-1c
replicas: 3
metadata:
name: test-cluster
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineCIDR: 192.168.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: eu-west-1
userTags:
adminContact: dmoessne
costCenter: 118
subnets:
- subnet-0ea3ec602f2e0baee
- subnet-0b032d4c5b631a6ea
- subnet-0e6e892f5f01f081b
pullSecret: '....'
fips: false
sshKey: '...'
publish: Internal
[ec2-user@ip-192-168-0-237 ~]$
- login to bastion and from there to the test server in one of the private networks and validate proxy is working
- vm created earlier in priv subnet: 192.168.12.227
- proxy server: 192.168.0.237:3128
-copy key to VM to be able to connect to proxy
<laptop>$ scp -i ~/.ssh/dmoessne-key.pem ~/.ssh/dmoessne-key.pem ec2-user@ec2-<....>.eu-west-1.compute.amazonaws.com:~/.ssh/
- connect to VM in priv subnet
[ec2-user@ip-192-168-0-237 ~]$ ssh -i ~/.ssh/dmoessne-key.pem 192.168.12.227
- check DNS is working
[ec2-user@ip-192-168-12-227 ~]$ nslookup google.com
- validate no connection is possible
[ec2-user@ip-192-168-12-227 ~]$ ping google.com
[ec2-user@ip-192-168-12-227 ~]$ curl -vv google.com
[ec2-user@ip-192-168-12-227 ~]$ curl -vv 74.125.193.138
- set proxy and retest
[ec2-user@ip-192-168-12-227 ~]$ export http_proxy=http://192.168.0.237:3128
[ec2-user@ip-192-168-12-227 ~]$ export https_proxy=$http_proxy
[ec2-user@ip-192-168-12-227 ~]$ curl -vv google.com
[ec2-user@ip-192-168-12-227 ~]$ curl -vv https://google.com
- logout again
[ec2-user@ip-192-168-12-227 ~]$ logout
- finally, let's deploy and see where we end up
[ec2-user@ip-192-168-0-237 ~]$ mkdir ~/cluster
[ec2-user@ip-192-168-0-237 ~]$ cp install-config.yaml ~/cluster
[ec2-user@ip-192-168-0-237 ~]$ openshift-install create cluster --dir=./cluster --log-level debug
[...]
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/ec2-user/cluster/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.test-cluster.dmoessne2.csa2-lab.org
INFO Login to the console with user: kubeadmin, password: <redacted>
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$
- at this point the cluster is deployed and we can login and validate the cluster (see below)
- in case it times out at this stage as not all operators are coming up (aut, ingress,..) we can still login via
export KUBECONFIG=/home/ec2-user/cluster/auth/kubeconfigand check what is going on. You might want to check if worker nodes have been created and if not, check ec32 and loadbalancer endpoints
- in case it times out at this stage as not all operators are coming up (aut, ingress,..) we can still login via
- check cluster
[ec2-user@ip-192-168-0-237 ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-10-124.eu-west-1.compute.internal Ready worker 11m v1.16.2
ip-192-168-10-174.eu-west-1.compute.internal Ready master 20m v1.16.2
ip-192-168-11-212.eu-west-1.compute.internal Ready master 20m v1.16.2
ip-192-168-11-231.eu-west-1.compute.internal Ready worker 11m v1.16.2
ip-192-168-12-236.eu-west-1.compute.internal Ready worker 11m v1.16.2
ip-192-168-12-250.eu-west-1.compute.internal Ready master 19m v1.16.2
[ec2-user@ip-192-168-0-237 ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.1 True False False 4m2s
cloud-credential 4.3.1 True False False 20m
cluster-autoscaler 4.3.1 True False False 13m
console 4.3.1 True False False 6m55s
dns 4.3.1 True False False 16m
image-registry 4.3.1 True False False 8m32s
ingress 4.3.1 True False False 9m23s
insights 4.3.1 True False False 18m
kube-apiserver 4.3.1 True False False 16m
kube-controller-manager 4.3.1 True False False 15m
kube-scheduler 4.3.1 True False False 16m
machine-api 4.3.1 True False False 17m
machine-config 4.3.1 True False False 17m
marketplace 4.3.1 True False False 13m
monitoring 4.3.1 True False False 6m42s
network 4.3.1 True False False 18m
node-tuning 4.3.1 True False False 14m
openshift-apiserver 4.3.1 True False False 12m
openshift-controller-manager 4.3.1 True False False 16m
openshift-samples 4.3.1 True False False 12m
operator-lifecycle-manager 4.3.1 True False False 17m
operator-lifecycle-manager-catalog 4.3.1 True False False 17m
operator-lifecycle-manager-packageserver 4.3.1 True False False 15m
service-ca 4.3.1 True False False 18m
service-catalog-apiserver 4.3.1 True False False 14m
service-catalog-controller-manager 4.3.1 True False False 14m
storage 4.3.1 True False False 13m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machines -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
test-cluster-wmd2d-master-0 Running m5.xlarge eu-west-1 eu-west-1a 20m
test-cluster-wmd2d-master-1 Running m5.xlarge eu-west-1 eu-west-1b 20m
test-cluster-wmd2d-master-2 Running m5.xlarge eu-west-1 eu-west-1c 20m
test-cluster-wmd2d-worker-eu-west-1a-bk768 Running m5.xlarge eu-west-1 eu-west-1a 17m
test-cluster-wmd2d-worker-eu-west-1b-tnrrl Running m5.xlarge eu-west-1 eu-west-1b 17m
test-cluster-wmd2d-worker-eu-west-1c-nm422 Running m5.xlarge eu-west-1 eu-west-1c 17m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 1 1 1 1 20m
test-cluster-wmd2d-worker-eu-west-1b 1 1 1 1 20m
test-cluster-wmd2d-worker-eu-west-1c 1 1 1 1 20m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc adm upgrade
Cluster version is 4.3.1
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.
[ec2-user@ip-192-168-0-237 ~]$
-
if there is a error not able to reach the update server, check that you have configured both http and https proxy even if they are the same! (
oc edit proxy/cluster- and then waitr some time until the config change has propagated through the cluster) -
to 'force' an update we change to candidate channel. Never do that for production and even be cautious for test environments!
[ec2-user@ip-192-168-0-237 ~]$ oc edit clusterversion
clusterversion.config.openshift.io/version edited
[ec2-user@ip-192-168-0-237 ~]$
--> set to candidate channel - by no means for production usage !
[ec2-user@ip-192-168-0-237 ~]$ oc adm upgrade
Cluster version is 4.3.1
Updates:
VERSION IMAGE
4.3.2 quay.io/openshift-release-dev/ocp-release@sha256:cadf53e7181639f6cc77d2430339102db2908de330210c1ff8c7a7dc1cb0e550
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc adm upgrade --to-latest
Updating to latest version 4.3.2
[ec2-user@ip-192-168-0-237 ~]$
- checking again later on:
- mind in between 4.3.3 has made it to the candidate channel ...
[ec2-user@ip-192-168-0-237 ~]$ oc adm upgrade
Cluster version is 4.3.2
Updates:
VERSION IMAGE
4.3.3 quay.io/openshift-release-dev/ocp-release@sha256:9b8708b67dd9b7720cb7ab3ed6d12c394f689cc8927df0e727c76809ab383f44
[ec2-user@ip-192-168-0-237 ~]$
- check cluster is updated
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 1 1 1 1 78m
test-cluster-wmd2d-worker-eu-west-1b 1 1 1 1 78m
test-cluster-wmd2d-worker-eu-west-1c 1 1 1 1 78m
[ec2-user@ip-192-168-0-237 ~]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.3.2 True False False 61m
cloud-credential 4.3.2 True False False 78m
cluster-autoscaler 4.3.2 True False False 70m
console 4.3.2 True False False 26m
dns 4.3.2 True False False 74m
image-registry 4.3.2 True False False 26m
ingress 4.3.2 True False False 25m
insights 4.3.2 True False False 76m
kube-apiserver 4.3.2 True False False 74m
kube-controller-manager 4.3.2 True False False 73m
kube-scheduler 4.3.2 True False False 74m
machine-api 4.3.2 True False False 75m
machine-config 4.3.2 True False False 75m
marketplace 4.3.2 True False False 26m
monitoring 4.3.2 True False False 40m
network 4.3.2 True False False 76m
node-tuning 4.3.2 True False False 26m
openshift-apiserver 4.3.2 True False False 19m
openshift-controller-manager 4.3.2 True False False 74m
openshift-samples 4.3.2 True False False 46m
operator-lifecycle-manager 4.3.2 True False False 75m
operator-lifecycle-manager-catalog 4.3.2 True False False 75m
operator-lifecycle-manager-packageserver 4.3.2 True False False 21m
service-ca 4.3.2 True False False 76m
service-catalog-apiserver 4.3.2 True False False 72m
service-catalog-controller-manager 4.3.2 True False False 72m
storage 4.3.2 True False False 46m
[ec2-user@ip-192-168-0-237 ~]$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-10-124.eu-west-1.compute.internal Ready worker 71m v1.16.2 192.168.10.124 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
ip-192-168-10-174.eu-west-1.compute.internal Ready master 80m v1.16.2 192.168.10.174 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
ip-192-168-11-212.eu-west-1.compute.internal Ready master 80m v1.16.2 192.168.11.212 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
ip-192-168-11-231.eu-west-1.compute.internal Ready worker 71m v1.16.2 192.168.11.231 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
ip-192-168-12-236.eu-west-1.compute.internal Ready worker 71m v1.16.2 192.168.12.236 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
ip-192-168-12-250.eu-west-1.compute.internal Ready master 79m v1.16.2 192.168.12.250 <none> Red Hat Enterprise Linux CoreOS 43.81.202002110953.0 (Ootpa) 4.18.0-147.5.1.el8_1.x86_64 cri-o://1.16.3-19.dev.rhaos4.3.git6c1f4bd.el8
[ec2-user@ip-192-168-0-237 ~]$
- let's test scaling mind if you have an issue here, check again endpoints
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 1 1 1 1 81m
test-cluster-wmd2d-worker-eu-west-1b 1 1 1 1 81m
test-cluster-wmd2d-worker-eu-west-1c 1 1 1 1 81m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=2 test-cluster-wmd2d-worker-eu-west-1a
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1a scaled
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=2 test-cluster-wmd2d-worker-eu-west-1b
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1b scaled
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=2 test-cluster-wmd2d-worker-eu-west-1c
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1c scaled
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 2 2 1 1 81m
test-cluster-wmd2d-worker-eu-west-1b 2 2 1 1 81m
test-cluster-wmd2d-worker-eu-west-1c 2 2 1 1 81m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machine -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
test-cluster-wmd2d-master-0 Running m5.xlarge eu-west-1 eu-west-1a 81m
test-cluster-wmd2d-master-1 Running m5.xlarge eu-west-1 eu-west-1b 81m
test-cluster-wmd2d-master-2 Running m5.xlarge eu-west-1 eu-west-1c 81m
test-cluster-wmd2d-worker-eu-west-1a-bgt5t Provisioning m5.xlarge eu-west-1 eu-west-1a 19s
test-cluster-wmd2d-worker-eu-west-1a-bk768 Running m5.xlarge eu-west-1 eu-west-1a 78m
test-cluster-wmd2d-worker-eu-west-1b-tnrrl Running m5.xlarge eu-west-1 eu-west-1b 78m
test-cluster-wmd2d-worker-eu-west-1b-zvvlq Provisioning m5.xlarge eu-west-1 eu-west-1b 17s
test-cluster-wmd2d-worker-eu-west-1c-nm422 Running m5.xlarge eu-west-1 eu-west-1c 78m
test-cluster-wmd2d-worker-eu-west-1c-vq64c Provisioning m5.xlarge eu-west-1 eu-west-1c 12s
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-10-124.eu-west-1.compute.internal Ready worker 73m v1.16.2
ip-192-168-10-174.eu-west-1.compute.internal Ready master 82m v1.16.2
ip-192-168-11-212.eu-west-1.compute.internal Ready master 82m v1.16.2
ip-192-168-11-231.eu-west-1.compute.internal Ready worker 73m v1.16.2
ip-192-168-12-236.eu-west-1.compute.internal Ready worker 72m v1.16.2
ip-192-168-12-250.eu-west-1.compute.internal Ready master 81m v1.16.2
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-10-124.eu-west-1.compute.internal Ready worker 84m v1.16.2
ip-192-168-10-174.eu-west-1.compute.internal Ready master 92m v1.16.2
ip-192-168-10-41.eu-west-1.compute.internal Ready worker 6m54s v1.16.2
ip-192-168-11-116.eu-west-1.compute.internal Ready worker 4m23s v1.16.2
ip-192-168-11-212.eu-west-1.compute.internal Ready master 92m v1.16.2
ip-192-168-11-231.eu-west-1.compute.internal Ready worker 83m v1.16.2
ip-192-168-12-236.eu-west-1.compute.internal Ready worker 83m v1.16.2
ip-192-168-12-250.eu-west-1.compute.internal Ready master 92m v1.16.2
ip-192-168-12-94.eu-west-1.compute.internal Ready worker 6m33s v1.16.2
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 2 2 2 2 92m
test-cluster-wmd2d-worker-eu-west-1b 2 2 2 2 92m
test-cluster-wmd2d-worker-eu-west-1c 2 2 2 2 92m
[ec2-user@ip-192-168-0-237 ~]$ oc get machine -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
test-cluster-wmd2d-master-0 Running m5.xlarge eu-west-1 eu-west-1a 92m
test-cluster-wmd2d-master-1 Running m5.xlarge eu-west-1 eu-west-1b 92m
test-cluster-wmd2d-master-2 Running m5.xlarge eu-west-1 eu-west-1c 92m
test-cluster-wmd2d-worker-eu-west-1a-bgt5t Running m5.xlarge eu-west-1 eu-west-1a 11m
test-cluster-wmd2d-worker-eu-west-1a-bk768 Running m5.xlarge eu-west-1 eu-west-1a 89m
test-cluster-wmd2d-worker-eu-west-1b-tnrrl Running m5.xlarge eu-west-1 eu-west-1b 89m
test-cluster-wmd2d-worker-eu-west-1b-zvvlq Running m5.xlarge eu-west-1 eu-west-1b 11m
test-cluster-wmd2d-worker-eu-west-1c-nm422 Running m5.xlarge eu-west-1 eu-west-1c 89m
test-cluster-wmd2d-worker-eu-west-1c-vq64c Running m5.xlarge eu-west-1 eu-west-1c 11m
[ec2-user@ip-192-168-0-237 ~]$
- scale down again
[ec2-user@ip-192-168-0-237 ~]$ oc get machine -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
test-cluster-wmd2d-master-0 Running m5.xlarge eu-west-1 eu-west-1a 92m
test-cluster-wmd2d-master-1 Running m5.xlarge eu-west-1 eu-west-1b 92m
test-cluster-wmd2d-master-2 Running m5.xlarge eu-west-1 eu-west-1c 92m
test-cluster-wmd2d-worker-eu-west-1a-bgt5t Running m5.xlarge eu-west-1 eu-west-1a 11m
test-cluster-wmd2d-worker-eu-west-1a-bk768 Running m5.xlarge eu-west-1 eu-west-1a 89m
test-cluster-wmd2d-worker-eu-west-1b-tnrrl Running m5.xlarge eu-west-1 eu-west-1b 89m
test-cluster-wmd2d-worker-eu-west-1b-zvvlq Running m5.xlarge eu-west-1 eu-west-1b 11m
test-cluster-wmd2d-worker-eu-west-1c-nm422 Running m5.xlarge eu-west-1 eu-west-1c 89m
test-cluster-wmd2d-worker-eu-west-1c-vq64c Running m5.xlarge eu-west-1 eu-west-1c 11m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=1 test-cluster-wmd2d-worker-eu-west-1a
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1a scaled
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=1 test-cluster-wmd2d-worker-eu-west-1b
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1b scaled
[ec2-user@ip-192-168-0-237 ~]$ oc -n openshift-machine-api scale machineset --replicas=1 test-cluster-wmd2d-worker-eu-west-1c
machineset.machine.openshift.io/test-cluster-wmd2d-worker-eu-west-1c scaled
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-10-124.eu-west-1.compute.internal Ready worker 98m v1.16.2
ip-192-168-10-174.eu-west-1.compute.internal Ready master 106m v1.16.2
ip-192-168-11-116.eu-west-1.compute.internal Ready worker 18m v1.16.2
ip-192-168-11-212.eu-west-1.compute.internal Ready master 106m v1.16.2
ip-192-168-12-250.eu-west-1.compute.internal Ready master 106m v1.16.2
ip-192-168-12-94.eu-west-1.compute.internal Ready worker 21m v1.16.2
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machine -n openshift-machine-api
NAME PHASE TYPE REGION ZONE AGE
test-cluster-wmd2d-master-0 Running m5.xlarge eu-west-1 eu-west-1a 106m
test-cluster-wmd2d-master-1 Running m5.xlarge eu-west-1 eu-west-1b 106m
test-cluster-wmd2d-master-2 Running m5.xlarge eu-west-1 eu-west-1c 106m
test-cluster-wmd2d-worker-eu-west-1a-bk768 Running m5.xlarge eu-west-1 eu-west-1a 103m
test-cluster-wmd2d-worker-eu-west-1b-zvvlq Running m5.xlarge eu-west-1 eu-west-1b 25m
test-cluster-wmd2d-worker-eu-west-1c-vq64c Running m5.xlarge eu-west-1 eu-west-1c 25m
[ec2-user@ip-192-168-0-237 ~]$
[ec2-user@ip-192-168-0-237 ~]$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
test-cluster-wmd2d-worker-eu-west-1a 1 1 1 1 107m
test-cluster-wmd2d-worker-eu-west-1b 1 1 1 1 107m
test-cluster-wmd2d-worker-eu-west-1c 1 1 1 1 107m
[ec2-user@ip-192-168-0-237 ~]$