Lab Length: Long
The goal of this lab is to learn how you can use a combination of Red Hat Ansible Tower and Red Hat Satellite to implement an automated patching strategy to automatically update and patch host systems at scale in different lifecycle environments. Specifically, you will learn how you can automatically apply patches to host systems and how to automate the tasks that have to happen behind the scenes to manage content in Red Hat Satellite.
Gartner found that "99% of the vulnerabilities exploited by the end of 2020 will continue to be ones known by security and IT professionals at the time of the incident." By having a consistent and automated patching strategy in place, you have a higher chance of being protected by these known vulnerabilities.
Patching is a necessary part of operating system lifecycle management. However, many organizations are struggling to remain current with the constant release of new patches and updates. At the same time, they are under pressure to provide near 100% availability of key business systems. With a consistent automated patch strategy that is routinely executed on your host systems, patching can run smoothly and in an automated fashion. Automation is a key component of having a successful patching strategy. Automation provides that continuous, daily improvement.
Frequent patching is a crucial component to maintainting a secure environment. It is often avoided, delayed, or heavlily lagging due to the involved manual process required to confidently roll new patch sets out to systems and applications.
In this lab, we will see how utilizing Red Hat Satellite as the content repository and Red Hat Ansible Tower as the automation platform can make patching must less of a burden with automation.
Lab 3.1 Viewing the Red Hat Satellite Pre-Configured Content Views Before Automated Patching of Host Systems
-
On the Red Hat Satellite server (https://sat64-GUID.rhpds.opentlc.com) log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Content → Content Views. Here you will find a content view called RHEL7_Standard. This is used for all base builds of RHEL7.
-
Click on the RHEL7_Standard content view.
-
Once the page loads, click the Versions tab at the top of the frame (if not already there). You will see one version (Version 1.0) associated with all lifecycle environments (RHEL7_Dev, RHEL7_QA, and RHEL7_Prod).
Lab 3.2 Automated Patching and Scanning of Host Systems with Red Hat Ansible Tower and Red Hat Satellite
To patch our systems, we will need to create a new version of the content view that contains any newly synchronized packages. Next, we want to promote that version to the lower environments (such as Dev and QA) to test the patches prior to releasing to higher environments (such as Production). This would all have to be done manually if we did not have automation in place. As the number of content views and environments grows, so does the workload in doing this manually.
In this lab exercise, we will automatically create and promote new content views with our updated patches and packages in Red Hat Satellite automatically using Red Hat Ansible Tower. After that, we will automatically patch our host systems with the new content view and do automated compliance scanning on our host systems as well.
-
On Red Hat Ansible Tower (https://tower-GUID.rhpds.opentlc.com) log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Templates and click the rocket ship next to the job template named PATCHING / 1 - Dev. This will launch the job and we will observe what actions it automates as it runs.
-
Notice how this job kicks off an automation workflow in Red Hat Ansible Tower. This automation workflow in this job will take about 15 minutes to complete. In the meantime, let’s take a deeper look at this automation workflow in Red Hat Ansible Tower to see what’s happening behind the scenes.
-
Notice that this automation workflow has several steps: Publish Content, Promote Content, Recalculate Errata, Install Updates, SCAP Scan, and Schedule Next (which schedules our next patching event). Click on the Expand Output button at the top right to see the full workflow. You can click the Expand Output button again if you want to exit the full workflow view. Also, feel free to click on Details in each of the workflow steps if you want to dive deeper into that particular job template and the automation tasks it is performing.
-
In this automation workflow, after clicking on the Details of each of these workflow steps, you will notice that the Recalculate Errata, Install Updates, and SCAP Scan steps are all run against the foreman_lifecycle_environment_rhel7_dev hosts.
-
Now, let’s find out which hosts are part of foreman_lifecycle_environment_rhel7_dev group.
-
Navigate to Inventories → Satellite Inventory → GROUPS → foreman_lifecycle_environment_rhel7_dev → HOSTS. Notice that there are 3 hosts that are part of the foreman_lifecycle_environment_rhel7_dev group: rhel7-vm3.hosts.example.com, rhel7-vm4.hosts.example.com, and rhel7-vm5.hosts.example.com. That means that the Recalculate Errata, Install Updates, and SCAP Scan job templates will run on these 3 hosts.
-
Let’s go back to our automated patching workflow by clicking on Jobs → PATCHING / 1 - Dev. In this automated patching workflow for our Dev environment, notice that the first step in our automation workflow is Publish Content. This step automates the publishing of a new version of content that has our new package updates and patches that have been released since our first version was created.
-
Go back to the Red Hat Satellite server (https://sat64-GUID.rhpds.opentlc.com) and log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Content → Content Views.
-
Click on the RHEL7_Standard content view.
-
Notice in the Versions tab that a new version is being created. This step of creating and publishing our new content view in Red Hat Satellite may take about 8 minutes to complete.
-
Next, notice that the RHEL7_Dev lifecycle environment is being promoted to use the new version of the content view so that our host in the Dev lifecycle environment will start receiving updates from the newer set of updated packages and patches.
-
Go back to Red Hat Ansible Tower (https://tower-GUID.rhpds.opentlc.com) and log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Jobs and click on your recently launched PATCHING / 1 - Dev job.
-
Notice that the second step of our automated patching workflow is to Promote Content which is why we saw that step execute in Red Hat Satellite previously.
-
Next, notice that the third step in our automated patching workflow is Recalculate Errata. In this step, we scan the hosts for new Errata. This simply updates Red Hat Satellite with the patches missing on the system now that we have a new version of content.
-
Next, notice that the next step of our automated workflow is Install Updates. In this step, Red Hat Ansible Tower will run a
yum updateon the hosts in the Dev lifecycle to install the new content. Click on Details while the Install Updates job is running.
-
Notice that the patching-non-ha.yml playbook is executing on the RHEL 7 Dev Hosts (foreman_lifecycle_environment_rhel7_dev). This playbook will go to each of the hosts in the Dev lifecycle environment, run
yum update, record the packages that have been installed and if a reboot is required based on any of the updated packages, only that host system that requires a reboot will be rebooted. In the Red Hat Ansible Tower log, you can see these tasks being executed. Specifically, notice that we’re gathering facts, checking that the yum utils is installed, checking for updates, and upgrading all packages. This job will take about 4 minutes to complete.
-
Click the back button in your browser to go back and monitor the full automation workflow. Notice that the next two jobs are SCAP Scan and Recalculate Errata which are running in parallel. These next 2 jobs will run in parallel since they are not dependent on each other.
-
The SCAP Scan job will run an OpenSCAP scan on the host systems post updates to provide the latest SCAP compliance report. Specifically, it will run the RHEL7_Standard compliance scan on these hosts in the Dev environment. You can confirm this by clicking on Details in the SCAP Scan workflow step box to look at this job template in more detail. Notice that this SCAP Scan job template is being run against the RHEL7_Standard compliance policy.
-
When the SCAP Scan job completes, you can take a look at the RHEL7_Standard compliance reports in Red Hat Satellite for the three hosts(rhel7-vm3.hosts.example.com, rhel7-vm4.hosts.example.com, and rhel7-vm5.hosts.example.com) in the foreman_lifecycle_environment_rhel7_dev group.
-
On the Red Hat Satellite server (https://sat64-GUID.rhpds.opentlc.com) log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Hosts → Reports.
-
Looking at the list of compliance reports in Red Hat Satellite, notice that there is a RHEL7_Standard compliance report for each of the three hosts(rhel7-vm3.hosts.example.com, rhel7-vm4.hosts.example.com, and rhel7-vm5.hosts.example.com) that are part of the foreman_lifecycle_environment_rhel7_dev group.
-
Go back to your Red Hat Ansible Tower(https://tower-GUID.rhpds.opentlc.com) and log in with admin as the user name and r3dh4t1! as the password (if not already logged in). Don’t forget to replace the GUID with your provided GUID.
-
Navigate to Jobs and click on PATCHING / 1 - Dev job (if not already there).
-
After the SCAP Scan job , notice that the Recalculate Errata job will run. This job will rescan the host again and upload the patch status to Red Hat Satellite.
-
Finally, if all of the previous steps were successful, a schedule will be created in Red Hat Ansible Tower to patch the QA environment 7 days from now.
-
Once the entire automation workflow is complete in Red Hat Ansible Tower, select Schedules from the navigation menu on the left. Then, click on the schedule titled Linux_patching_* The date that you see after Linux_patching will be 7 days from when you ran this PATCHING / 1 - Dev Red Hat Ansible Tower job workflow.
-
Inspect the schedule to take note of the automation workflow it will run and the date that is scheduled for. From this page you can disable the schedule, reschedule the schedule, cancel the schedule, etc. If no changes are made, it will automatically promote and patch your QA environment. Since we do not have 7 days to wait, if you would like to watch the process again, return to the Templates page in Red Hat Ansible Tower and manually run the PATCHING / 2 - QA job template. You will notice that this workflow is similar to the one for our Dev lifecycle environment patch automation workflow. This patch automation workflow for our QA lifecycle environment will aoso promote the new content view, patch the QA host systems, and perform an OpenSCAP compliance scan on the QA host systems against the RHEL7_Standard compliance profile.
NoteWe don’t have to do the Publish Content step in our automation workflow for the QA lifecycle host systems since we’re just moving QA to use the version that we created for our Dev lifecycle environment. As a result, we’re just going to do the Promote Content and Install Updates steps in our patching automation workflow on those QA host systems (in addition to doing the SCAP Scan and Recalculate Errata steps afterwards).
