Custom repos #482

pirat89 · 2020-04-02T03:22:54Z

The PR introduces new way to add custom repositories into the IPU process, with possibility to skip rhsm without need of any devel switch.

The PR covers these possible scenarios:

IPU with subscribed system, but without other custom repositories
IPU with subscribed system, and with additional custom repositories
(but the primary one around RHEL are provided still by RHSM)
IPU with subscribed system, but RHSM should not be used during IPU and only custom repositories should be used
IPU with unsubscribed system, everything provided through custom repositories

To add custom repositories into the IPU process:

use the --enablerepo option
create the repo file in /etc/leapp/files/leapp_upgrade_repositories.repo

Both solutions can be combined (but it is not so much expected).

In case of the --enablerepo option, that one can be specified multiple times and for each obtained repoid is produced the CustomTargetRepository message. The repo has to be already specified under the /etc/yum.repos.d otherwise raised errors are later expected.

The repofile will be automatically parsed during the scan phase and copied later into the containers. It is possible to use the repofile in combination with RHSM (e.g. get RH repositories via RHSM and add additional ones via the repofile) and without RHSM as well. In the second case, you need the leapp build from the PR 622. Actually, it is possible to test it without the new build as well, but part of the functionality (e.g. --no-rhsm option) and compatibility with the original devel solution is kept only with builds from that PR.

With this PR, the RHSM is set into the container mode inside the containers every time - despite the set of --no-rhsm. It's seatbelt to ensure we really will not affect the host by mistake. As well, all calls of yum & dnf includes --disableplugin subscription-manager when RHSM should be skipped. So we will be pretty sure, that yum/dnf will not fail when used for any reason connected to RHSM. Even the repofile should be kept as it is in such mode. Which is what we want.

As well, several other actors and models have been added, to be able to improve checks regarding the official possibility of skip of RHSM. New model:

CustomTargetRepositoryFile - representing information about the custom repository files. It's expected that most of the time, only the default custom repo file could be defined as that's handled automatically. But in case that users will want to use the mechanism for other repository files, it's possible as well. Just the scanning has to be handled on their own. All such files will be copied into the containers automatically

Regarding the checks, I tried to effectively minimalized possibility of raised errors, so I created inhibitor(s) that could reduce number of reported "issues". Better solution will be part of future work. See the issue #486 (not only this problem, but big part of it).

All RHSM related texts have been updated to reflect new possibilities.

Part of the PR is fix of the @with_rhsm decorator. So the tests are not affected now by the environment variables, as really everything..at least the LEAPP_NO_RHSM... is read from the configuration of the actor. (thanks @vinzenz for big help - Vinnie is practically author of that one).

Note: There are still missing many unit-tests. Regarding the situation, I expect to cover everything in following PRs. Now I will spend rather more time with additional testing.

Requries PR #480
Requires Leapp PR oamg/leapp/pull/622

leapp-bot · 2020-04-02T03:23:29Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines, pass tests and linter checks before it can be merged.

If you want to re-run tests or request review, you can use following commands as a comment:

leapp-ci build to run unit tests and copr build
e2e tests to run unit tests, copr build and end-to-end tests (OAMG members only)
review please to notify leapp developers of review request

pirat89 · 2020-04-02T10:24:40Z

.. network issue?.. retesting

pirat89 · 2020-04-02T10:41:32Z

and py2/py3 pylint false possitive.

repos/system_upgrade/el7toel8/actors/targetuserspacecreator/libraries/userspacegen.py

pirat89 · 2020-04-06T23:19:20Z

@oamg/developers some parts of the PR are really bad to see even, but we are lack of time now. For that reason I created issue #486, so we will be able to improve some parts later, properly.

pirat89 · 2020-04-06T23:40:08Z

e2e tests

pirat89 · 2020-04-07T00:02:21Z

e2e tests

repos/system_upgrade/el7toel8/actors/checktargetrepos/libraries/library.py

vinzenz · 2020-04-07T09:58:23Z

Code wise I am ok with it as is. Michal Reznik is testing it, I will rely on his vote for it.

Rezney

Looks good to me, just a couple of text suggestions. Testing it right now...

repos/system_upgrade/el7toel8/actors/checktargetrepos/libraries/library.py

repos/system_upgrade/el7toel8/actors/reportsettargetrelease/libraries/library.py

repos/system_upgrade/el7toel8/libraries/rhsm.py

The development variable LEAPP_DEVEL_SKIP_RHSM is being deprecated and replaced with LEAPP_NO_RHSM, which is going to be officially supported. At the framework level, setting LEAPP_DEVEL_SKIP_RHSM to 1 also sets LEAPP_NO_RHSM to 1. This is intentional not to break existing tests and scripts. Therefore, it makes sense to only check for the latter one in tests. Bump leapp-framework requirement to 1.2 regarding the change in the leapp.

In case the system is subscribed but user wants to use own repositories for the IPU, rhsm on the host could be affected when run any yum/dnf commands inside the container. For that reason, set the container mode for rhsm everytime, despite the skip_rhsm().

The actor scan the custom repo file that user could use to define own repositories that should be used during the IPU as well. The expected path of the file is: /etc/leapp/files/leapp_upgrade_repositories.repo These repositories will be used automatically for the IPU despite the enable/disable settings. It's required the file and defined repositories has to be valid. If the file doesn't exist, nothing happens.

The model is used to share information about custom repo files that should be extra handled during the IPU. Basically we have just one file that we want to use for these purposes now, but I haven't found good library for the constant and I haven't wanted to duplicate the string several times. As well, maybe someone will want to handle own repofile. For that reasons, I rather created the new model. Note: Feel free to drop it and hardcode the path.

…ration Direct reading of LEAPP & LEAPP_DEVEL evars in actors should happen from the IPUConfig using the leapp.libraries.common.config library. Update according to guildelines. As well, I had to do some refactoring because of the change inside checkrhsmku and enablerhsmreposonrhel8 actors, including fix of tests. Co-authored-by: Vinzenz 'evilissimo' Feenstra <evilissimo@gmail.com>

…f rhsm skpped This is part of the scenario, when user wants to use only custom yum/dnf repositories (rhsm is skipped), but the system is subscribed in the same time. YUm by default uses the subscription-manager plugin; DNF uses it when configured. If the plugin is loaded, the redhat.repo file will be regenerated (e.g. it could lead into unexpected duplicated). As well, maybe it could happen that an error will be raised because of troubles to obtain expected data (e.g.). So we disable it).

pirat89 · 2020-04-07T11:37:24Z

bumped leapp-framework requirement to 1.2 (regarding the change in the leapp).

Rezney · 2020-04-07T14:02:42Z

@pirat89 please remove the "WIP"

pirat89 · 2020-04-08T10:49:24Z

@Rezney replaced WIP by DO_NO_MERGE. Waiting for QE to update their tests. @drehak is finishin unit tests. That's the last possible expected change in the PR unless you find something :)

pirat89 · 2020-04-08T10:57:37Z

Just keep others in picture, there is implemented another way to enable custom repositories during the IPU via the --enablerepo leapp option.

Rezney

Some additional typos found on the way. Will have to test the --enablerepo functionality too...

repos/system_upgrade/el7toel8/actors/checkrhsmsku/libraries/library.py

repos/system_upgrade/el7toel8/actors/checktargetrepos/libraries/library.py

Rezney · 2020-04-08T13:35:02Z

--enablerepo seems to be working fine. Ready to be approved.

AloisMahdal · 2020-04-08T21:30:48Z

If leapp_upgrade_repositories.repo is empty, we ge this inhibitor:

Risk Factor: high (inhibitor)
Title: Using RHSM has been skipped but no custom repositories have been delivered.
Summary: Leapp is run in the mode when the Red Hat Subscription Manager is not used (the --no-rhsm option or the LEAPP_NO_RHSM=1 environment variable has been set) so leapp is not able to obtain YUM/DNF repositories with the content for the target system in the standard way. The content has to be delivered by the user manually.
Remediation: [hint] Create the repository file according to instructions in the referred document on the following path with all repositories that should be used during the upgrade: "/etc/leapp/files/leapp_upgrade_repositories.repo".

The text mentions "the referred document" but does not refer any document.

How is cust going to know how to add the repos? (Merely cat'ing may not work if they already deployed the destination repos, because of duplicate id's.)

pirat89 · 2020-04-09T07:11:35Z

@AloisMahdal we expect that customer understand the repo file format and we will document it for them. So they are expected e.g. to open editor and write it inside.

It's seems that it's broken leapp. The link is really reffered in the code. Have you copied the whole report or just these lines?

 61             reporting.Remediation(hint=(                                        
 62                 'Create the repository file according to instructions in the'   
 63                 ' referred document on the following path with all'             
 64                 ' repositories that should be used during the upgrade: "{}".'   
 65                 '\n\n{}'                                                        
 66                 .format(CUSTOM_REPO_PATH, summary_ctrf)                         
 67             )),                                                                 
 68             reporting.Severity(reporting.Severity.HIGH),                        
 69             reporting.Tags([reporting.Tags.SANITY]),                            
 70             reporting.Flags([reporting.Flags.INHIBITOR]),                       
 71             reporting.ExternalLink(url=_IPU_DOC_URL, title='UPGRADING TO RHEL 8'),

The content (target repositories) for the target system is supposed to be delivered using RHSM by default. However it is possible to skip it using the --no-rhsm option (or LEAPP_NO_RHSM=1). User should be notified that if problem is encountered around rhsm (e.g. system is not registered), there is possibility to skip RHSM and provide the RHEL 8 content via custom repositories. Modify all related texts and reports.

…target repositories Previously when any custom target repository was not available inside the container (read, was not defined in a repo file), the fact was ignored. Currently, as user is able to proceed the upgrade without RHSM, completely, we should focus more on these values as well, otherwise it could happen we will not be able to create the target userspace (which in such case could end with ugly pretty-long output of errors immadiately after the fail of the bootstrap installation). As well, if rhsm is skipped, check whether there are not duplicate repository definitions inside the container. If so, inhibit the upgrade (as it happens in case when RHSM is used). Regarding the current state, the new issue has been created refactor and modify significantly the actor and related labrares. Issue: oamg#486

The actor produce CustomTargetRepository messages for repoids obtained from the cmdline when user use the --enablerepo option. The values are stored inside LEAPP_ENABLE_REPOS envar now, which is stored inside the configuration of the actor.

Update texts to inform even about the second way how to enable custom target repositories using the --enablerepo option for leapp, and do not produce info report about those new way, when already used this or custom repofile one.

pirat89 · 2020-04-09T07:12:39Z

@Rezney typos fixed :) thanks for reading.

AloisMahdal · 2020-04-09T07:54:44Z

I've only copied the inhibitor section and I'm pretty sure there was no link in that section. I'll double-check today.

AloisMahdal · 2020-04-09T13:26:35Z

leapp-ci build

AloisMahdal · 2020-04-09T13:29:07Z

e2e tests

pirat89 added enhancement New feature or request WIP Work In Progress labels Apr 2, 2020

pirat89 added the Requires Leapp PR The patch has to be merged/tested with Leapp PR label Apr 2, 2020

drehak mentioned this pull request Apr 2, 2020

Add --no-rhsm and LEAPP_NO_RHSM for skipping RHSM oamg/leapp#622

Merged

pirat89 force-pushed the custom_repos branch from 2112a0a to 72ceb41 Compare April 2, 2020 10:25

vinzenz reviewed Apr 2, 2020

View reviewed changes

repos/system_upgrade/el7toel8/actors/targetuserspacecreator/libraries/userspacegen.py Outdated Show resolved Hide resolved

pirat89 force-pushed the custom_repos branch 5 times, most recently from 2bcc869 to d32a695 Compare April 6, 2020 16:44

pirat89 mentioned this pull request Apr 6, 2020

"refactor" the targetuserspacecreator actor (and other related) #486

Open

pirat89 force-pushed the custom_repos branch from d32a695 to e0cfc40 Compare April 6, 2020 21:56

pirat89 removed the WIP Work In Progress label Apr 7, 2020

Rezney reviewed Apr 7, 2020

View reviewed changes

repos/system_upgrade/el7toel8/actors/checktargetrepos/libraries/library.py Outdated Show resolved Hide resolved

Rezney reviewed Apr 7, 2020

View reviewed changes

pirat89 force-pushed the custom_repos branch 2 times, most recently from aaef0e3 to 0e9801d Compare April 7, 2020 10:35

drehak and others added 5 commits April 7, 2020 13:34

pirat89 force-pushed the custom_repos branch from 0e9801d to 7677388 Compare April 7, 2020 11:36

Rezney previously approved these changes Apr 7, 2020

View reviewed changes

pirat89 dismissed Rezney’s stale review via a0c46e9 April 8, 2020 08:18

pirat89 force-pushed the custom_repos branch 2 times, most recently from f151ddc to 341b9cb Compare April 8, 2020 10:39

pirat89 changed the title ~~[WIP] Custom repos~~ [DO_NOT_MERGE] Custom repos Apr 8, 2020

Rezney requested changes Apr 8, 2020

View reviewed changes

pirat89 and others added 5 commits April 9, 2020 09:12

[RFE enablerepo 2/2] update texts and checktargetrepos

a3ba615

Update texts to inform even about the second way how to enable custom target repositories using the --enablerepo option for leapp, and do not produce info report about those new way, when already used this or custom repofile one.

Add tests for Checktargetrepos and reword report messages

66b077e

pirat89 force-pushed the custom_repos branch from 9061a04 to 66b077e Compare April 9, 2020 07:12

Rezney approved these changes Apr 9, 2020

View reviewed changes

drehak mentioned this pull request Apr 15, 2020

Implement generic CurrentActorMocked and logger_mocked #489

Merged

pirat89 changed the title ~~[DO_NOT_MERGE] Custom repos~~ Custom repos Apr 16, 2020

pirat89 merged commit fc5ae67 into oamg:master Apr 16, 2020

pirat89 deleted the custom_repos branch June 15, 2020 16:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom repos #482

Custom repos #482

pirat89 commented Apr 2, 2020 •

edited

Loading

leapp-bot commented Apr 2, 2020

pirat89 commented Apr 2, 2020

pirat89 commented Apr 2, 2020

pirat89 commented Apr 6, 2020

pirat89 commented Apr 6, 2020

pirat89 commented Apr 7, 2020

vinzenz commented Apr 7, 2020

Rezney left a comment

pirat89 commented Apr 7, 2020

Rezney commented Apr 7, 2020

pirat89 commented Apr 8, 2020

pirat89 commented Apr 8, 2020

Rezney left a comment

Rezney commented Apr 8, 2020

AloisMahdal commented Apr 8, 2020

pirat89 commented Apr 9, 2020

pirat89 commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

Custom repos #482

Custom repos #482

Conversation

pirat89 commented Apr 2, 2020 • edited Loading

leapp-bot commented Apr 2, 2020

Thank you for contributing to the Leapp project!

pirat89 commented Apr 2, 2020

pirat89 commented Apr 2, 2020

pirat89 commented Apr 6, 2020

pirat89 commented Apr 6, 2020

pirat89 commented Apr 7, 2020

vinzenz commented Apr 7, 2020

Rezney left a comment

Choose a reason for hiding this comment

pirat89 commented Apr 7, 2020

Rezney commented Apr 7, 2020

pirat89 commented Apr 8, 2020

pirat89 commented Apr 8, 2020

Rezney left a comment

Choose a reason for hiding this comment

Rezney commented Apr 8, 2020

AloisMahdal commented Apr 8, 2020

pirat89 commented Apr 9, 2020

pirat89 commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

AloisMahdal commented Apr 9, 2020

pirat89 commented Apr 2, 2020 •

edited

Loading