Snippet for setting up APT repos based on if the distro is EOL or not

[mikaelarguedas]

This is WIP, based on #66 and a partial implementation to address the migration of EOL distros to the snapshots repository (osrf/docker_images#281).

WIll update this description as work progresses

[mikaelarguedas]

I believe this is ready for a first round of feedback.

What's this does:

Update the templates to handle the snapshots repository and automatically switch images from packages.ros.org to snapshots.ros.org (with key updates as well) when new distros are added to the EOL lists defined in eol_distro.py

This allows to update the following images automatically:

• ROS legacy images
• ROS(1 or 2) active or EOL images
• ROS2 ros1_bridge images (that use both ROS1 and ROS2 distros)

What it doesn't do:

• use the snapshots repository for active ROS distributions by leveraging the SYNC_DATE repos
  • this would ensure rebuilds of the images on ROS syncs but requires extra infrastructure that is out of the initial scope of this PR
• use os_name to switch from the ubuntu to the debian subdirectory (https://discourse.ros.org/t/announcing-ros-snapshot-repositories/7705/10) instead it hardcodes ubuntu in the URL like it's done for packages.ros.org
• provide multiple tags for legaxy distros that have variants/metapackages
• merge legacy template with the regular ROS images templates

It would superseed/fix:

• Update Dockerfiles for ROS repository changes. docker_images#273
• [WIP] Rotate keys for ros1 apt repos docker_images#269
• Should EOL tags be swapped to use snapshots before decommissioning? docker_images#288

What it does poorly:

• packages.py hardcodes the URL templates for the apt repos, this PR hence checks twice if an image is relying on an EOL distro:
  • once in the python scripts to use the right URL to look up packages
  • one in the docker snippet to set the URL and keys accordingly

Resulting dockerfiles

Resulting Dockerfiles can be found at: https://github.com/osrf/docker_images/compare/use_docker_templates_67

[ruffsl]

Do we need to make sure the old ubuntu distros like zesty include the apt change for using archive?

RUN find /etc/apt/ -name *.list -exec sed -i -e 's/archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' {} \;

I was thinking it was missing from this file:

https://github.com/osrf/docker_images/compare/use_docker_templates_67#diff-a0fe8af50abffe34aed0f4124f6bdab0

[ruffsl]

• use the snapshots repository for active ROS distributions by leveraging the SYNC_DATE repos
  • this would ensure rebuilds of the images on ROS syncs but requires extra infrastructure that is out of the initial scope of this PR

SYNC_DATE repos? Could you elaborate what you envision here?

---

use os_name to switch from the ubuntu to the debian subdirectory instead it hardcodes ubuntu in the URL like it's done for packages.ros.org

I suppose that make everything a little more straight forward. Sounds ok.

---

provide multiple tags for legaxy distros that have variants/metapackages

Currently, there's just ros (from what I see from your liked osrf/docker_image diff), but the plan is to add more that have them?

---

packages.py hardcodes the URL templates for the apt repos, this PR hence checks twice if an image is relying on an EOL distro

As long as both stages are operating upon the same input/context (platform.yaml), that doesn't sound too bad.

[mikaelarguedas]

Do we need to make sure the old ubuntu distros like zesty include the apt change for using archive?

You're absolutely right I was planning on adding it everywhere and totally forgot when implementing it
Added in 52fa00a

SYNC_DATE repos? Could you elaborate what you envision here?

Sorry I should have updated osrf/docker_images#281 before submitting this.
What I was thinking is that from the announcement, it appears that each active ROS distro will get a snapshot uploaded after each sync, e.g. latest melodic sync available at http://snapshots.ros.org/melodic/2019-07-03/
It would be valuable to have all images (not only the EOL) tracking the snapshots repository, as it would provide both better reproducibility of the builds and fix the running issue of the OSRF docker images going out of date.
In order for this to work we would need a hook or a way to trigger an update of the dockerfiles each time a sync is pushed. At this point we would update the repository URL to the new timestamp and look up the package versions to install from that repo.
I left it out of this PR to allow us to get the repository keys and URL updated in a timely manner.

Currently, there's just ros (from what I see from your liked osrf/docker_image diff), but the plan is to add more that have them?

Yeah it occurred to me that groovy and later had variants like more recent distros (ros-core ros-base, desktop and friends). But I don't think there is much usage of these images and that anyone would benefit from it.
If all distros had it we could use the same templates/config fles for legacy and "current" images; but it's not the case so there is no real value in implementing it.

[ruffsl]

It would be valuable to have all images (not only the EOL) tracking the snapshots repository

Sounds useful.
But wouldn't this affect downstream users who apt-update +6 months after when the snapshot url is removed, or is this a degree of expected obsolescence we're willing to inject into official docker images?

Some folks may pin/archive a docker image for some time (all prior to a distro getting a final snapshot). Some of our images can have a long tail/chain of child images propagating through downstream projects (think caffe or openpose). Pinning the version of a package installed is one thing, but constantly revolving the apt repository URL seem more invasive.

Could we just pin the snapshot package version without changing the URL until /final is released?
Is there a /latest folder that we could target as the holdover to /final, or would that nullify the point of separating syncs into time-stamped repo URLs, or even not just using the packages.ros repo instead?

@nuclearsandwich am I correctly understanding the intended repo structure/lifecycle for snapshots?

[mikaelarguedas]

Could we just pin the snapshot package version without changing the URL until /final is released?
Is there a /latest folder that we could target as the holdover to /final, or would that nullify the point of separating syncs into time-stamped repo URLs, or even not just using the packages.ros repo instead?

I guess this would leave us with the same as the current state as images will go out of date and pinning exact versions would require different package version per architecture as they embed the build timestamp.

---

From https://answers.ros.org/question/328801/will-the-snapshot-repo-provide-debian-stretch-packages-for-melodic/ it looks like when a sync happens only the targeted Ubuntu LTS packages get synced to the snapshots repos so this rules out using the "SYNC_DATE" snapshots to ensure up-to-date images (at least for the non Ubuntu LTS base images)

[nuclearsandwich]

Could we just pin the snapshot package version without changing the URL until /final is released?
Is there a /latest folder that we could target as the holdover to /final, or would that nullify the point of separating syncs into time-stamped repo URLs, or even not just using the packages.ros repo instead?

Tracking a latest repo wouldn't provide any benefit over using the primary ROS repositories. The friction point that the snapshot repos can assuage is making sure that a docker image created prior to a sync can install new packages down the line without having to run a full apt-update to get the image onto the latest sync. Which comes with the trade-off that docker users would need to explicitly pull a new base image in order to get the current latest sync.

wouldn't [using the dated snapshot repos] affect downstream users who apt-update +6 months after when the snapshot url is removed, or is this a degree of expected obsolescence we're willing to inject into official docker images?
...
@nuclearsandwich am I correctly understanding the intended repo structure/lifecycle for snapshots?

Almost. In an ideal world with infinite storage resources, I would keep every snapshot forever. When we launched the snapshots project we didn't have a good estimate for the bandwidth and storage costs that would be associated with it so we could not, and have not, committed to keeping snapshots indefinitely. But did commit to providing them for at least six months.

The first snapshots would have "expired" in May of this year but they're still up. We don't want to have to take them down, we just don't have resources

Providing an atomically updateable repository for the Docker images to use was one of the initial use-cases I had in mind for the snapshots repositories. If I remember correctly the attempts to use the full package versions from the buildfarm to version releases stalled because the build timestamps in the full version numbers are platform dependent. The SYNC_DATE provides a single value capable of busting a docker build cache that atomically switches from one consistent state of all packages to another.

It would further complicate our template logic with combinatoric explosions but is it worth using the snapshot repositories for the LTS releases on Ubuntu LTS even if we can't use them elsewhere? We've identified a few of the shortcomings with the current approach (that snapshots are not available for all platforms we want to provide docker images for) but before we can justify the extra effort in resolving that shortcoming I think we need to determine that the snapshots will work for this purpose.

[ruffsl]

The friction point that the snapshot repos can assuage is making sure that a docker image created prior to a sync can install new packages down the line without having to run a full apt-update to get the image onto the latest sync.

For clarification, all dockerfiles here expressly wipe the apt cache before finalizing any images layer. Thus downstream child images must invoke apt-update to apt-install further debian packages anyhow.

As for apt-upgrades, that of course depends upon to what extent the difference syncs bump ros.org package versions, no? Users may of course use --no-upgrade flag in apt-install to avoid silently upgrading existing packages when installing new ones.

But I think understand your point as users could still install accupaning packages from the same sync for as long as the snapshot URL is archived, which I think is worthwhile if its in perpetuity.

---

In an ideal world with infinite storage resources, I would keep every snapshot forever.

Is there an example of how other official library repos have similarly handled this issue. Other debian packaging project must have to deal with shallow archive issues, thus the snapshot.debian.org , but I’ve not seen its use in other official library repos yet.

If I remember correctly the attempts to use the full package versions from the buildfarm to version releases stalled because the build timestamps in the full version numbers are platform dependent.

What purpose do we currently append the build timestamps to the package version string. Is there cause to release the same package with a newer build date, yet not yank the previous package with the same version number? Omitting the dates would help avoid the issue of all platforms not being built simultaneously.

The SYNC_DATE provides a single value capable of busting a docker build cache that atomically switches from one consistent state of all packages to another.

Then again, I suppose the larger issue was that not all platforms are synced simultaneously and would not have the same point version number available that we could use to break the build cache. I think if we wanted to provide SYNC_DATE tags for the official library images, we’d have to bite the bullet and refactor to allow for separate tags per platform architecture, e.g.:

Tags: dashing-ros-core, dashing-ros-core-bionic
Architectures: amd64, arm32v7, arm64v8
amd64-GitCommit: 1...
amd64-Directory: ros/dashing/ubuntu/bionic/amd64/ros-core
arm32v7-GitCommit: 2...
arm32v7-Directory: ros/dashing/ubuntu/bionic/arm32v7/ros-core
arm64v8-GitCommit: 3...
arm64v8-Directory: ros/dashing/ubuntu/bionic/arm64v8/ros-core

Tags: dashing-ros-core-bionic-20190613 #<--
Architectures: amd64
amd64-GitCommit: 1...
amd64-Directory: ros/dashing/ubuntu/bionic/amd64/ros-core

Tags: dashing-ros-core-bionic-20190702 #<--
Architectures: arm32v7, arm64v8
arm32v7-GitCommit: 2...
arm32v7-Directory: ros/dashing/ubuntu/bionic/arm32v7/ros-core
arm64v8-GitCommit: 3...
arm64v8-Directory: ros/dashing/ubuntu/bionic/arm64v8/ros-core

---

I suppose were in a sort of limbo; attempting to facilitate reproducibility for a meta level ecosystem on top of a moving target OS. So I’d like to ask what degree of reproducibility we are attempting to achieve here?

• To achieve reproducible library images from the official Dockerfile?
  • This of itself is difficult as the parent images we build FROM are only pinned down to the OS distro release, not the point version of that release nor even the SHA256 digest to a particular OS docker image repo.
  • Using the snapshots.ros, we could achieve ABI stability in the face of package upgrades violating semantic versioning, but I think addressing the semantic versioning violations would address the root issue there.
  • Given parent OS docker images are fresh, installing older ros snapshots may necessitate downgrade of system packages, potentially bloating the resulting ros images.
• To achieve reproducible child images from the official Docker Hub images?
  • This is equally difficult given the underpinned OS image explained previously, so freezing the system state can only be really accomplished by pinning the SHA256 of the ros image be used.
  • If this child image is for building source code, then I think breaking out all dependencies into sub-parent image and archiving/pining that image would be more effective.

---

I guess I see the official library images as being more of a rolling release of a configured environment of binaries, and if you want to freeze the image at a given time, then you should pin down that environment in a self contained archive with all its necessary packages, e.g. using its tag or SHA256 digest. If your dependencies change or expand, then it's a lot to ask that same setup to retroactively retrieve packages you did not archive and that could have an ephemeral existence.

[mikaelarguedas]

It would further complicate our template logic with combinatoric explosions but is it worth using the snapshot repositories for the LTS releases on Ubuntu LTS even if we can't use them elsewhere? We've identified a few of the shortcomings with the current approach (that snapshots are not available for all platforms we want to provide docker images for) but before we can justify the extra effort in resolving that shortcoming I think we need to determine that the snapshots will work for this purpose.

I don't think it would be worth doing as consistency across images and maintainability of the templates would be impacted. So IMO it would be worthwhile iif all distros getting debs generated by the ROS buildfarm have snapshots as well. Otherwise it may be better to use the snapshots repository for EOL images only and pursue another approach to keep the live images up to date.

Looking at all active ROS distros, all non-LTS base images are not targeted or already EOL so no problem there. The only remaining issue would be for Debian LTS (Stretch) that is targeted by Melodic but doesnt have snapshots (similarly best guess is the same question will come up for Noetic if it targets Debian Buster).

What purpose do we currently append the build timestamps to the package version string. Is there cause to release the same package with a newer build date, yet not yank the previous package with the same version number? Omitting the dates would help avoid the issue of all platforms not being built simultaneously.

My understanding is that the ROS buildfarm rebuilds downstream packages when a package is released (to work around ABI incompatibility). By adding the build timestamp it ensures that users doing an apt-upgrade will get the new packages and their newly built dependants.
On the Dockerfiles side it would allow to bust the cache if we were using the full version (otherwise: ros-base would have the same version number for the entire lifetime of the distro as these metapackages very rarely change).

Then again, I suppose the larger issue was that not all platforms are synced simultaneously and would not have the same point version number available that we could use to break the build cache. I think if we wanted to provide SYNC_DATE tags for the official library images, we’d have to bite the bullet and refactor to allow for separate tags per platform architecture, e.g.:

Hmm this is surprising are you sure this is accurate? While the individual debs are built at different times, Based on what I understood from the snapshots structure, the SYNC_DATE is consistent for all platforms / archs of a given ROS distro.

---

Originally I was hoping of solving the "docker images outdated" issue with this but you're right that pinning snapshot images has been a requested feature as well.

I do believe that the distributed images should always reflect the state of the last sync. I'm not sure we should go all the way to pinning down the OS base image as well but dont feel very strongly about it.

Regarding providing tagged versions of the image, we could indeed add a custom tag everytime we switch to a newer SYNC_DATE:

# latest dashing sync
Tags: dashing-ros-base, dashing-ros-base-bionic, dashing
Architectures: amd64, arm64v8
GitCommit: 2....
Directory: ros/dashing/ubuntu/bionic/ros-base

# dashing sync on 2019-06-14
Tags: dashing-ros-base-bionic-20190614
Architectures: amd64, arm64v8
GitCommit: 1....
Directory: ros/dashing/ubuntu/bionic/ros-base

# dashing sync on 2019-05-31
Tags: dashing-ros-base-bionic-20190531
Architectures: amd64, arm64v8
GitCommit: f....
Directory: ros/dashing/ubuntu/bionic/ros-base

It does imply tracking different commit hashes for each entry, but if my understanding of the SYNC_DATE is correct we shouldnt have to separate architectures or update this entries once they've been created.
But maybe I misunderstood the content of the files the manifest extract you provided is pointing to.
In your extract, would the images be pulling packages from packages.ros.org or snapshots.ros.org?

[nuclearsandwich]

What purpose do we currently append the build timestamps to the package version string...

My understanding is that the ROS buildfarm rebuilds downstream packages when a package is released (to work around ABI incompatibility). By adding the build timestamp it ensures that users doing an apt-upgrade will get the new packages and their newly built dependants.

👍

Then again, I suppose the larger issue was that not all platforms are synced simultaneously and would not have the same point version number available that we could use to break the build cache.

This isn't the case. The relevant portions of the sync process are:

1. A sync for a specific rosdistro (let's use melodic in this example) is announced for a specific target date.

2. When the build state is good, the ROS release manager triggers a rosdistro specific sync of packages to main (e.g. Mrel_sync-packages-to-main) which triggers an import of all ros-melodic-* packages across all platforms and architectures and their dependencies from testing to main then triggers a job which causes packages.ros.org to rsync the main repository from the build farm repository.

3. Once the rsync is completed, an aptly mirror is updated from packages.ros.org comprising all target architectures for the Ubuntu LTS supported by that rosdistro, 18.04 for Melodic. In the same script, a new datestamped aptly snapshot is made from that mirror and then published to a datestamped directory of snapshots.ros.org.

If ever we got to a point where we have full build farm support for on debian-based distros we would need to expand the above process to include those platforms but for the sake of maintainability we would almost certainly keep syncs as monolithic as possible.

[mikaelarguedas]

@nuclearsandwich @ruffsl It seems that we agree to move forward for using the snapshots repository for the EOL images. And don't tackle the images for active distros as its unclear the more suitable approach is for those.

If you get a chance, could you 👀 the current patch ? thanks!

[ruffsl]

Ok, from looking at the diff this can generate, this LGTM.
https://github.com/osrf/docker_images/compare/use_docker_templates_67

[ruffsl]

And don't tackle the images for active distros as its unclear the more suitable approach is for those.

After some thought, I've come around and thinking the benefits in using snapshot repo with SYNC_DATE outweigh most of the cons I had first been thinking of. I'd say if we can sync a nice trigger pipeline from to update the docker images upon a syncs, then this is mostly worth going for.

Perhaps we could consolidate the proposal for switching to snapshots into a briefe ticket on docker_images, and cc/ping the official docker hub librarians so they know what up.

[mikaelarguedas]

another effect of the template change I forgot to mention is the name of the source list files.

before it was ros-latest.list and ros2-latest.list, with this PR it will be:

ros1-latest.list
ros2-latest.list
ros1-snapshots.list
ros2-snapshots.list

[mikaelarguedas]

After some though, I've come around and thinking the benefits in using snapshot repo with SYNC_DATE outweigh most of the cons I had first been thinking of. I'd say if we can sync a nice trigger pipeline from to update the docker images upon a syncs, then this is mostly worth going for.

Yeah my main concern is the fact that we need to segregate Ubuntu LTS (to use snapshots.ros.org) and Debian LTS that doesnt get snapshots (using packages.ros.org) for a given ROS distro (e.g Melodic)

Perhaps we could consolidate the proposal for switching to snapshots into a briefe ticket on docker_images, and cc/ping the official docker hub librarians so they know what up.

👍
After some offline discussion with @ruffsl we'll get this merged and open follow-up tickets to discuss the use of snapshots repo for non-EOL images and separately the need to provide Debian LTS images

[mikaelarguedas]

Follow-up issues opened at osrf/docker_images#297 and osrf/docker_images#298