Add support for OCI engine #914

ezekg · 2024-11-08T17:06:59Z

Closes #911. In the end, it should be as simple as docker save keygen/api:1.0.0 > keygen-1.0.0.tar and keygen upload keygen-1.0.0.tar — the rest should be automatically handled by the OCI engine.

Like with npm and friends, docker push will not be supported right away.

Prereqs

Implement the Docker/OCI manifest and blob API endpoints.
Test parsing of popular images, e.g. Alpine, Ubuntu, etc.
Test multi-platform images using docker buildx.
Test OCI conformance (namely OCI_TEST_PULL=1).¹
Write tests for OCI spec conformance.
Write tests for manifests API.
Write tests for blobs API.
Helm compatibility (should be fine?)
Make EE-only.

Subreqs

Results from OCI conformance test can be found here. ↩

spec/workers/process_docker_image_worker_spec.rb

app/workers/process_docker_image_worker.rb

spec/support/matchers/upload_matcher.rb

ezekg · 2024-11-10T00:34:11Z

app/workers/process_docker_image_worker.rb

+require 'minitar'
+require 'zlib'
+
+class ProcessDockerImageWorker < BaseWorker


This worker should be in a high_memory queue.

app/workers/process_docker_image_worker.rb

reverts commit 04dabba.

app/controllers/api/v1/release_engines/oci/blobs_controller.rb

app/controllers/api/v1/release_engines/oci/manifests_controller.rb

app/controllers/application_controller.rb

app/controllers/concerns/rendering.rb

- oci expects any manifest to be referenceable not just the index

app/controllers/api/v1/release_engines/oci/blobs_controller.rb

app/controllers/api/v1/release_engines/oci/manifests_controller.rb

app/models/release_artifact.rb

app/models/release_manifest.rb

db/migrate/20241113134529_create_release_descriptors.rb

db/migrate/20241114163421_add_content_attributes_to_release_manifests.rb

features/api/v1/engines/oci/conformance.feature

ezekg · 2024-11-16T00:00:53Z

lib/enumerator_io.rb

+      @buffer.clear
+    end
+
+    buffer.empty? ? nil : buffer


buffer.presence should suffice?

lib/oci_image_layout.rb

ezekg · 2024-11-17T05:26:19Z

lib/oci_image_layout.rb

+      # a Minitar::Reader, and that closes an entry's reader io after
+      # reading the entry. This lets us avoid buffering entries into
+      # memory, since some of the blobs could be large layers.
+      io.each do |entry|


Move all this outside of initializer?

ezekg · 2024-11-17T05:31:00Z

lib/oci_image_layout.rb

+      io.rewind
+    end
+
+    def each(&)


Should we add return to_enum(:each) unless block_given??

db/migrate/20241114163421_add_content_attributes_to_release_manifests.rb

db/migrate/20241113134529_create_release_descriptors.rb

ezekg · 2024-11-21T23:23:39Z

db/migrate/20241114163421_add_content_attributes_to_release_manifests.rb

+  disable_ddl_transaction!
+
+  def change
+    add_column :release_manifests, :content_path,   :string, null: true


Eventually make these not-null.

- blocker: keygen-sh/keygen-api#914

ezekg · 2024-11-28T01:48:52Z

app/workers/process_oci_image_worker.rb

+
+class ProcessOciImageWorker < BaseWorker
+  MIN_TARBALL_SIZE  = 512.bytes     # to avoid processing empty or invalid tarballs
+  MAX_TARBALL_SIZE  = 512.megabytes # to avoid downloading excessive tarballs


This is unlikely to be sufficient for some applications, but I feel like it's a good default. Maybe bumping it up to 1GB would be fine, since I've seen quite a few 800MB images. I think we should eventually add a max artifact size to accounts.

E.g. Sentry is 250MB, Plausible is 50MB, Keygen is 90MB, Cal.com is 1.4GB, Caddy on Windows is 2.2GB and 80MB on Linux, and PostHog is 1.5GB.

Maybe add max_upload for individual uploads, and max_ingress and max_egress for monthly ingress/egress limits (which can be handled like max_reqs).

- eventually add max_upload to plans

ezekg force-pushed the feature/docker-engine branch 3 times, most recently from 49c2e66 to ba9255f Compare November 8, 2024 23:08

ezekg commented Nov 9, 2024

View reviewed changes

spec/workers/process_docker_image_worker_spec.rb Outdated Show resolved Hide resolved

ezekg commented Nov 10, 2024

View reviewed changes

app/workers/process_docker_image_worker.rb Outdated Show resolved Hide resolved

app/workers/process_docker_image_worker.rb Outdated Show resolved Hide resolved

spec/support/matchers/upload_matcher.rb Outdated Show resolved Hide resolved

ezekg commented Nov 10, 2024

View reviewed changes

app/workers/process_docker_image_worker.rb Outdated Show resolved Hide resolved

ezekg commented Nov 10, 2024

View reviewed changes

app/workers/process_docker_image_worker.rb Outdated Show resolved Hide resolved

ezekg force-pushed the feature/docker-engine branch 4 times, most recently from 576478c to 39740c4 Compare November 11, 2024 22:21

ezekg added 11 commits November 13, 2024 07:31

revert "temp: remove docker image processor"

ebc1c00

reverts commit 04dabba.

add tests for docker image worker

3b80495

add enumeratorio to improve memory efficiency

573e1db

refactor npm worker to use enumeratorio

9de0639

update naming

1d8014d

fix typo

afcbf80

update image processor to use index.json

49cabe8

adjust tarball min/max sizes

7548ded

s/docker/oci/

91207ec

add regex support to upload matcher

82c343b

add base for oci routes

19c6359

ezekg force-pushed the feature/docker-engine branch 2 times, most recently from 7758cb0 to a4c8292 Compare November 15, 2024 00:10

ezekg commented Nov 15, 2024

View reviewed changes

ezekg force-pushed the feature/docker-engine branch from a4c8292 to 306a26c Compare November 15, 2024 01:13

ezekg added 3 commits November 14, 2024 20:19

refactor oci image processing to store media types

8775471

remove unsupported chunked writes

c1051ab

add support for multiple manifests per artifact

da5fa6b

- oci expects any manifest to be referenceable not just the index

ezekg added 4 commits November 15, 2024 16:03

remove json minification

1b4a67c

fix format

ad8808f

fix typo

ce3ad8c

remove superfluous requires

f106a2c

ezekg force-pushed the feature/docker-engine branch from ff1fc3e to f106a2c Compare November 15, 2024 22:06

fix max size mismatch between factory and worker

83a2fb3

ezekg commented Nov 16, 2024

View reviewed changes

lib/oci_image_layout.rb Show resolved Hide resolved

ezekg commented Nov 18, 2024

View reviewed changes

ezekg added 2 commits November 21, 2024 15:06

rename filename to path

c3bc0be

fix indexes

bbac712

ezekg commented Nov 21, 2024

View reviewed changes

ezekg added a commit to keygen-sh/keygen-relay that referenced this pull request Nov 26, 2024

temp: comment image recipes

9425ebd

- blocker: keygen-sh/keygen-api#914

add tests for blobs

aaa4603

ezekg commented Nov 28, 2024

View reviewed changes

fix content type indexes

8a3a382

ezekg force-pushed the feature/docker-engine branch from 04e8ccb to 8a3a382 Compare December 5, 2024 21:49

ezekg mentioned this pull request Dec 21, 2024

Publish Docker image keygen-sh/keygen-relay#26

Open

ezekg added 7 commits December 21, 2024 00:59

add tests for vanity urls

214e51f

temp: update max tarball size

835da44

- eventually add max_upload to plans

s/delete/yank/

ca6cc61

add schema version to index

2c62297

refactor digestio into lib

3d89247

update oci engine to be ee-only

5f1e96a

refactor content-type/accept logic

416d743

keygen-sh locked and limited conversation to collaborators Jan 12, 2025

keygen-sh unlocked this conversation Jan 12, 2025

refactor jsonapi/json content-type logic

54f1aa2

ezekg changed the title ~~Add support for Docker engine~~ Add support for OCI engine Jan 13, 2025

ezekg marked this pull request as ready for review January 13, 2025 21:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for OCI engine #914

Add support for OCI engine #914

ezekg commented Nov 8, 2024 •

edited

Loading

ezekg Nov 10, 2024

ezekg Nov 16, 2024

ezekg Nov 17, 2024

ezekg Nov 17, 2024

ezekg Nov 21, 2024

ezekg Nov 28, 2024

ezekg Nov 28, 2024 •

edited

Loading

ezekg Nov 28, 2024

Add support for OCI engine #914

Are you sure you want to change the base?

Add support for OCI engine #914

Conversation

ezekg commented Nov 8, 2024 • edited Loading

Prereqs

Subreqs

Footnotes

ezekg Nov 10, 2024

Choose a reason for hiding this comment

ezekg Nov 16, 2024

Choose a reason for hiding this comment

ezekg Nov 17, 2024

Choose a reason for hiding this comment

ezekg Nov 17, 2024

Choose a reason for hiding this comment

ezekg Nov 21, 2024

Choose a reason for hiding this comment

ezekg Nov 28, 2024

Choose a reason for hiding this comment

ezekg Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

ezekg Nov 28, 2024

Choose a reason for hiding this comment

ezekg commented Nov 8, 2024 •

edited

Loading

ezekg Nov 28, 2024 •

edited

Loading