redis driver for blob cache information and metadb

[andaaron]

Rebased #2412 and continued work on missing method implementation for cache driver, and implemented MetaDB methods.

Still need to:

• run linter
• make sure all existing tests pass
• add more tests to cover redis
  • cache driver
    • note: https://github.com/go-redis/redismock is very useful but for some reason doesn't get the TxPipelined coverage
  • metadb
• account for separate redis configuration for different substores
• see if we can do anything for image signatures (right now testing with local storage, but we should have an alternative).
• use an effective locking system:
  • need to check what support they offer in this library for concurrent goroutines
  • even if they have support for concurrent goroutines in the library we need a solution to work with multiple zot instances using the same database instance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[elee1766]

@andaaron

this is awesome work. thank you.

i wish i could finish this but company needs put me elsewhere. will be super excited to use this feature when it is ready.

I wanted to note a few things, since i see some notes in your PR.

1. for locks, i have had good success in https://github.com/go-redsync/redsync in the past, using it with redis-v9 driver. however, they only provide a somewhat-safe mutex, and you still need to implement either a leadership election on top of the mutex primative, or force the hot-paths to use mutex. for my needs, this has been okay, since mostly I am using mutexes to save on duplication of work.

2. this is a slight tangent, but the more modern redis library is https://github.com/redis/rueidis
   which has https://github.com/redis/rueidis/tree/main/rueidislock. i have been using it for some more recent work, as it claims function to better in a redis cluster, however, the locking package is much less tested and used vs redsync/v4. I wouldn't use it for this project personally, but the option does exist, so i thought i would say

3. redis cluster, maybe you know, has very interesting behavior + latency during failover, reshuffle, etc, or at least in my experience. i'm not sure if this is in scope for the project, but especially if a lock is going to be in the hot path, it may be important to test that zot does not explode when there are 10-15 second timeouts on random queries to redis

4. fwiw, i dont think that redis is a great choice for a cache driver. the max value size is 512mb, which is much smaller than the max size for an oci blob (i think its a few gb?), there are many layers with size above 512mb. You could potentially shard/multipart the blob into multiple keys, by creating some container format around blob parts to indicate where the next "part" of the blob is, (now you get into some weird stuff with eviction, so you probably want this to be a lua script), but i dont know, it seems a lot more effort than its worth.

[rchincha]

@andaaron thx for taking this up

[andaaron]

4. fwiw, i dont think that redis is a great choice for a cache driver. the max value size is 512mb, which is much smaller than the max size for an oci blob (i think its a few gb?), there are many layers with size above 512mb. You could potentially shard/multipart the blob into multiple keys, by creating some container format around blob parts to indicate where the next "part" of the blob is, (now you get into some weird stuff with eviction, so you probably want this to be a lua script), but i dont know, it seems a lot more effort than its worth.

With regards to this specific point, we have a misunderstanding. The role of the cache driver is to store information on what blob was deduped and and where are the duplicates. The role of metadb is to store information taken from manifests, as well as other metadata about manifests, repos, and users.

We wouldn't be using it to store and serve the actual image blobs, that is a separate storage implementation, where we support local folders and S3.

[codecov]

Codecov Report

Attention: Patch coverage is 68.66430% with 617 lines in your changes missing coverage. Please review.

Project coverage is 90.49%. Comparing base (fba695a) to head (c04f85d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/meta/redisdb/redis.go	64.25%	457 Missing and 135 partials ⚠️
pkg/storage/cache/redis.go	90.71%	18 Missing and 4 partials ⚠️
pkg/meta/meta.go	92.30%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2865      +/-   ##
==========================================
- Coverage   91.90%   90.49%   -1.41%     
==========================================
  Files         170      172       +2     
  Lines       30291    32246    +1955     
==========================================
+ Hits        27838    29181    +1343     
- Misses       1825     2300     +475     
- Partials      628      765     +137     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[JakeCooper]

Just commenting here; we'd love to move to Zot at Railway.com for our Cloud+Baremetal clusters. This, we believe, would be the last thing blocking us

Would love to help get it across the line in any way we can