[Bug]: Cannot sync from an Harbor registry

[pedroosorio]

zot version

v2.1.0

Describe the bug

I'm configuring Zot to mirror from an Harbor registry - https://harbor.skao.int. The pull-through (on demand) is working fine, but i cannot get the scheduled scan to work.

To reproduce

My configuration is:

{
  "distSpecVersion": "1.1.0",
  "storage": {
    "rootDirectory": "/data/zot",
    "dedupe": true,
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "9090",
    "realm": "zot",
    "auth": {
      "htpasswd": {
        "path": "/etc/zot/htpasswd"
      },
      "failDelay": 5
    },
    "accessControl": {
      "repositories": {
        "**": {
          "anonymousPolicy": ["read"]
        }
      },
      "adminPolicy": {
        "users": ["admin"],
        "actions": ["read", "create", "update", "delete"]
      }
    }
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "search": {
      "enable": true
    },
    "ui": {
      "enable": true
    },
    "sync": {
      "credentialsFile": "/etc/zot/credentials.json",
      "registries": [
        {
          "urls": ["https://harbor.skao.int"],
          "pollInterval": "1m",
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 3,
          "retryDelay": "5m",
          "content": [
            {
              "prefix": "/test-promotion/ska-tango-tangogql"
            }
          ]
        }
      ]
    }
  }
}

Expected behavior

I would expect the sync to actually work. I suspect this might come from the fact that we have custom Nginx code in front of Harbor to allow it to "answer" to other domains on a specific project. Although, the registry works just fine with podman/docker CLI login and pull/push wise. It also works with ORAS cli, so if there was something really wrong under the hood, some of these would fail i guess.

Screenshots

Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"info","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-01-14T16:15:54.690285291Z","message":"getting available client"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"?account=<redacted>&scope=&service=","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:275","time":"2025-01-14T16:15:54.741970344Z","message":"failed to make request"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"https://harbor.skao.int/v2/_catalog","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:244","time":"2025-01-14T16:15:54.742057614Z","message":"failed to get token from authorization realm"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"https://harbor.skao.int/v2/_catalog","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:206","time":"2025-01-14T16:15:54.742122775Z","message":"failed to make request"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","errorType":"*url.Error","remote registry":"https://harbor.skao.int","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:228","time":"2025-01-14T16:15:54.742173066Z","message":"failed to get repository list from remote registry"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","component":"scheduler","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","generator":"SyncGenerator","goroutine":40,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:468","time":"2025-01-14T16:15:54.742215717Z","message":"failed to execute generator"}

Additional context

No response

[rchincha]

@pedroosorio can you also report logs from harbor side?
There should really be no difference unless there is some uri mismatch (somehow)

[eusebiu-constantin-petu-dbk]

Hello @pedroosorio

I need to know how the www-authenticate header looks if you make an unathenticated GET on the catalog endpoint.
Can you please show that? I think I can fix it if I know this information.

like curl -vvv https://harbor.skao.int/v2/_catalog

Thank you!

[andaaron]

Hello @pedroosorio

I need to know how the www-authenticate header looks if you make an unathenticated GET on the catalog endpoint. Can you please show that? I think I can fix it if I know this information.

like curl -vvv https://harbor.skao.int/v2/_catalog

Thank you!

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 401
< date: Thu, 16 Jan 2025 09:58:17 GMT
< content-type: application/json; charset=utf-8
< content-length: 108
< server: nginx
< docker-distribution-api-version: registry/2.0
< set-cookie: sid=04d85f456e26e9df59abf82e1450572c; Path=/; HttpOnly
< x-request-id: 0de31032-2c01-4b63-a64f-878d47f87108
< www-authenticate: Basic realm="harbor"
<
{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to list catalog: unauthorized to list catalog"}]}