Skip to content

Commit

Permalink
feat: git Access, AccessMethod through BlobAccess (#869)
Browse files Browse the repository at this point in the history
#### What this PR does / why we need it:

~This is an extremely early (WIP) draft of implementing ctf archives in
directory format tracked via git repositories, ultimately serving me as
a playground to get familiar with OCM and also to play around with its
feature set. So we do not need to merge this if there is no interest or
disagreement on the implementation.~

This PR has developed into the development of a pure access method and
input method that can be used to work with arbitrary git repositories.
Examples for a component constructor could be:

```
name: test.de/x
version: %s
provider:
  name: ocm
resources:
- name: hello-world-access
  type: git
  version: 0.0.1
  access:
    type: git
    commit: "7fd1a60b01f91b314f59955a4e4d4e80d8edf11d"
    ref: refs/heads/master
    repository: https://github.com/octocat/Hello-World.git
- name: hello-world-input
  type: git
  version: 0.0.1
  input:
    type: git
    ref: refs/heads/master
    commit: "7fd1a60b01f91b314f59955a4e4d4e80d8edf11d"
    repository: https://github.com/octocat/Hello-World.git
```


~Not only does this allow using git repositories for storing blobs, it
also attempts to add git downloader functionality for a generic git
access method based on in memory cloning and I plan to add a OCM
implementation on it as well.~ While the download is available, this is
not laid out to the CLI, library only.

~Currently the implementation is written so that any modification to the
CTF in the work directory also triggers a dedicated commit, and any
lookup triggers a refres/pull cycle on git.~

For testing I use `file://` URLs which makes it quite easy to verify E2E
in the test suites.

TODO:

- [X] Implement Access Credential mapping
- [x] Make Git Client thread-safe, optionally in-memory and allow more
customization (currently hard bount to one target ref that I am creating
or syncing against, see tests for examples)
- [X] Expand Git Options - Authentication support for repo and client

#### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
-->
~As this is my own "experimentation" issue, it doesnt necessarily fix
anything, it might as well be moved to a plugin for example or be
scrapped alltogether~

Creates an access method for arbitrary git repositories using git-go.

---------

Co-authored-by: Gergely Brautigam <182850+skarlso@users.noreply.github.com>
  • Loading branch information
jakobmoellerdev and Skarlso authored Jan 6, 2025
1 parent d8559ce commit 48c3ea2
Show file tree
Hide file tree
Showing 45 changed files with 2,458 additions and 9 deletions.
1 change: 1 addition & 0 deletions api/credentials/builtin/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ package builtin

import (
_ "ocm.software/ocm/api/credentials/builtin/github"
_ "ocm.software/ocm/api/tech/git/identity"
)
12 changes: 6 additions & 6 deletions api/oci/cpi/support/base.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,18 @@ func (a *artifactBase) IsReadOnly() bool {
}

func (a *artifactBase) IsIndex() bool {
d := a.state.GetState().(*artdesc.Artifact)
return d.IsIndex()
d, ok := a.state.GetState().(*artdesc.Artifact)
return ok && d.IsIndex()
}

func (a *artifactBase) IsManifest() bool {
d := a.state.GetState().(*artdesc.Artifact)
return d.IsManifest()
d, ok := a.state.GetState().(*artdesc.Artifact)
return ok && d.IsManifest()
}

func (a *artifactBase) IsValid() bool {
d := a.state.GetState().(*artdesc.Artifact)
return d.IsValid()
d, ok := a.state.GetState().(*artdesc.Artifact)
return ok && d.IsValid()
}

func (a *artifactBase) blob() (cpi.BlobAccess, error) {
Expand Down
2 changes: 1 addition & 1 deletion api/oci/internal/uniform.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ type UniformRepositorySpec struct {
// Host is the hostname of an oci ref.
Host string `json:"host,omitempty"`
// Info is the file path used to host ctf component versions
Info string `json:"filePath,omitempty"`
Info string `json:"info,omitempty"`

// CreateIfMissing indicates whether a file based or dynamic repo should be created if it does not exist
CreateIfMissing bool `json:"createIfMissing,omitempty"`
Expand Down
58 changes: 58 additions & 0 deletions api/ocm/elements/artifactaccess/gitaccess/options.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
package gitaccess

import (
"github.com/mandelsoft/goutils/optionutils"
)

type Option = optionutils.Option[*Options]

type Options struct {
URL string
Ref string
Commit string
}

var _ Option = (*Options)(nil)

func (o *Options) ApplyTo(opts *Options) {
if o.URL != "" {
opts.URL = o.URL
}
}

func (o *Options) Apply(opts ...Option) {
optionutils.ApplyOptions(o, opts...)
}

// //////////////////////////////////////////////////////////////////////////////
// Local options

type url string

func (h url) ApplyTo(opts *Options) {
opts.URL = string(h)
}

func WithURL(h string) Option {
return url(h)
}

type ref string

func (h ref) ApplyTo(opts *Options) {
opts.Ref = string(h)
}

func WithRef(h string) Option {
return ref(h)
}

type commitSpec string

func (h commitSpec) ApplyTo(opts *Options) {
opts.Commit = string(h)
}

func WithCommit(c string) Option {
return commitSpec(c)
}
25 changes: 25 additions & 0 deletions api/ocm/elements/artifactaccess/gitaccess/resource.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package gitaccess

import (
"github.com/mandelsoft/goutils/optionutils"

"ocm.software/ocm/api/ocm"
"ocm.software/ocm/api/ocm/compdesc"
"ocm.software/ocm/api/ocm/cpi"
"ocm.software/ocm/api/ocm/elements/artifactaccess/genericaccess"
access "ocm.software/ocm/api/ocm/extensions/accessmethods/git"
resourcetypes "ocm.software/ocm/api/ocm/extensions/artifacttypes"
)

const TYPE = resourcetypes.DIRECTORY_TREE

func Access[M any, P compdesc.ArtifactMetaPointer[M]](ctx ocm.Context, meta P, opts ...Option) cpi.ArtifactAccess[M] {
eff := optionutils.EvalOptions(opts...)
if meta.GetType() == "" {
meta.SetType(TYPE)
}

spec := access.New(eff.URL, access.WithRef(eff.Ref), access.WithCommit(eff.Commit))
// is global access, must work, otherwise there is an error in the lib.
return genericaccess.MustAccess(ctx, meta, spec)
}
93 changes: 93 additions & 0 deletions api/ocm/extensions/accessmethods/git/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@

# Access Method `git` - Git Commit Access

## Synopsis

```yaml
type: git/v1
```
Provided blobs use the following media type for: `application/x-tgz`

The artifact content is provided as gnu-zipped tar archive

### Description

This method implements the access of the content of a git commit stored in a
git repository.

Supported specification version is `v1alpha1`

### Specification Versions

#### Version `v1alpha1`

The type specific specification fields are:

- **`repository`** *string*

Repository URL with or without scheme.

- **`ref`** (optional) *string*

Original ref used to get the commit from

- **`commit`** *string*

The sha/id of the git commit

### Go Bindings

The go binding can be found [here](method.go)

#### Example

```go
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"fmt"
"io"
"ocm.software/ocm/api/ocm"
"ocm.software/ocm/api/ocm/cpi"
me "ocm.software/ocm/api/ocm/extensions/accessmethods/git"
)
func main() {
ctx := ocm.New()
accessSpec := me.New(
"https://github.com/octocat/Hello-World.git",
me.WithRef("refs/heads/master"),
)
method, err := accessSpec.AccessMethod(&cpi.DummyComponentVersionAccess{Context: ctx})
if err != nil {
panic(err)
}
content, err := method.GetContent()
if err != nil {
panic(err)
}
unzippedContent, err := gzip.NewReader(bytes.NewReader(content))
r := tar.NewReader(unzippedContent)
file, err := r.Next()
if err != nil {
panic(err)
}
if file.Name != "README.md" {
panic("Expected README.md")
}
data, err := io.ReadAll(r)
if err != nil {
panic(err)
}
fmt.Println(string(data))
}
```
43 changes: 43 additions & 0 deletions api/ocm/extensions/accessmethods/git/cli.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package git

import (
"ocm.software/ocm/api/ocm/extensions/accessmethods/options"
"ocm.software/ocm/api/utils/cobrautils/flagsets"
)

func ConfigHandler() flagsets.ConfigOptionTypeSetHandler {
return flagsets.NewConfigOptionTypeSetHandler(
Type, AddConfig,
options.RepositoryOption,
options.ReferenceOption,
options.CommitOption,
)
}

func AddConfig(opts flagsets.ConfigOptions, config flagsets.Config) error {
flagsets.AddFieldByOptionP(opts, options.RepositoryOption, config, "repository", "repo", "repoUrl", "repoURL")
flagsets.AddFieldByOptionP(opts, options.CommitOption, config, "commit")
flagsets.AddFieldByOptionP(opts, options.ReferenceOption, config, "ref")
return nil
}

var usage = `
This method implements the access of the content of a git commit stored in a
Git repository.
`

var formatV1 = `
The type specific specification fields are:
- **<code>repoUrl</code>** *string*
Repository URL with or without scheme.
- **<code>ref</code>** (optional) *string*
Original ref used to get the commit from
- **<code>commit</code>** *string*
The sha/id of the git commit
`
Loading

0 comments on commit 48c3ea2

Please sign in to comment.