More work to documentation and README and some minor cleanups that we…

…re encountered

README.md

@@ -11,19 +11,27 @@ Bubbly is a release readiness platform helping software teams release compliant
 
 **Release Readiness** is a term that we use to define the state of being ready to release software. This project aims at helping teams help teams define their Release Readiness policies (using the [Rego Policy Language](https://www.openpolicyagent.org/docs/latest/policy-language/)), collect the data to enforce those policies, and measure their performance getting to their Release Readiness goals (whether it be speed or stability).
 
-![bubbly-in-a-bubble](/docs/static/img/bubbly-in-a-bubble.svg)
+## How it works
 
-## Project Status
+Bubbly's main entity is the `Release` entity. A Release has a one-to-one relationship with a version of your software (i.e. commit) and the purpose of the release is to collect all the data we want to inform our Release Readiness decision.
 
-This project is currently in **Alpha**
+Getting data in to Bubbly was one of the core challenges faced, and the `Adapters` are the solution here.
+After much work with different config languages, we landed on the [Rego Policy Language](https://www.openpolicyagent.org/docs/latest/policy-language/) for the adapter implementation and it works really well.
+Check the [documentation on adapters](https://docs.bubbly.dev/adapters/adapters) for more.
 
-In terms of stability, the existing features should work and if they do not please raise an issue. Also we do not guarantee backwards compatability with the APIs and schema.
+For each type of result that is "adapted" into Bubbly and added to a release, a `release entry` is created which forms the release log.
+Also, every time a new piece of data is added to a release, the policies that apply to that release are re-evaluated and any violations created.
+Policies use the [Rego Policy Language](https://www.openpolicyagent.org/docs/latest/policy-language/) which is great for defining rules such as `require` (stating that some event needs to happen) and `deny` (defining something that cannot happen in a release). Also violations have a severity, so you can create suggestions, warnings and blockers.
 
-In terms of feature completeness, we are working on implementing the core features of Bubbly, and hence there are missing features, such as Subscriptions and Monitoring of vulnerabilities.
+Subscriptions can be created to inform about new events in Bubbly (such as a failed release, or a new vulnerability).
+Also a GraphQL interface exists to help extract data from Bubbly.
+One of our design goals is to have the right data in the right place at the right time, and therefore getting data out of Bubbly to integrate into your existing process (dashboards, PRs) is something we want to make possible.
 
-If you are interested in getting involved in the project, or want some help getting started, please reach out to us though our website: [https://bubbly.dev](https://bubbly.dev)
+![bubbly-in-a-bubble](docs/static/img/bubbly-in-a-bubble.svg)
 
-## Problem Statement
+## Motivation
+
+### Problem Statement
 
 The problem around Release Readiness is one of data. All the tools used in the software process produce data, and that data should be used to drive the Release Readiness decision.
 
@@ -34,7 +42,7 @@ The real problem is that this data gets scattered all over the place which resul
 3. Overhead maintaining multiple data stores and data bases, ensuring that the data is up to date and accurate
 4. Lack of understanding of the data's hierarchy
 
-## Proposed Solution
+### Proposed Solution
 
 Bubbly has been built to address the core problems mentioned above. This has been achieved by implementing a simple engine built over the [Rego Policy Language](https://www.openpolicyagent.org/docs/latest/policy-language/) with a versatile database schema, built using [entgo](https://entgo.io/).
 
@@ -66,7 +74,17 @@ The things that Bubbly is not aiming to be:
 * A testing tool/framework.
 * A traditional monitoring tool (think Ops metrics, like Prometheus).
 * Another dashboard (like Grafana).
-
+
+## Project Status
+
+This project is currently in **Alpha**
+
+In terms of stability, the existing features should work and if they do not please raise an issue. Also we do not guarantee backwards compatability with the APIs and schema.
+
+In terms of feature completeness, we are working on implementing the core features of Bubbly, and hence there are missing features, such as Subscriptions and Monitoring of vulnerabilities.
+
+If you are interested in getting involved in the project, or want some help getting started, please reach out to us though our website: [https://bubbly.dev](https://bubbly.dev)
+
 ## Architecture
 
 See the [ARCHITECTURE.md](./ARCHITECTURE.md) file for the structure of this repository.

client/client.go

@@ -80,6 +80,12 @@ func handleGetRequest(bCtx *env.BubblyContext, resp interface{}, urlsuffix strin
 		q.Add(name, value)
 	}
 	httpReq.URL.RawQuery = q.Encode()
+
+	bCtx.Logger.Debug().
+		Str("query", q.Encode()).
+		Str("method", http.MethodGet).
+		Msg("Client making request")
+
 	httpResp, err := http.DefaultClient.Do(httpReq)
 	if err != nil {
 		return err
@@ -102,7 +108,12 @@ func handlePostRequest(bCtx *env.BubblyContext, req interface{}, urlsuffix strin
 		return err
 	}
 
-	fmt.Printf("Post Request:\n%s\n", b)
+	bCtx.Logger.Debug().
+		Bytes("data", b.Bytes()).
+		Str("endpoint", urlsuffix).
+		Str("method", http.MethodPost).
+		Msg("Client making request")
+
 	url := bCtx.ClientConfig.V1() + "/" + urlsuffix
 	httpReq, err := http.NewRequest(http.MethodPost, url, b)
 	if err != nil {
@@ -125,7 +136,12 @@ func handlePutRequest(bCtx *env.BubblyContext, req interface{}, urlsuffix string
 		return err
 	}
 
-	fmt.Printf("Put Request:\n%s\n", b)
+	bCtx.Logger.Debug().
+		Bytes("data", b.Bytes()).
+		Str("endpoint", urlsuffix).
+		Str("method", http.MethodPut).
+		Msg("Client making request")
+
 	url := bCtx.ClientConfig.V1() + "/" + urlsuffix
 	httpReq, err := http.NewRequest(http.MethodPut, url, b)
 	if err != nil {

cmd/adapter/run/run.go

@@ -105,7 +105,7 @@ func New(bCtx *env.BubblyContext) *cobra.Command {
 				return fmt.Errorf("unknown output format: %s", output)
 
 			}
-			fmt.Printf("\n\nAdapter %s ran successfully!\n", adapterID)
+			fmt.Printf("\nAdapter %s ran successfully!\n", adapterID)
 			return nil
 		},
 	}

cmd/policy/set/set.go

@@ -29,7 +29,6 @@ var (
 func New(bCtx *env.BubblyContext) *cobra.Command {
 
 	var (
-		name           string
 		setProjects    []string
 		notSetProjects []string
 		setRepos       []string
@@ -59,12 +58,6 @@ func New(bCtx *env.BubblyContext) *cobra.Command {
 	}
 
 	f := cmd.PersistentFlags()
-	f.StringVar(
-		&name,
-		"name",
-		"",
-		`Provide the name of the policy (default is filename without extension)`,
-	)
 	f.StringSliceVar(&setProjects, "projects", nil, "List of project (names) to associate the policy with")
 	f.StringSliceVar(&notSetProjects, "not-projects", nil, "List of project (names) to disassociate the policy with")
 	f.StringSliceVar(&setRepos, "repos", nil, "List of repo (names) to associate the policy with")

cmd/release/view/view.go

@@ -37,6 +37,7 @@ func New(bCtx *env.BubblyContext) *cobra.Command {
 		commit   string
 		repo     string
 		policies bool
+		log      bool
 	)
 	cmd := &cobra.Command{
 		Use:     "view [flags]",
@@ -76,6 +77,7 @@ func New(bCtx *env.BubblyContext) *cobra.Command {
 	cmd.Flags().StringVar(&commit, "commit", "", "Specify the git commit for which to view a release")
 	cmd.Flags().StringVar(&repo, "repo", "", "Specify the repo name for which to view a release. Requires also the --commit flag")
 	cmd.Flags().BoolVar(&policies, "policies", false, "Whether to also fetch the policies that apply to this release")
+	cmd.Flags().BoolVar(&log, "log", false, "Whether to also fetch the release log")
 
 	return cmd
 }
@@ -114,5 +116,16 @@ func printRelease(rel *api.ReleaseRead, policies bool) {
 		fmt.Println(columnize.SimpleFormat(violationLines))
 	}
 
+	if policies {
+		fmt.Println("")
+		fmt.Println("=== Log ===")
+		if len(rel.Entries) == 0 {
+			fmt.Println("No entries")
+		}
+		for _, e := range rel.Entries {
+			fmt.Println("  - " + e.Type.String() + " at " + e.Time.String())
+		}
+	}
+
 	fmt.Println("")
 }

docs/docs/adapters/adapters.md

@@ -8,4 +8,8 @@ keywords:
 - adapters
 ---
 
-TODO
+:::note TODO
+This documentation is a TODO.
+
+Please check back soon as we start to fill this page with useful examples of queries you may want to run.
+:::

docs/docs/getting-started/getting-started.md

@@ -11,15 +11,15 @@ keywords:
 - agent
 ---
 
-:::note Under Active Development
-The content for this page is *under active development*. It
-may be
-incorrect and/or
-incomplete and is liable to change at any time.
-:::
 
 ## Installation
 
+Bubbly is a single Go binary containing botht the server, client and UI. Run it using Docker or download and run natively.
+
+### Docker
+
+`docker run valocode/bubbly:<version>`
+
 ### Homebrew
 
 `brew install valocode/tap/bubbly`
@@ -31,44 +31,155 @@ Binaries for macOS, Windows and Linux AMD64/ARM are available to download on the
 
 ## Deployment
 
-### Single-Server
+The choice of database is important.
+Bubbly uses [entgo](https://entgo.io) which has the following [supported dialects](https://entgo.io/docs/dialects/).
+
+Entgo supports SQLite, and be default Bubbly uses an in-memory SQLite database.
+So if you just want to spin up Bubbly and try it, don't worry about a database choice for now but do not get angry when your data is not persisted... :)
+
+Once you have chosen your database, run bubbly with your preferred method:
+
+### Docker
+
+If using docker, it's easy to integrate Bubbly with any orchestrator.
+Either provide command line arguments or set environment variables to configure your backend database.
+
+### From binary
+
+```bash
+# run a demo instance of bubbly, which uses sqlite and has some dummy data so that
+# you can immediately see something
+bubbly demo
+
+# view the help to know the flags
+bubbly server --help
+
+# start the bubbly server
+bubbly server
+
+# skip the ui
+bubbly server --ui=false
+```
+
+Check the [documentation](../cli/bubbly_server)
+
+## Basic Commands
+
+There are a few main actions/commands in Bubbly.
+
+### Creating a Release
+
+Creating a release is done via the command line, so that it can be integrated into CI.
+
+```bash
+# view the help for creating a release
+bubbly release create --help
+
+# create a release using the local .git directory
+bubbly release create
+```
+
+This command will check for a `.bubbly.json` file which can be used to override the default release options (name and version).
+
+Check the [documentation](../cli/bubbly_release_create)
+
+### Applying a Policy
+
+Applying a policy is the way to define your Release Readiness policy.
+
+A policy is a set of rules, that either `require` certain events in the release log, or `deny` certain results.
+Each time a rule is violated, a violation is created against your release and the violation can be customised by the rule, such as setting the severity(`suggestion`, `warning` or `blocker`).
+
+Below is a policy that looks for `code_issues` with a severity of `high` and creates a violation if there are *any* such code issues.
+
+```rego
+package policy
+
+deny[violation] {
+ issues := code_issues()
+ high_issues := [issue | issues[i].severity == "high"; issue := issues[i]]
+ count(high_issues) > 0
+ violation = {
+  "message": sprintf("%d high issue(s)", [count(high_issues)]),
+  "severity": "warning",
+ }
+}
+```
+
+To save this policy, put it in a file called `code_issue_high_severity.rego` and use the Bubbly command line:
+
+```bash
+# view the help for bubbly policy command
+bubbly policy --help
+
+# save the policy and apply it to the project "default"
+bubbly policy save code_issue_high_severity.rego --set-projects default
+
+# alternatively do this in two steps: first save and then associate
+bubbly policy save code_issue_high_severity.rego
+bubbly policy set code_issue_high_severity --projects default
+```
+
+This policy now affects/applies to the project default.
+Any releases created within a repository belonging to the project default will inherit this policy.
+
+### Running an adapter
+
+Running an adapter is the way to get results into Bubbly. First you need an adapter to run.
+
+As a short example, here is an adapter for import [gosec](https://github.com/securego/gosec) results.
+
+```rego
+package adapter
 
-#### Pre-requisites
+code_scan[scan] {
+ scan := {
+  "tool": "gosec",
+  "metadata": {"env": {"some_var": "some_value"}},
+ }
+}
 
-Bubbly requires an existing postgresql database.
+code_issue[issue] {
+ some i
+ iss := input[_].Issues[i]
+ issue := {
+  # providing the i is necessary so that we get all unique code_issues
+  "i": i,
+  "rule_id": iss.rule_id,
+  "message": iss.details,
+  "severity": lower(iss.severity),
+  "type": "security",
+ }
+}
+```
 
-#### Running
+If we save this in a file called `gosec.rego` and then use the bubbly command line we can either run the adapter directly from the file, or save it and run by fetching it from the remote server.
 
-Start the [Bubbly Agent](../introduction/core-concepts#agent) in single server mode, 
-which starts all of the Agent's features on the same machine: 
+We will need a results file as input also, which can be found here (this is a result from running gosec over bubbly some time back): <https://raw.githubusercontent.com/valocode/bubbly/main/adapter/testdata/adapters/gosec.json>
 
-`bubbly agent`
+```bash
+# run the adapter by providing the path to it as input
+bubbly adapter run ./gosec.rego
+```
 
-The `bubbly agent` command defaults to connecting to a postgres database available on
-`postgres:5432`. Use `bubbly agent -h` for command-line flags and `bubbly env` for
-environment variables for configuring this.
+This command checks for a local git repository and uses the `HEAD` commit to associate the results with.
+A release must exist for this commit.
 
+Check the [documentation](../cli/bubbly_adapter_run.md)
 
-### Multi-Server
+### View Release
 
-:::note Under Active Development
-The content for this page is *under active development*. It
-may be
-incorrect and/or
-incomplete and is liable to change at any time.
-:::
+Either do this through the Bubbly UI or via the command line:
 
-In future, Bubbly will support a distributed deployment model in which you can
-freely customise the number of instances of each Bubbly Agent feature (as long
-as you have at least one of each).
+```bash
+# view the local release
+bubbly release view
 
-For this, you have two strategies:
+# view the local release with policies that apply to it
+bubbly release view --policies
+```
 
-1. Native: All communication between the running Bubbly Agents handled by our Bubbly-embedded NATS server.
-2. Custom: Roll your own dedicated [NATS server](https://github.com/nats-io/nats-server).
-   We recommend this strategy for more demanding deployments.
-
 ## Next Steps
 
-- [Tutorials: Getting Started](../tutorials/github-metrics) 
+- [Tutorials: Getting Started](../tutorials/github-metrics)
   provides a practical tutorial for immediately getting started with Bubbly