Document release process in the code base

.gitignore

@@ -93,3 +93,6 @@ package.json
 artifacts
 deposit-*
 cabal.project.freeze
+
+# Emacs
+*~

docs/site/src/SUMMARY.md

@@ -54,8 +54,14 @@
       - [Testing](contributor/how/testing.md)
       - [Continuous Integration](contributor/how/continuous-integration.md)
       - [Release Process](contributor/how/release-process.md)
-        - [Release checklist](contributor/how/release-checklist.md)
+        - [Release Checklist template](contributor/how/release-checklist.md)
       - [Code Review Guidelines](contributor/how/code-review-guidelines.md)
     - [Notes](contributor/notes.md)
       - [Updating Dependencies](contributor/notes/updating-dependencies.md)
       - [Notes from upgrading GHC version](contributor/notes/notes-from-upgrading-ghc-version.md)
+    - [Decisions Record](contributor/decisions.md)
+      - [2022-10-04 - Document Storage](contributor/decisions/2022-10-04-document-storage.md)
+      - [2023-01-27 - Continuous Integration](contributor/decisions/2023-01-27-continuous-integration.md)
+      - [2023-07-28 - Team Workflow](contributor/decisions/2023-07-28-workflow-review.md)
+      - [2024-03-23 - Release Process](contributor/decisions/2024-03-13-release-process.md)
+      - [2024-12-03 - Store documents alongside code](contributor/decisions/2024-12-03-document-with-code.md)

docs/site/src/contributor/decisions.md

@@ -0,0 +1,3 @@
+# Decisions Record
+
+We keep in these documents a timestamped record of process related decisions.

docs/site/src/contributor/decisions/2022-10-04-document-storage.md

@@ -0,0 +1,42 @@
+# **Document Storage :: AP**
+
+|         |            |
+|---------|------------|
+| Started | 2022-09-30 |
+| Decided | 2022-10-14 |
+
+## **Why**
+
+We need to store our documents — such as decision records, design documents — somewhere, but there are a plethora of options: Google Docs, Confluence, Github, Custom wiki, … with various pros and cons. This decision records our current choice and its rationale.
+
+## **Decision**
+
+The entry point to our documentation is the [Adrestia (Haskell) Team Dashboard](https://docs.google.com/document/d/1jAmRZDLG7_0VOS88dHvaJDbRoVSvfn_BiL2FXBuk3Z4/edit?skip_itp2_check=true&pli=1#) on Google Docs. All other documents are hyperlinked from this dashboard.
+
+For documents that are not public, such as meeting minutes or Advice Process records, we use Google Docs.
+
+For documents that may eventually be public, such as design documentation or release checklists, we use Rodney’s HedgeDoc instance at [https://md.adrestia.iohkdev.io/](https://md.adrestia.iohkdev.io/) . (As Rodney is no longer with us, I’m actually not happy with that state of affairs. Moritz Angerman has set up another instance at [https://hd.devx.iog.io/](https://hd.devx.iog.io/) ).
+
+## **Details**
+
+See Section “Decision”.
+
+## **Rationale**
+
+**Making** a **decision** — Document storage systems come with various trade-offs, and none seems to be perfect. The act of simply picking one system is at least as important as making sure that the pick is reasonable — we can’t even record our choice unless we have already made it\! Thus, I (the decision maker) have decided to favor a speedy decision over an optimal one.
+
+What do we want out of a document storage system? Here is a list of features, in order of importance, and some thoughts as to how these are satisfied with my current decision.
+
+**Sharing** and **Access Control** — We need to be able to share documents with team members, and maybe external parties. Also, we need to keep some types of documents internal to the company or team.
+
+**Readability** — Documents should be pleasant on the eyes. For me, this means that they should have a rendered view, but they do not need to be WSIWYG.
+
+**Commenting** and **Tracking Changes** — It should be pleasant to comment on documents and track changes that different contributors made. For me, this means that it should be possible to comment on the *rendered* view. For example, markdown documents in a Github repository do not satisfy this property, as I can only see changes in the source code, and comments are also attached to the source code rather than the rendering.
+
+**Writing format** — Documents should be easy to export to Markdown, so that we can more easily move them between different storage solutions, e.g. from HedgeDoc to our Github repositories. Unfortunately, Google Docs does not allow easy export to Markdown; this is particularly painful for documents with a lot of source code — hence we use HedgeDoc for more technical documents.
+
+**Navigation** — Documents should be easy to **search** (“I know what I’m looking for, but I cannot find it.”) and to **discover** (“I don’t know what I’m looking for, but I find it anyway”).
+
+From my experience, good discoverability comes from good curation. In my view, Hyperlinks are the best tool for curating document listings. For example, Confluence has a hierarchical view for each space, but I still find it hard to navigate if the sections are not curated properly. The [Haskell wikibook](https://en.wikibooks.org/wiki/Haskell) uses ToC templates and manually curated tables of hyperlinks, I think it works very well.
+
+For searching a document, curation helps, too, but a search box is usually more effective.

docs/site/src/contributor/decisions/2023-01-27-continuous-integration.md

@@ -0,0 +1,47 @@
+# **Continuous Integration**
+
+|              |            |
+|--------------|------------|
+| Started      | 2022-12-06 |
+| Decided      | 2023-01-27 |
+| Last amended | 2023-04-04 |
+
+## **Why**
+
+The sudden decommission of Hydra forces us to revisit our Continuous Integration (CI) setup.
+
+## **Decision**
+
+We predominantly rely on [Buildkite](https://buildkite.com) as CI system. For Windows, we have to use Github Actions.
+
+We build **artifacts** and run **checks** on them. Artifacts include compiled executables, but also the source code itself. Checks include unit and integration tests, but also source code linters. We specify our artifacts in \`flake.nix\`, and most of our checks, too.
+
+We perform these builds and checks with different **granularity** — in order to keep our computing resources within reasonable limits, we don’t automatically check everything on every commit. Instead, the granularities are:
+
+* `post-commit`	\= (at most once) after each commit; for a quick sanity check after a push
+* `pre-merge`	\= before merging each pull request; for a reasonably complete, automated check that master will satisfy functional requirements
+* `post-merge`	\= after merging each pull request to master; for an exhaustive check that master satisfies functional requirements on all platforms
+* `nightly`		\= every night; for an exhaustive, automated check of functional and non-functional requirements
+
+The following **table** lists our artifacts, the checks performed on them (\`.\` for build), the granularity at which the check is performed, and the CI system used for doing that:
+
+| Artifact | Check | Granularity | CI System | (Status) |
+| :---- | :---- | :---- | :---- | ----- |
+| Source code | Code formatting style | post-commit | Buildkite | 🔵 |
+| Documentation | . | post-commit | Github Action | 🔵 |
+| Pull request (PR) | Mergeable to master concurrently with other PRs | pre-merge | Bors | 🔵 |
+| Compiled modules | . | post-commit | Buildkite | 🔵 |
+|  | Unit tests (linux) | post-commit | Buildkite | 🔵 |
+|  | Unit tests (macos) | post-merge | Buildkite | 🔵 |
+|  | Unit tests (windows) | nightly | Github Action | 🔵 |
+| Executables / Release archive | . (linux) | pre-merge | Buildkite | 🟡[ADP-2502](https://cardanofoundation.atlassian.net/browse/ADP-2502)  |
+|  | . (macos) | post-merge | Buildkite | 🔵  |
+|  | . (windows, cross-compiled) | pre-merge | Buildkite | 🔵 |
+| Executables | Integration tests (linux) | pre-merge | Buildkite | 🔵 |
+|  | ~~Integration tests (macos)~~ |  | Buildkite | 🔴[ADP-2522](https://cardanofoundation.atlassian.net/browse/ADP-2522) |
+|  | ~~Integration tests (windows)~~ |  | Github Action | 🔴[ADP-2517](https://cardanofoundation.atlassian.net/browse/ADP-2517) |
+|  | Benchmarks | nightly | Buildkite | 🔵 |
+| Release archive | E2E tests | nightly | Github Action | 🔵  |
+| Docker image | . | post-commit | Buildkite | 🔵 |
+
+Legend: Status 🔵\= working; 🟡= needs work; 🔴\= not working

docs/site/src/contributor/decisions/2023-07-28-workflow-review.md

@@ -0,0 +1,34 @@
+# **Workflow Review **
+
+|         |            |
+|---------|------------|
+| Started | 2023-07-07 |
+| Decided | 2023-07-28 |
+
+
+## **Why**
+
+As we have now moved to the Cardano Foundation, it’s an appropriate time to review how we operate as a team and see what improvements we can implement.
+
+## **Decision**
+
+1. Due to the distributed nature of the team, where possible and practical we will continue to favor asynchronous working.
+2. We will adopt a Kanban-esque approach to managing our workflow, therefore:
+   1. Work items will be continuously prioritized in line with agreed goals and roadmap.
+   2. Weekly planning & replenishment meetings will be held.
+   3. We will no longer plan work into sprints or iterations.
+   4. Updated meetings and ceremonies:
+
+| Meeting / Activity            | Frequency             | How                                                                         |
+|:------------------------------|:----------------------|:----------------------------------------------------------------------------|
+| Daily Updates                 | Daily / as applicable | Provided in slack \#hal-daily                                               |
+| Planning & Replenishment      | Weekly                | Friday meeting  (½ hr)                                                      |
+| Development Meeting           | Weekly                | Wednesday (1hr)                                                             |
+| Backlog Refinement & Grooming | Continuously          | Asynchronously, supported by breakout sessions and ad hoc team meetings.    |
+| Demos                         | Ad hoc                | Scheduled upon feature completion. Will demo as part of a feature showcase. |
+| Retro                         | Quarterly / ad hoc    | Deep Dive meeting (3 hrs)                                                   |
+
+3. To determine how best to implement an internal team “No Meeting Day”, this will be progressed via its own advice process.
+4. Public team channel (\#hal-public) created to facilitate communications and collaboration within the Cardano Foundation and IOG.
+5. Current ticketing system JIRA will continue to be used, subject to future Advice Process.
+6. Cardano Wallet [public forum on Github](https://github.com/cardano-foundation/cardano-wallet/discussions) to be utilized for engaging with the community and for also announcing new releases.

docs/site/src/contributor/decisions/2024-03-13-release-process.md

@@ -0,0 +1,24 @@
+# Update Release Process
+
+|         |            |
+|---------|------------|
+| Started | 2023-07-19 |
+| Decided | 2024-03-23 |
+
+## Why
+
+The release v2023-07-18 of cardano-wallet highlighted the need to further automate the release process, and to clarify consistency between artifacts and tests. The release v2023-12-18 highlighted further gaps in test and dependency maintenance.
+
+## Decision
+
+* We continue to use **trunk-based development**, where all code is merged into the master branch frequently, such that this branch is (almost) ready to be released at any point in time.
+* A **release** is a collection of artifacts that have been tested together and will be published.
+  * We **create** a release from a specific Git commit, marked by a **Git tag**.
+  * We create and **test artifacts** without human involvement using **automation**.
+  * We publish human-readable release notes, specifically a **changelog** and a list of **known issues**.
+  * We **publish** a release by clicking a button on [Github Releases](https://docs.github.com/en/repositories/releasing-projects-on-github). Artifacts are automatically pushed to other platforms.
+* The release is made under **human supervision** using a **release checklist**.
+
+## Details
+
+see [Release process](../how/release-process.md)

docs/site/src/contributor/decisions/2024-12-03-document-with-code.md

@@ -0,0 +1,48 @@
+# **Store Documents alongside Code**
+
+|         |             |
+|---------|-------------|
+| Started | 13 Nov 2024 |
+| Decided | 3 Dec 2024  |
+
+## **Why**
+
+When developing open-source projects, it helps if all the core “artifacts” related to the project are easily accessible along with the code:
+
+- Links between various artifacts, between code and documents, or documents and code, are easier to add, verify, and maintain over time
+- Versioning everything in the same way increases the likelihood of maintaining consistency over time
+- It reduces “friction” as one only needs to check-out the project and look in some folder to find the needed information
+- Documents can be used as build resources to produce e.g. websites or PDFs, and conversely up-to-date code can be included in documents if needed
+- Wannabe contributors have all the needed context to contribute more effectively
+
+## **Decision**
+
+Store all documents related to the team's *product* (roadmap, design documents, specifications) and *process* (Advice Process decisions, Logbook, Meeting minutes and recordings, etc.) to the relevant repository (ie. usually https://github.com/cardano-foundation/cardano-wallet).
+
+## **Details**
+
+* This Advice Process supersedes [Document Storage](2022-10-04-document-storage.md)
+* The documents shall be moved manually and converted to markdown as needed, i.e. we won't waste time migrating everything all at once and might keep some links to Google docs active for some time
+* New documents shall be created in the relevant repository
+* Folder structure is not fixed in this advice, and will be defined and refined as needed. Tentative structure could be:
+  * docs/ : toplevel documentation directory
+  * docs/process : advice process directory
+  * docs/process/README.md: contains the Advice Process logbook and summary
+  * docs/specifications: specification documents
+* The team maintains several code repositories but for the time being, we still consider cardano-wallet to be our main focus, therefore more general documents related to process or products should go there.
+* Private documents will be kept in Google Docs
+* Public documents that require some collaborative writing or editing process might live for a while in Google Docs or [https://hackmd.io](https://hackmd.io). The latter should be preferred as it ensures we stick to markdown format.
+
+## **Rationale**
+
+The “Why” section already states a few benefits of keeping documents in the same place as the code the team is responsible for. The main rationale behind this proposal is to increase the visibility of the open source Cardano community on the work the HAL team is doing, in order to maximise the likelihood this work will be deemed relevant and important, increase or lower the barrier of entry for potential contribution, and more generally increase the level of transparency of our work.
+
+The original Advice Process’ rationale stated some requirements for documents storage which were valid and this proposal tries to stick with those:
+
+- **Sharing** and **Access Control**: There are very few documents we need to keep private and by default, everything related to the product and process should be public, so no need for specific access control policies
+- **Readability**: GH is able to render sophisticated document, and if needed it's possible to enrich raw markdown to produce web sites or PDFs
+- **Commenting** and **Tracking Changes**: it's possible and easy to comment through change proposals. While it's not possible to comment directly on rendered document, I don't think this is a showstopper as the documents we manage are not so sophisticated the rendering is completely decorrelated from the source
+- **Writing format**: markdown is very versatile and easy to work with. Should we need a WYSIWYG collaborative editor, it's always possible to work in a [https://hackmd.io](https://hackmd.io) document, even in private, and commit the result
+- **Navigation**
+  - **Search**: repository is indexed and easily searchable through GH interface or directly on the filesystem
+  - **Discover**: ditto

docs/site/src/contributor/how/release-checklist.md

@@ -0,0 +1,53 @@
+> This is the latest release checklist template that one needs to go through in order to check quality of the release
+
+## Prepare the release process
+
+- [ ] Made a copy of the latest release
+      [Release-vYYYY-MM-DD]()
+      documented as a new page on cardano-wallet's wiki called [Release-vYYYY-MM-DD']()
+      and substituted the version number in the title and the links.
+- [ ] Picked up the latest green commgit from the release pipeline and prove it's green by linking the successful buildkite and github actions builds here.
+    - [ ] [release pipeline build](https://buildkite.com/cardano-foundation/cardano-wallet-release/)
+    - [ ] check for performance regressions on the `release commit` artifacts:
+    - [ ] check that the [restoration benchmarks](https://buildkite.com/cardano-foundation/cardano-wallet-restoration-benchmarks) ran in
+- [ ] Unblock the release block on [release pipeline build](https://buildkite.com/cardano-foundation/cardano-wallet-release/)
+
+
+## Update the release page
+
+- [ ] Fix the release note on the draft release automatically created by the release pipeline
+    https://github.com/cardano-foundation/cardano-wallet/releases
+
+## Check out the docker image is on the docker hub
+
+- [ ] link: https://hub.docker.com/r/cardanofoundation/cardano-wallet/tags
+- [ ] Report the link in the release notes
+     [link to docker hub](https://hub.docker.com/layers/cardanofoundation/cardano-wallet/)
+
+## Check out the bump.sh portal has the new release
+
+- [ ] https://bump.sh/hal-cardano-foundation/doc/cardano-wallet-backend
+
+## Check sensitive fields in the API
+
+- [ ] Verify that sensitive fields listed in
+  [Cardano/Wallet/Api/Server](https://github.com/cardano-foundation/cardano-wallet/blob/89faf170f388f9b475974896c349ad7676f0f44c/lib/exe/lib/Cardano/Wallet/Application/Server.hs#L128)
+  are still accurate and aren't missing any new ones.
+  ```
+  sensitive =
+      [ "passphrase"
+      , "old_passphrase"
+      , "new_passphrase"
+      , "mnemonic_sentence"
+      , "mnemonic_second_factor"
+      ]
+  ```
+- [ ] Check out the documentation has been published by the [github action](https://github.com/cardano-foundation/cardano-wallet/actions/)
+
+## Publication
+
+- [ ] Once two engineers have signed off, publish the release draft.
+
+- [ ] Merge the (administrative) commits created for the release tag back into the `master` branch.
+
+- [ ] Remember to leave this checklist in an up-to-date status for the next releaser