Merge branch 'main' into add-mutation-testing

CHANGELOG.md

@@ -1,5 +1,89 @@
 # Changelog
 
+## [12.4.0](https://github.com/uktrade/platform-tools/compare/12.3.0...12.4.0) (2024-12-06)
+
+
+### Features
+
+* DBTP-1568 - Add s3 support for cross environment service access ([#654](https://github.com/uktrade/platform-tools/issues/654)) ([7e1d75f](https://github.com/uktrade/platform-tools/commit/7e1d75f95cacb68e01f0f62448359166509c20b0))
+
+
+### Bug Fixes
+
+* DBTP-1498 - Add option for database dump filename ([#681](https://github.com/uktrade/platform-tools/issues/681)) ([d06ddcc](https://github.com/uktrade/platform-tools/commit/d06ddcc0253a76950f54b881af84be14b0981b66))
+* DBTP-1498 - Add validation for database copy pipeline ([#683](https://github.com/uktrade/platform-tools/issues/683)) ([cda1e7b](https://github.com/uktrade/platform-tools/commit/cda1e7bc9daa1732e9032c7d6566716e3151b961))
+
+
+### Documentation
+
+* Document new dbt-platform-helper architecture ([#669](https://github.com/uktrade/platform-tools/issues/669)) ([ae4862d](https://github.com/uktrade/platform-tools/commit/ae4862da9e3e3d39c82c99222fa21450191f260a))
+
+## [12.3.0](https://github.com/uktrade/platform-tools/compare/12.2.4...12.3.0) (2024-12-03)
+
+
+### Features
+
+* DBTP-1299 - Cross account database copy ([#657](https://github.com/uktrade/platform-tools/issues/657)) ([7d35599](https://github.com/uktrade/platform-tools/commit/7d35599533b55f15fb08801c50ce538a8a32b847))
+
+
+### Refactor
+
+* Improving provider structure and exception handling" ([#670](https://github.com/uktrade/platform-tools/issues/670)) ([331e8b8](https://github.com/uktrade/platform-tools/commit/331e8b89d60fec4e29a9ea4473ffa44cba8e92c7))
+
+## [12.2.4](https://github.com/uktrade/platform-tools/compare/12.2.3...12.2.4) (2024-12-02)
+
+
+### Bug Fixes
+
+* DBTP-1572 - Fix _validate_exension_supported_versions incorrectly raising an error when no version is supplied ([#660](https://github.com/uktrade/platform-tools/issues/660)) ([2ce98bf](https://github.com/uktrade/platform-tools/commit/2ce98bfdcd22b880867306e3181f4815e46c6acb))
+
+## [12.2.3](https://github.com/uktrade/platform-tools/compare/12.2.2...12.2.3) (2024-11-29)
+
+
+### Bug Fixes
+
+* DBTP-1524 Make subnet order from environment generate match CloudFormation exports ([#665](https://github.com/uktrade/platform-tools/issues/665)) ([f0f561b](https://github.com/uktrade/platform-tools/commit/f0f561beba2239f757fec62cd530483432bb953b))
+
+## [12.2.2](https://github.com/uktrade/platform-tools/compare/12.2.1...12.2.2) (2024-11-26)
+
+
+### Bug Fixes
+
+* Fixing json loads ([#664](https://github.com/uktrade/platform-tools/issues/664)) ([46eddff](https://github.com/uktrade/platform-tools/commit/46eddff14ba2460ebe4beee1378ac75b617a8821))
+
+## [12.2.1](https://github.com/uktrade/platform-tools/compare/12.2.0...12.2.1) (2024-11-26)
+
+
+### Miscellaneous Chores
+
+* Don't install poetry in Dockerfile.debian ([#655](https://github.com/uktrade/platform-tools/issues/655)) ([9ad8c67](https://github.com/uktrade/platform-tools/commit/9ad8c67d8abc8ad61a4123bb90d361b3e26eacd3))
+
+## [12.2.0](https://github.com/uktrade/platform-tools/compare/12.1.0...12.2.0) (2024-11-26)
+
+
+### Features
+
+* DBTP-1395 Add validation for new slack alert channel Id that will be set in <application>-alb in platform-config file ([#635](https://github.com/uktrade/platform-tools/issues/635)) ([729c082](https://github.com/uktrade/platform-tools/commit/729c0821bdbc96f49c832a79bf2211475a737bf9))
+* DBTP-1568 - Add s3 support for external role access ([#652](https://github.com/uktrade/platform-tools/issues/652)) ([02bebd6](https://github.com/uktrade/platform-tools/commit/02bebd6d331fd8a10cb317460a91634c5745b462))
+
+
+### Bug Fixes
+
+* DBTP-1577 Fix conduit (ecs) exec race condition  ([#656](https://github.com/uktrade/platform-tools/issues/656)) ([22eafa0](https://github.com/uktrade/platform-tools/commit/22eafa0c8388b3132663d953bf97c85887c94999))
+
+## [12.1.0](https://github.com/uktrade/platform-tools/compare/12.0.2...12.1.0) (2024-11-21)
+
+
+### Features
+
+* DBTP-1380 Get Opensearch/Redis versions from AWS API - Platform-tools changes/Caching of AWS API calls ([#624](https://github.com/uktrade/platform-tools/issues/624)) ([72d0dd7](https://github.com/uktrade/platform-tools/commit/72d0dd70396a4632e5cb5b1f6c80b2df772a89ad))
+* DBTP-1434 - CDN cache policy ([#642](https://github.com/uktrade/platform-tools/issues/642)) ([8cc2c0c](https://github.com/uktrade/platform-tools/commit/8cc2c0caf137889115c9d84c1c9895dae2a808c9))
+
+
+### Reverts
+
+* DBTP-1520 refactor conduit command ([#647](https://github.com/uktrade/platform-tools/issues/647)) ([7b56c5e](https://github.com/uktrade/platform-tools/commit/7b56c5e1a4324fbfb2585877dd38c4857c1544cc))
+
 ## [12.0.2](https://github.com/uktrade/platform-tools/compare/12.0.1...12.0.2) (2024-11-13)
 
 

README.md

@@ -42,8 +42,59 @@ If you are migrating a service to DBT PaaS, [GOV.UK PaaS to DBT PaaS Migration](
     poetry run pre-commit install
     ```
 
+### Platform Helper architecture
+
+`platform-helper` is split into the following layers:
+
+Commands (UI) -> Domains -> Providers
+
+Code written for `platform-helper` should adhere to the following architecture, shown below.
+
+![platform-helper architecture](doc/platform-helper-architecture.png)
+
+#### Commands
+
+This is the (essentially) UI level of `platform-helper`
+
+We try to follow a noun/domain verb/action pattern, `platform-helper thing action`.
+
+E.g. `codebase` has the following commands (all to do with the codebase Domain):
+
+- `build`
+- `deploy`
+- `list`
+- `prepare`
+
+Each command has an associated Domain. 
+
+There should be no business logic within the command as this is implelemented in the Domain layer.
+
+CLI arguments for a command are pulled in via [click](https://click.palletsprojects.com/en/stable/) and passed to the Domain along with any dependencies.
+
+#### Domains
+
+Domains are where the business logic for a given Command lives.
+
+Each Domain is a class.
+
+Any logged information (`click.secho`) from the Provider level should live within the Domain level.
+
+Any common/reusable elements should be implemented in a Provider.
+
+#### Providers
+
+Providers are groups of similar logic that are linked by the resource/tool/thing they use rather than the result of what their actions is.
+
+E.g. I have a method that lists *thing* from the *thing-service* AWS using a boto3 client.
+
+This method is not specific to the Domain so it should go into the *thing-service* Provider.
+
 ### Testing
 
+#### Testing approach
+
+See the following [Confluence](https://uktrade.atlassian.net/wiki/spaces/DBTP/pages/4325376119/Testing+approach) page the `platform-tools` testing approach
+
 #### Requirements
 
 The following tools are required to run the full test suite.
@@ -78,7 +129,9 @@ You can run the mutation tests locally, but:
 
 You may want to test any CLI changes locally.
 
-Run `poetry build` to build your package resulting in a package file (e.g. `dbt_platform_tools-0.1.40.tar.gz`) in a `dist` folder. You may need to bump up the package version before doing so.
+##### Option 1 - Build and install `platform-helper` from your local source code
+
+Run `poetry build` to build your package resulting in a package file (e.g. `dbt_platform_tools-0.1.40.tar.gz`) in a `dist` folder. You may need to bump up the package version before doing so. To bump the version go to `pyproject.toml/version`
 
 Copy the package file(s) to the directory where you would like to test your changes, and make sure you are in a virtual environment. Run `platform-helper --version` to check the installed package version (e.g. `0.1.39`).
 
@@ -90,6 +143,19 @@ Run `pip install <file>` and confirm the installation has worked by running `pla
 > [!IMPORTANT]
 > When testing is complete, do not forget to revert the `dbt-platform-helper` installation back to what it was; e.g. `pip install dbt-platform-helper==0.1.39`.
 
+##### Option 2 - Run the python files directly.
+
+This assumes that the virtual python environment where you are running them from already has the dependencies installed and the directory is at the same level as your platform-tools directory.
+
+Example usage:
+
+```
+# From <application>-deploy
+
+../platform-tools/platform_helper.py <command> <options>
+```
+
+
 #### End to end testing
 
 Because this codebase is only fully exercised in conjunction with several others, we have [platform-end-to-end-tests](https://github.com/uktrade/platform-end-to-end-tests), which orchestrates the testing of them working together.

dbt_platform_helper/COMMANDS.md

@@ -256,7 +256,8 @@ platform-helper codebase deploy --app <application> --env <environment> --codeba
 
 [↩ Parent](#platform-helper)
 
-    Create a conduit connection to an addon.
+    Opens a shell for a given addon_name create a conduit connection to
+    interact with postgres, opensearch or redis.
 
 ## Usage
 
@@ -272,11 +273,11 @@ platform-helper conduit <addon_name>
 ## Options
 
 - `--app <text>`
-  - AWS application name
+  - Application name
 - `--env <text>`
-  - AWS environment name
+  - Environment name
 - `--access <choice>` _Defaults to read._
-  - Allow write or admin access to database addons
+  - Allow read, write or admin access to the database addons.
 - `--help <boolean>` _Defaults to False._
   - Show this message and exit.
 
@@ -764,6 +765,7 @@ platform-helper database (dump|load|copy)
 ```
 platform-helper database dump --from <from_env> --database <database> 
                               [--app <application>] [--from-vpc <from_vpc>] 
+                              [--filename <filename>] 
 ```
 
 ## Options
@@ -776,6 +778,8 @@ platform-helper database dump --from <from_env> --database <database>
   - The name of the database you are dumping data from
 - `--from-vpc <text>`
   - The vpc the specified environment is running in. Required unless you are running the command from your deploy repo
+- `--filename <text>`
+  - Specify a name for the database dump file. Recommended if the same dump database is being used for multiple load environments
 - `--help <boolean>` _Defaults to False._
   - Show this message and exit.
 
@@ -790,7 +794,7 @@ platform-helper database dump --from <from_env> --database <database>
 ```
 platform-helper database load --to <to_env> --database <database> 
                               [--app <application>] [--to-vpc <to_vpc>] 
-                              [--auto-approve] 
+                              [--filename <filename>] [--auto-approve] 
 ```
 
 ## Options
@@ -805,6 +809,8 @@ platform-helper database load --to <to_env> --database <database>
   - The vpc the specified environment is running in. Required unless you are running the command from your deploy repo
 - `--auto-approve <boolean>` _Defaults to False._
 
+- `--filename <text>`
+  - Specify a name for the database dump file. Recommended if the same dump database is being used for multiple load environments
 - `--help <boolean>` _Defaults to False._
   - Show this message and exit.
 

dbt_platform_helper/commands/application.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 
 # application commands are deprecated, do not spend time refactoring them
+# Service teams are trained to use them as a replacement for cf app(s)
 
 import time
 from datetime import datetime