[Utilities] Introduced Set-Module script (#4067)

* Moved diverse scripts around and updated them to latest AVM

* Moved further scripts

* Small fix

* Further fixes

* Several improvements and fixes

* Private registry specific pdates

* Updated docs

* Updated path

* Renamed function

* Ran generation for KVLT to enable testing of pipeline

* Updated & reduced tests where it made sense

* Fixed pester tests

* Moved cross-ref script back to tools

* Moved local-ref script back to pipes

* Fixed ref

* Fixed ref

* Fixed ref

* Fixed ref

* Push updated Readme file(s)

* Rollback test changes

* Updated test file overview markdown via scrip

* Roll KVLT back post updates

* Updated AAD considerations

* Removed default headers

* Update to latest

* Enabled notes

* Update to latest

* Update to latest

* Update to latest

* Update to latest

* Remvoed notes feature

* ReadMe rollback (as handled in different PR)

* Added latest improvements

* Updated readme script to ONLY allow notes and delete all else

* Added support for top-level diff

* Updated logic that detects notes

* Update to latest

* Added should process

* Added cross ref cache

* Small fix

* Fixed script refs

* Another fix

* Improve Cross ref runtime

* Update to latest

* Update docs/wiki/Contribution guide - Generate module readme.md

Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

* Update docs/wiki/Contribution guide - Generate module readme.md

Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

* Added docs

* Ajusted usage examples header

* Renamed header

* Regenerated modules 1-60  (#4084)

* Regenerated first 60 files

* Small fixes

* Fixed templates

* Regen

* Renamed header

* Regenerated modules 61-120 (#4085)

* Regen readmes 2

* Fixed script refs

* Fixed templates

* Regen

* Renamed header

* Regenerated modules 121-180 (#4086)

* Regen readmes 3

* Fixed templates

* Regen

* Renamed header

* Regenerated modules 181-240 (#4087)

* Regen readmes 4

* Fixed templates

* Regen

* Renamed header

* Regenerated modules 300-366 (#4089)

* Regen readmes 6

* Fixed templates

* Small update

* Regen

* Renamed header

* Regenerated modules 241-300 (#4088)

* Regen readmes 5

* Fixed templates

* Regen

* Renamed header

* Updated readme script

* Updated usage example description

* [Utilities] Enabled progress bar on module update & depth param (#4098)

* Tested logic & added depth param

* Added finally block

* Enabled cancel

* ReadMe fallback

* Added docs

* Added silent continue on finally for non-cancel

* Update utilities/tools/Set-Module.ps1

Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

---------

Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

* Cross ref fix

* Update to latest

* Added verbosity

* Added verbosity

* Update to latest

* Adjusted syntax

* Adjusted syntax

* Adjusted verbosity

* Added force

* Cleanup

---------

Co-authored-by: CARMLPipelinePrincipal <CARML@noreply.github.com>
Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

docs/wiki/Contribution guide - Generate module readme.md

@@ -1,6 +1,13 @@
 As per the module design structure, every module in the CARML library requires a ReadMe markdown file documenting the set of deployable resource types, input and output parameters and a set of relevant template references from the official Azure Resource Reference documentation.
 
-The ReadMe generator utility aims to simplify contributing to the CARML library, as it supports creating the module ReadMe markdown file from scratch or updating it.
+The `Set-Module` utility aims to simplify contributing to the AVM library, as it supports
+- idempotently generating the AVM folder structure for a module (including any child resource)
+- generating the module's ReadMe file from scratch or updating it
+- compiling/building the module template
+
+To ease maintenance, you can run the utility with a `Recurse` flag from the root of your folder to update all files automatically.
+
+> **Note:** If you want to add any non-generated content to the Readme you can do so by adding it to a `## Notes` section at the bottom of the corresponding readme.
 
 ---
 
@@ -13,14 +20,18 @@ The ReadMe generator utility aims to simplify contributing to the CARML library,
 ---
 # Location
 
-You can find the script under [`/utilities/tools/Set-ModuleReadMe.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities/tools/Set-ModuleReadMe.ps1)
+You can find the script under [`/utilities/tools/Set-Module.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities/tools/Set-Module.ps1)
 
 # How it works
 
-1. Using the provided template path, the script first converts it to ARM/JSON if necessary (i.e., if a path to a Bicep file was provided)
-1. If the intended readMe file does not yet exist in the expected path, it is generated with a skeleton (with e.g., a generated header name)
-1. The script then goes through all sections defined as `SectionsToRefresh` (by default all) and refreshes the sections' content (for example, for the `Parameters`) based on the values in the ARM/JSON Template. It detects sections by their header and always regenerates the full section.
-1. Once all are refreshed, the current ReadMe file is overwritten. **Note:** The script can be invoked combining the `WhatIf` and `Verbose` switches to just receive an console-output of the updated content.
+Using the provided template path, the script
+1. validates the module's folder structure
+   - To do so, it searches for any required folder path / file missing and adds them. For several files, it will also provide some default content to get you started. The sources files for this action can be found [here](https://github.com/Azure/ResourceModules/tree/main/utilities/tools/helper/src)
+1. compiles its bicep template
+1. updates the readme (recursively, specified)
+   1. If the intended ReadMe file does not yet exist in the expected path, it is generated with a skeleton (with e.g., a generated header name)
+   1. The script then goes through all sections defined as `SectionsToRefresh` (by default all) and refreshes the sections' content (for example, for the `Parameters`) based on the values in the ARM/JSON Template. It detects sections by their header and always regenerates the full section.
+   1. Once all sections are refreshed, the current ReadMe file is overwritten. **Note:** The script can be invoked combining the `WhatIf` and `Verbose` switches to just receive an console-output of the updated content.
 
 # How to use it
 

docs/wiki/Interoperability - Bicep to ARM conversion.md

@@ -5,7 +5,7 @@ However, for users who still prefer using ARM templates over Bicep, the CARML li
 
 This page documents the conversion utility and how to use it.
 
-> **NOTE:** As Bicep & ARM template files work slightly different (e.g., references as specified differently), the ReadMe we generate out of them using the [`/utilities/tools/Set-ModuleReadMe.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities/tools/Set-ModuleReadMe.ps1) utility may look differently. To this end, make sure to regenerate all ReadMEs after you converted the repository from Bicep to ARM. If you don't, the Pester tests in the pipeline may fail when reviewing the ReadMEs.
+> **NOTE:** As Bicep & ARM template files work slightly different (e.g., references as specified differently), the ReadMe we generate out of them using the [`/utilities/tools/Set-Module.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities/tools/Set-Module.ps1) utility may look differently. To this end, make sure to regenerate all ReadMEs after you converted the repository from Bicep to ARM. If you don't, the Pester tests in the pipeline may fail when reviewing the ReadMEs.
 
 ---
 

docs/wiki/The CI environment - Static validation.md

@@ -35,15 +35,6 @@ The following activities are performed by the [`utilities/pipelines/staticValida
 - **Module tests**
   - **Readme content tests**
     1. `README.md` file should not be empty.
-    1. `README.md` file should contain these sections in order: Navigation, Resource Types, Parameters, Outputs, Cross-referenced modules, Deployment examples.
-    1. Resources section should contain all resources from the template file.
-    1. Resources section should not contain more resources than the template file.
-    1. Parameters section should contain a table for each existing parameter category in the following order: Required, Conditional, Optional, Generated.
-    1. Parameter tables should provide columns in the following order: Parameter Name, Type, Default Value, Allowed Values, Description. Each column should be present unless empty for all the rows.
-    1. Parameters section should contain all parameters from the template file.
-    1. Outputs section should contain a table with these column names in order: Output Name, Type.
-    1. Output section should contain all outputs defined in the template file.
-    1. Dependencies section should contain all cross-references defined in the template file.
     1. `Set-ModuleReadMe` script should not apply any updates.
   - **Compiled ARM template tests**
     1. Compiled ARM template should be latest.

docs/wiki/The library - Module design.md

@@ -88,7 +88,7 @@ Microsoft.Sql
   └─ databases [child-module/resource]
 ```
 
-In this folder, we recommend to place the child resource-template alongside a ReadMe (that can be generated via the [Set-ModuleReadMe](./Contribution%20guide%20-%20Generate%20module%20Readme) script) and optionally further nest additional folders for it's child resources.
+In this folder, we recommend to place the child resource-template alongside a ReadMe (that can be generated via the [Set-Module](./Contribution%20guide%20-%20Generate%20module%20Readme) script) and optionally further nest additional folders for it's child resources.
 
 The parent template should reference all it's direct child-templates to allow for an end-to-end deployment experience while allowing any user to also reference 'just' the child resource itself. In case of the SQL server example, the server template would reference the database module and encapsulate it in a loop to allow for the deployment of multiple databases. For example
 

modules/aad/domain-service/.test/common/main.test.bicep

@@ -1,5 +1,8 @@
 targetScope = 'subscription'
 
+metadata name = 'Using large parameter set'
+metadata description = 'This instance deploys the module with most of its features enabled.'
+
 // ========== //
 // Parameters //
 // ========== //