Change Kustomize so that policies can be enforced as groups

[galshi]

@michaelkotelnikov I'm still not sure about the design, I would like your input.
I haven't checked that everything works properly, I'll proceed once you approve.

[momoadc]

נראה טוב

[michaelkotelnikov]

@galshi I like the changes -

• It's more native to have a kustomization file per directory, instead of having a large one.
• The way you can control the enforcementAction from the kustomization file.

Some points to think about -

• I would tweak the docu a bit, it's not very clear atm. Maybe add another section for Applying all policies as well.
• Note that the current state of the kustomization is not too native. If you apply the resources, all constraints will fail unless your GitOps tool is consistently syncing. The constraints.yaml depends on the template.yaml. If the ConstraintTemplate is not created, the resource at the constraints.yaml fails to create.

$ kustomize build ./ | oc apply -f -
constrainttemplate.templates.gatekeeper.sh/k8sdisallowemptydir created
error: unable to recognize "STDIN": no matches for kind "K8sDisallowEmptyDir" in version "constraints.gatekeeper.sh/v1beta1"

[galshi]

@michaelkotelnikov
I added a section about applying all policies, I don't think we can do something about your second point, OPA's CRs and CRDs aren't designed very well.
It also looks like removing policies isn't possible,
you can include multiple kustomization files for each policy-group instead of importing all of them through the main kustomization. You still can't disable individual policies though (you would have to clone this repo and remove undesired policies, like the flow described at the end of the paragraph in the link above).

I don't think there's anything better which can be done with native kustomize.

[michaelkotelnikov]

@galshi I'd add an example of implementation using ArgoCD at the end - which configurations should be applied in the Application for both template and constraint resources to be created and not result in constraint creation failure due to kustomization issues.

I know it might be an overkill, but as for now, if a simple user tries to use the examples we provide, it'll fail. The issue should be covered, and a solution needs to be provided for a complete GitOps solution.

[galshi]

Need to evaluate implementing this with Kustomize components.