Function unable to run intermittently when multiple claims applied to cluster

[markphillips100]

What happened?

When applying multiple claims to a cluster using a single manifest file (using "---" separator between claims) Crossplane initially reports the following error when describing most of the composites:

"Message: cannot compose resources: cannot run Composition pipeline step "normal": cannot run Function "kcl-function": rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Eventually all composites are reconciled correctly but it would appear like the function cannot handle concurrent execution perhaps? This has the effect of degrading the performance of reconciling many resources at once.

NOTE: Applying each claim separately and waiting for a result works without the error occurring.

How can we reproduce it?

Create a composite and apply several claims in one go. Observe each of the composites to see the error message.

What environment did it happen in?

Function version: v0.4.0
Crossplane: v1.15.0
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.0

[markphillips100]

Also seeing something a bit worrying in the events of one composite whereby it reported the following:

Warning  ComposeResources         117s (x2 over 7m)      defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose resources: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to compile the kcl package
EvaluationError
attribute 'replicationLocations' of Config is required and can't be None or Undefined

The "replicationLocations" property ONLY exists in a completely separate composition to the one it got reported in. How is that possible? Some issue related to the concurrency and the fact all of my compositions have metadata.name: Basic maybe?

[markphillips100]

Also seeing the following message in some composites. So multiple things not happy with running functions concurrently for different compositions:

  Warning  ComposeResources  85s (x21 over 19m)  defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose resources: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to compile the kcl package
run linker failed: stdout , stderr: /tmp/.kclvm/cache/0.8.3-0c20ab3eb4b9179219d6837a57f5d352/x86_64-unknown-linux-gnu/models.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

[markphillips100]

I made a mistake earlier saying all composites eventually reconciled. The composite that reported the previous error still has not. Maybe it would if I waited a really long time but it's an issue nonetheless.

[Peefy]

Make sense. This is because the KCL basic image has set a common cache path /tmp instead of the kcl.mod path for different modules. Later, I will unset this environment variable and rebuild the function-kcl v0.5.1 image.

[markphillips100]

Thanks @Peefy . Would that explain all the issues I was seeing do you think?

[Peefy]

Yes.

[Peefy]

I've fixed it in the PR #69 and released the function-kcl v0.5.1 package.
https://marketplace.upbound.io/functions/crossplane-contrib/function-kcl/v0.5.1

[markphillips100]

Yep installing already :-) Thanks for the speedy fix @Peefy

[markphillips100]

@Peefy still seeing the errors using v0.5.1

Describe status of one of the composites:

Status:
Conditions:
Last Transition Time: 2024-04-16T06:25:23Z
Message: cannot compose resources: cannot run Composition pipeline step "normal": cannot run Function "kcl-function": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Reason: ReconcileError
Status: False
Type: Synced
Events:
Type Reason Age From Message

---

Normal SelectComposition 27s defined/compositeresourcedefinition.apiextensions.crossplane.io Successfully selected composition: document-composite
Normal SelectComposition 27s defined/compositeresourcedefinition.apiextensions.crossplane.io Selected composition revision: document-composite-2a97c1f
Warning ComposeResources 18s defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose resources: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to compile the kcl package
run linker failed: stdout , stderr: /tmp/.kclvm/cache/0.8.3-0c20ab3eb4b9179219d6837a57f5d352/x86_64-unknown-linux-gnu/models.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
Warning ComposeResources 8s defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose resources: cannot run Composition pipeline step "normal": cannot run Function "kcl-function": rpc error: code = DeadlineExceeded desc = context deadline exceeded

[Peefy]

Sorry, I've released https://marketplace.upbound.io/functions/crossplane-contrib/function-kcl/v0.5.2 with the PR #70

[markphillips100]

Thank you. Will let you know shortly if all good.

[markphillips100]

I don't see the models.o compile issue anymore but I do see the "context deadline exceeded". It appeared in a few composite events but eventually they came good and the KCL code executed for each of the composites.

This shows there seems to be still some conflict re /tmp across concurrent usage:

Warning ComposeResources 7m49s defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose resources: pipeline step "normal" returned a fatal result: failed to run kcl function pipelines: failed to compile the kcl package
symlink /tmp/k8s_1.29 /tmp/k8s: file exists
Warning ComposeResources 6m59s (x5 over 7m39s) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose resources: cannot run Composition pipeline step "normal": cannot run Function "kcl-function": rpc error: code = DeadlineExceeded desc = context deadline exceeded

[Peefy]

cc @zong-zhe Could you please help to fix it?

[markphillips100]

FYI all the compositions do complete eventually this time. Still think that last error needs a resolution though.

[markphillips100]

Hi @Peefy is 0.5.3 (with the lock) something you want me to test?

[Peefy]

Haven't Yet. @zong-zhe he is doing some fix for the package symlink.

[markphillips100]

Ah okay. Cool.