[UII] Add status tracking for agentless integrations (elastic#199567)

## Summary

Resolves elastic/ingest-dev#3933. For
deployments that support agentless, integrations with agentless
deployment mode enabled will allow the status of agentless integration
policies to be tracked.

### Key technical changes

- A new field `supports_agentless` was added to package policies. This
field already exists on agent policies. When an agentless integration is
created, `supports_agentless: true` is now added to both the package
policy and its parent agent policy.
- This allows easier filtering for agentless integrations as we avoid
having to retrieve & check against every parent agent policy.
- This also means existing agentless policies do not get this new status
tracking UI, only new ones created after this change. Since agentless is
not yet GA, I think this is okay.
- `/api/fleet/agent_status/data` now takes optional query params
`pkgName` and `pkgVersion`. When both are specified, the API will check
if agent(s) have ingested data for only that package's datastreams.

## UI walkthrough
<details>
<summary>🖼️ Click to show screenshots</summary>

1. **Integration policies** page now shows two tables for integrations
meeting the above condition, one for agentless policies and one for
agent-based policies:

![image](https://github.com/user-attachments/assets/58c6a932-9bda-4229-ba5f-d341bdbd539a)

2. Clicking the status badge in the agentless policies table opens a
flyout with two steps: confirm agentless enrollment and confirm incoming
data:

![image](https://github.com/user-attachments/assets/e19e6ba0-f40d-48a7-a524-0373934ac46a)

3. Confirm agentless enrollment polls for an agent enrolled into that
integration policy's agent policy. If that agent is reporting an
unhealthy status, the integration component UI is shown. This UI is the
same one used on Fleet > Agents > Agent details page and shows all
components reported by that agent:

![image](https://github.com/user-attachments/assets/ce214f7f-4bdd-48e5-a5eb-a1e8fcc7a512)

4. Once a healthy agentless enrollment is established, confirm incoming
data starts polling for data for that integration ingested by that agent
ID in the past 5 minutes:

![image](https://github.com/user-attachments/assets/7f3de40b-3418-4174-b529-e805407949b6)

5. If data could not be retrieved in 5 minutes, an error message shows
while polling continues in the background:

![image](https://github.com/user-attachments/assets/a3fd198e-1570-4357-9b7f-e541a769d33f)

6. If data is retrieved, a success message is shown:

![image](https://github.com/user-attachments/assets/f4e442af-ca60-4448-9bfb-3f244cd03c2d)
</details>

## Testing
Easiest way to test is use the Cloud deployment from this PR. Enable
Beta integrations and navigate to CSPM. Add a CSPM integration using
`Agentless` setup technology. Then you can track the status of the
agentless deployment on the Integrations policies tab.

For local testing, the following is required to simulate agentless
agent:
1. Add the following to kibana.dev.yml:
```
xpack.cloud.id: 'anything-to-pass-cloud-validation-checks'
xpack.fleet.agentless.enabled: true
xpack.fleet.agentless.api.url: 'https://localhost:8443'
xpack.fleet.agentless.api.tls.certificate: './config/certs/ess-client.crt'
xpack.fleet.agentless.api.tls.key: './config/certs/ess-client.key'
xpack.fleet.agentless.api.tls.ca: './config/certs/ca.crt'
```
2. Apply [this
patch](https://gist.github.com/jen-huang/dfc3e02ceb63976ad54bd1f50c524cb4)
to prevent attempt to create agentless pod
3. Enroll a Fleet Server as usual
4. Enable Beta integrations and navigate to CSPM. Add a CSPM integration
using `Agentless` setup technology.
5. Enroll a normal Elastic Agent to the agent policy for that CSPM
integration by using the token from Enrollment tokens

## To-do
- [x] API tests
- [x] Unit UI tests
- [x] Manual Cloud tests
- [x] File docs request
  - elastic/ingest-docs#1466
- [ ] Update troubleshooting guide link once available

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 3188cda)

# Conflicts:
#	oas_docs/bundle.json
#	oas_docs/bundle.serverless.json
#	oas_docs/output/kibana.serverless.yaml
#	oas_docs/output/kibana.yaml
#	src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts
#	x-pack/plugins/fleet/public/components/package_policy_actions_menu.test.tsx
#	x-pack/plugins/fleet/public/components/package_policy_actions_menu.tsx
#	x-pack/plugins/fleet/server/routes/agent/handlers.ts
#	x-pack/plugins/fleet/server/types/models/package_policy.ts

packages/kbn-check-mappings-update-cli/current_fields.json

@@ -533,6 +533,7 @@
     "revision",
     "secret_references",
     "secret_references.id",
+    "supports_agentless",
     "updated_at",
     "updated_by",
     "vars"
@@ -715,6 +716,7 @@
     "revision",
     "secret_references",
     "secret_references.id",
+    "supports_agentless",
     "updated_at",
     "updated_by",
     "vars"

packages/kbn-check-mappings-update-cli/current_mappings.json

@@ -1786,6 +1786,9 @@
           }
         }
       },
+      "supports_agentless": {
+        "type": "boolean"
+      },
       "updated_at": {
         "type": "date"
       },
@@ -2374,6 +2377,9 @@
           }
         }
       },
+      "supports_agentless": {
+        "type": "boolean"
+      },
       "updated_at": {
         "type": "date"
       },

src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts

@@ -108,7 +108,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
         "fleet-agent-policies": "f57d3b70e4175a19a18f18ee72a379ceec82e1fc",
         "fleet-fleet-server-host": "69be15f6b6f2a2875ad3c7050ddea7a87f505417",
         "fleet-message-signing-keys": "93421f43fed2526b59092a4e3c65d64bc2266c0f",
-        "fleet-package-policies": "8be2cabfed89e103e0d413f2900e9cf6cd31bc68",
+        "fleet-package-policies": "0206c20f27286787b91814a2e7872f06dc1e8e47",
         "fleet-preconfiguration-deletion-record": "c52ea1e13c919afe8a5e8e3adbb7080980ecc08e",
         "fleet-proxy": "6cb688f0d2dd856400c1dbc998b28704ff70363d",
         "fleet-setup-lock": "0dc784792c79b5af5a6e6b5dcac06b0dbaa90bde",
@@ -123,8 +123,8 @@ describe('checking migration metadata changes on all registered SO types', () =>
         "infrastructure-ui-source": "113182d6895764378dfe7fa9fa027244f3a457c4",
         "ingest-agent-policies": "5e95e539826a40ad08fd0c1d161da0a4d86ffc6d",
         "ingest-download-sources": "279a68147e62e4d8858c09ad1cf03bd5551ce58d",
-        "ingest-outputs": "daafff49255ab700e07491376fe89f04fc998b91",
-        "ingest-package-policies": "dfa7b1045a2667a822181f40f012786724492439",
+        "ingest-outputs": "55988d5f778bbe0e76caa7e6468707a0a056bdd8",
+        "ingest-package-policies": "60d43f475f91417d14d9df05476acf2e63e99435",
         "ingest_manager_settings": "111a616eb72627c002029c19feb9e6c439a10505",
         "inventory-view": "b8683c8e352a286b4aca1ab21003115a4800af83",
         "kql-telemetry": "93c1d16c1a0dfca9c8842062cf5ef8f62ae401ad",

x-pack/plugins/fleet/common/constants/mappings.ts

@@ -72,6 +72,7 @@ export const PACKAGE_POLICIES_MAPPINGS = {
       properties: {},
     },
     secret_references: { properties: { id: { type: 'keyword' } } },
+    supports_agentless: { type: 'boolean' },
     revision: { type: 'integer' },
     updated_at: { type: 'date' },
     updated_by: { type: 'keyword' },

x-pack/plugins/fleet/common/services/simplified_package_policy_helper.ts

@@ -49,6 +49,7 @@ export interface SimplifiedPackagePolicy {
   description?: string;
   vars?: SimplifiedVars;
   inputs?: SimplifiedInputs;
+  supports_agentless?: boolean | null;
 }
 
 export interface FormattedPackagePolicy extends Omit<PackagePolicy, 'inputs' | 'vars'> {
@@ -154,18 +155,19 @@ export function simplifiedPackagePolicytoNewPackagePolicy(
     description,
     inputs = {},
     vars: packageLevelVars,
+    supports_agentless: supportsAgentless,
   } = data;
-  const packagePolicy = packageToPackagePolicy(
-    packageInfo,
-    policyId && isEmpty(policyIds) ? policyId : policyIds,
-    namespace,
-    name,
-    description
-  );
-
-  if (outputId) {
-    packagePolicy.output_id = outputId;
-  }
+  const packagePolicy = {
+    ...packageToPackagePolicy(
+      packageInfo,
+      policyId && isEmpty(policyIds) ? policyId : policyIds,
+      namespace,
+      name,
+      description
+    ),
+    supports_agentless: supportsAgentless,
+    output_id: outputId,
+  };
 
   if (packagePolicy.package && options?.experimental_data_stream_features) {
     packagePolicy.package.experimental_data_stream_features =

x-pack/plugins/fleet/common/types/models/package_policy.ts

@@ -92,6 +92,7 @@ export interface NewPackagePolicy {
     };
   };
   overrides?: { inputs?: { [key: string]: any } } | null;
+  supports_agentless?: boolean | null;
 }
 
 export interface UpdatePackagePolicy extends NewPackagePolicy {

x-pack/plugins/fleet/common/types/rest_spec/agent.ts

@@ -234,6 +234,8 @@ export interface GetAgentStatusResponse {
 export interface GetAgentIncomingDataRequest {
   query: {
     agentsIds: string[];
+    pkgName?: string;
+    pkgVersion?: string;
     previewData?: boolean;
   };
 }

x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/multi_page_layout/components/confirm_incoming_data_with_preview.tsx

@@ -131,11 +131,11 @@ export const ConfirmIncomingDataWithPreview: React.FunctionComponent<Props> = ({
   setAgentDataConfirmed,
   troubleshootLink,
 }) => {
-  const { incomingData, dataPreview, isLoading, hasReachedTimeout } = usePollingIncomingData(
+  const { incomingData, dataPreview, isLoading, hasReachedTimeout } = usePollingIncomingData({
     agentIds,
-    true,
-    MAX_AGENT_DATA_PREVIEW_COUNT
-  );
+    previewData: true,
+    stopPollingAfterPreviewLength: MAX_AGENT_DATA_PREVIEW_COUNT,
+  });
   const { enrolledAgents, numAgentsWithData } = useGetAgentIncomingData(incomingData, packageInfo);
 
   const isGuidedOnboardingActive = useIsGuidedOnboardingActive(packageInfo?.name);

x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/single_page_layout/hooks/setup_technology.test.ts

@@ -115,6 +115,7 @@ describe('useAgentless', () => {
 describe('useSetupTechnology', () => {
   const setNewAgentPolicy = jest.fn();
   const updateAgentPoliciesMock = jest.fn();
+  const updatePackagePolicyMock = jest.fn();
   const setSelectedPolicyTabMock = jest.fn();
   const newAgentPolicyMock = {
     name: 'mock_new_agent_policy',
@@ -183,6 +184,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -212,6 +214,7 @@ describe('useSetupTechnology', () => {
         packagePolicy: packagePolicyMock,
         isEditPage: true,
         agentPolicies: [{ id: 'agentless-policy-id', supports_agentless: true } as any],
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -239,6 +242,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -248,6 +252,7 @@ describe('useSetupTechnology', () => {
       result.current.handleSetupTechnologyChange(SetupTechnology.AGENTLESS);
     });
     await waitFor(() => {
+      expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: true });
       expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENTLESS);
       expect(setNewAgentPolicy).toHaveBeenCalledWith({
         name: 'Agentless policy for endpoint-1',
@@ -278,6 +283,7 @@ describe('useSetupTechnology', () => {
       updateAgentPolicies: updateAgentPoliciesMock,
       setSelectedPolicyTab: setSelectedPolicyTabMock,
       packagePolicy: packagePolicyMock,
+      updatePackagePolicy: updatePackagePolicyMock,
     };
 
     const { result, rerender } = renderHook((props = initialProps) => useSetupTechnology(props), {
@@ -291,6 +297,7 @@ describe('useSetupTechnology', () => {
     });
 
     expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENTLESS);
+    expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: true });
     expect(setNewAgentPolicy).toHaveBeenCalledWith({
       inactivity_timeout: 3600,
       name: 'Agentless policy for endpoint-1',
@@ -306,9 +313,11 @@ describe('useSetupTechnology', () => {
         ...packagePolicyMock,
         name: 'endpoint-2',
       },
+      updatePackagePolicy: updatePackagePolicyMock,
     });
 
     await waitFor(() => {
+      expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENTLESS);
       expect(setNewAgentPolicy).toHaveBeenCalledWith({
         name: 'Agentless policy for endpoint-2',
         inactivity_timeout: 3600,
@@ -332,6 +341,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -366,6 +376,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -376,12 +387,14 @@ describe('useSetupTechnology', () => {
     });
 
     expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENTLESS);
+    expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: true });
 
     act(() => {
       result.current.handleSetupTechnologyChange(SetupTechnology.AGENT_BASED);
     });
 
     expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENT_BASED);
+    expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: false });
 
     await waitFor(() => {
       expect(setNewAgentPolicy).toHaveBeenCalledWith(newAgentPolicyMock);
@@ -397,6 +410,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -410,6 +424,7 @@ describe('useSetupTechnology', () => {
 
     expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENT_BASED);
 
+    expect(updatePackagePolicyMock).not.toHaveBeenCalled();
     expect(setNewAgentPolicy).not.toHaveBeenCalled();
     expect(setSelectedPolicyTabMock).not.toHaveBeenCalled();
   });
@@ -435,6 +450,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -454,6 +470,7 @@ describe('useSetupTechnology', () => {
         supports_agentless: true,
         inactivity_timeout: 3600,
       });
+      expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: true });
     });
 
     act(() => {
@@ -463,6 +480,7 @@ describe('useSetupTechnology', () => {
     await waitFor(() => {
       expect(result.current.selectedSetupTechnology).toBe(SetupTechnology.AGENT_BASED);
       expect(setNewAgentPolicy).toHaveBeenCalledWith(newAgentPolicyMock);
+      expect(updatePackagePolicyMock).toHaveBeenCalledWith({ supports_agentless: false });
     });
   });
 
@@ -489,6 +507,7 @@ describe('useSetupTechnology', () => {
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
         packageInfo: packageInfoMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -533,6 +552,7 @@ describe('useSetupTechnology', () => {
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
         packageInfo: packageInfoMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -582,6 +602,7 @@ describe('useSetupTechnology', () => {
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
         packageInfo: packageInfoMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -627,6 +648,7 @@ describe('useSetupTechnology', () => {
         updateAgentPolicies: updateAgentPoliciesMock,
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );
 
@@ -673,6 +695,7 @@ describe('useSetupTechnology', () => {
         setSelectedPolicyTab: setSelectedPolicyTabMock,
         packagePolicy: packagePolicyMock,
         packageInfo: packageInfoMock,
+        updatePackagePolicy: updatePackagePolicyMock,
       })
     );