Use the logs.ecs index instead of logs

[rdner]

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
  - [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

1. Built this altered integration package with elastic-package build
2. Then run the registry with elastic-package stack up --version 9.4.0-SNAPSHOT
3. Create an agent policy with the altered integration (note 2.4.0 version):

4. Create a new Fleet output which is allowed to write to data streams:

5. Select this new output in the policy settings:

6. Install an agent locally (I used 9.3.2, does not have to be 9.4.0):

sudo ./elastic-agent install --develop

7. Enroll the installed agent with:

sudo elastic-development-agent enroll --url=https://fleet-server:8220 --enrollment-token=<token> --insecure

8. Check that the test logs got ingested into the logs.ecs index:

We can also inspect the computed filestream configuration in the agent diagnostics:

beat-rendered-config.yml

apm: {}
features:
    features:
        fqdn:
            enabled: false
inputs:
    - clean_inactive: -1
      file_identity:
        fingerprint: null
      id: filestream-filestream.filestream-546d6aa6-3510-4e7c-83fd-c9e8bf21155f
      index: logs.ecs
      paths:
        - /logs/*.log
      processors:
        - add_fields:
            fields:
                input_id: filestream-filestream-546d6aa6-3510-4e7c-83fd-c9e8bf21155f
            target: '@metadata'
        - add_fields:
            fields:
                dataset: filestream.filestream
                namespace: default
                type: logs
            target: data_stream
        - add_fields:
            fields:
                dataset: filestream.filestream
            target: event
        - add_fields:
            fields:
                stream_id: filestream-filestream.filestream-546d6aa6-3510-4e7c-83fd-c9e8bf21155f
            target: '@metadata'
        - add_fields:
            fields:
                id: 6335dc4d-c911-49cf-8ff7-9ff98f7ff161
                snapshot: false
                version: 9.3.2
            target: elastic_agent
        - add_fields:
            fields:
                id: 6335dc4d-c911-49cf-8ff7-9ff98f7ff161
            target: agent
      prospector:
        scanner:
            exclude_files:
                - \.gz$
            fingerprint:
                enabled: true
                length: 1024
                offset: 0
            recursive_glob: true
      type: filestream
outputs:
    elasticsearch:
        api_key: <REDACTED>
        bulk_max_size: 1600
        compression_level: 1
        hosts:
            - https://elasticsearch:9200
        idle_connection_timeout: 3s
        preset: balanced
        queue:
            mem:
                events: 3200
                flush:
                    min_events: 1600
                    timeout: 10s
        ssl:
            ca_trusted_fingerprint: <REDACTED>
            certificate: <REDACTED>
            certificate_authorities: []
        type: elasticsearch
        worker: 1

Related issues

• Closes [Filestream] Ship data to logs.ecs instead of logs #17664

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ddb0e16

cc @rdner

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)