Add kibana_system_password and elasticsearch_keystore_entries

docs/roles/elasticsearch.md

@@ -382,6 +382,22 @@ elasticsearch_logging_audit: true
 
     JSON logs use `ECSJsonLayout` with `dataset` fields (Elastic Common Schema). Deprecation logs add a `RateLimitingFilter` to prevent log flooding and a `HeaderWarningAppender` for HTTP response warnings. Indexing slow log logger name changed to `index.indexing.slowlog.index`.
 
+### Custom Keystore Entries
+
+```yaml
+elasticsearch_keystore_entries:
+  xpack.notification.slack.account.monitoring.secure_url: "https://hooks.slack.com/services/T00/B00/XXX"
+  xpack.notification.email.account.work.smtp.secure_password: "smtp-password"
+```
+
+`elasticsearch_keystore_entries` is a dictionary of custom entries to add to the Elasticsearch keystore. Each key-value pair is set using `elasticsearch-keystore add -f -x`, with the value passed via stdin so it never appears in process listings or Ansible logs.
+
+Use this for any sensitive Elasticsearch setting that belongs in the keystore rather than in `elasticsearch.yml` — Watcher notification credentials, repository passwords, LDAP bind passwords, custom plugin secrets, etc.
+
+The role manages a fixed set of keystore keys internally (SSL keystore/truststore passwords, bootstrap password). If you try to set any of these via `elasticsearch_keystore_entries`, the playbook fails immediately with an error listing exactly which keys are reserved and why. The reserved keys are: `bootstrap.password`, `autoconfiguration.password_hash`, and the six `xpack.security.*.ssl.*` password entries. Use the dedicated role variables for those instead.
+
+On each run, the role reads the current value of each custom entry and only writes it if the value has changed, so the keystore is not unnecessarily modified and Elasticsearch is only restarted when an entry actually changes.
+
 ### Extra Configuration
 
 ```yaml

docs/roles/kibana.md

@@ -51,6 +51,7 @@ kibana_config_backup: false
 ```yaml
 # kibana_elasticsearch_hosts: (undefined by default)
 kibana_security: true
+kibana_system_password: ""
 kibana_sniff_on_start: false
 kibana_sniff_on_connection_fault: false
 ```
@@ -66,6 +67,8 @@ kibana_sniff_on_connection_fault: false
 
 `kibana_security` enables authenticated, encrypted connections to Elasticsearch. When `true`, Kibana connects over HTTPS using the `kibana_system` user and the password from the Elasticsearch security setup. The CA certificate is deployed automatically from the ES CA host.
 
+`kibana_system_password` lets you set a specific password for the `kibana_system` Elasticsearch user. When empty (the default), Kibana uses the auto-generated password from the initial security setup. When set, the role changes the password via the Elasticsearch `/_security/user/kibana_system/_password` API on every run and uses the new value for Kibana's connection to Elasticsearch. This is useful when you need a known password for external monitoring, when you rotate credentials on a schedule, or when multiple Kibana instances need a consistent password that isn't tied to the initial setup file.
+
 `kibana_sniff_on_start` and `kibana_sniff_on_connection_fault` control Elasticsearch node discovery. When enabled, Kibana queries the ES cluster for the full list of nodes at startup or when a connection drops. These settings only apply to Elastic Stack versions prior to 9.x (Kibana 9.x removed sniffing support).
 
 ### TLS for the Kibana Web Interface

roles/elasticsearch/defaults/main.yml

@@ -171,6 +171,29 @@ elasticsearch_validate_api_certs: false
 
 elasticsearch_unsafe_upgrade_restart: false
 
+# @var elasticsearch_keystore_entries:description: >
+#   Custom entries to add to the Elasticsearch keystore. Each key-value pair
+#   is added with `elasticsearch-keystore add -f -x`. Values are passed via
+#   stdin so they never appear in process listings or Ansible logs.
+#
+#   The role manages these keystore keys internally — do NOT set them here:
+#     bootstrap.password,
+#     xpack.security.http.ssl.keystore.secure_password,
+#     xpack.security.http.ssl.truststore.secure_password,
+#     xpack.security.transport.ssl.keystore.secure_password,
+#     xpack.security.transport.ssl.truststore.secure_password,
+#     xpack.security.http.ssl.secure_key_passphrase,
+#     xpack.security.transport.ssl.secure_key_passphrase,
+#     autoconfiguration.password_hash
+#   The playbook will fail with an error if any of these keys appear in
+#   elasticsearch_keystore_entries.
+# @var elasticsearch_keystore_entries:example: >
+# elasticsearch_keystore_entries:
+#   xpack.notification.slack.account.monitoring.secure_url: "https://hooks.slack.com/services/T00/B00/XXX"
+#   xpack.notification.email.account.work.smtp.secure_password: "smtp-password"
+# @end
+elasticsearch_keystore_entries: {}
+
 # @var elasticsearch_extra_config:description: >
 #   Additional key-value pairs merged into elasticsearch.yml. Keys that
 #   conflict with dedicated role variables (e.g. cluster.name) are

roles/elasticsearch/tasks/elasticsearch-keystore.yml

@@ -306,6 +306,72 @@
   notify:
     - Restart Elasticsearch
 
+# ============================================================
+# Custom keystore entries (elasticsearch_keystore_entries)
+# ============================================================
+#
+# These tasks run AFTER the built-in keystore management above.
+# The role owns a fixed set of keystore keys (SSL passwords,
+# bootstrap password, etc.) — those are off-limits. User-provided
+# entries in elasticsearch_keystore_entries must not overlap with
+# them or the playbook fails with a clear error explaining why.
+
+- name: elasticsearch-keystore | Validate custom entries do not conflict with role-managed keys
+  ansible.builtin.assert:
+    that:
+      - item.key not in _elasticsearch_reserved_keystore_keys
+    fail_msg: >-
+      Cannot set '{{ item.key }}' via elasticsearch_keystore_entries because
+      this key is managed automatically by the elasticsearch role. Remove it
+      from elasticsearch_keystore_entries. The role manages these keys
+      internally: {{ _elasticsearch_reserved_keystore_keys | join(', ') }}
+  loop: "{{ elasticsearch_keystore_entries | dict2items }}"
+  loop_control:
+    label: "{{ item.key }}"
+  vars:
+    _elasticsearch_reserved_keystore_keys:
+      - bootstrap.password
+      - autoconfiguration.password_hash
+      - xpack.security.http.ssl.keystore.secure_password
+      - xpack.security.http.ssl.truststore.secure_password
+      - xpack.security.transport.ssl.keystore.secure_password
+      - xpack.security.transport.ssl.truststore.secure_password
+      - xpack.security.http.ssl.secure_key_passphrase
+      - xpack.security.transport.ssl.secure_key_passphrase
+  when: elasticsearch_keystore_entries | length > 0
+
+- name: elasticsearch-keystore | Get current value for custom entry '{{ item.key }}'
+  ansible.builtin.command: >
+    /usr/share/elasticsearch/bin/elasticsearch-keystore
+    show {{ item.key }}
+  loop: "{{ elasticsearch_keystore_entries | dict2items }}"
+  loop_control:
+    label: "{{ item.key }}"
+  register: _elasticsearch_custom_keystore_current
+  changed_when: false
+  no_log: true
+  ignore_errors: true
+  when: elasticsearch_keystore_entries | length > 0
+
+- name: elasticsearch-keystore | Set custom entry '{{ item.item.key }}'
+  ansible.builtin.command: >
+    /usr/share/elasticsearch/bin/elasticsearch-keystore
+    add -f -x '{{ item.item.key }}'
+  args:
+    stdin: "{{ item.item.value }}"
+  loop: "{{ _elasticsearch_custom_keystore_current.results | default([]) }}"
+  loop_control:
+    label: "{{ item.item.key }}"
+  changed_when: true
+  no_log: true
+  when:
+    - elasticsearch_keystore_entries | length > 0
+    - item.rc != 0 or item.stdout != item.item.value | string
+  notify:
+    - Restart Elasticsearch
+
+# ============================================================
+
 - name: elasticsearch-keystore | Check keystore exists
   ansible.builtin.stat:
     path: /etc/elasticsearch/elasticsearch.keystore

roles/kibana/defaults/main.yml

@@ -21,6 +21,14 @@ kibana_config_backup: false
 # @var kibana_manage_yaml:description: Let the role manage kibana.yml. Set to false to manage it yourself
 kibana_manage_yaml: true
 
+# @var kibana_system_password:description: >
+#   User-defined password for the kibana_system user. When set, the role
+#   changes the auto-generated password via the Elasticsearch security API
+#   and uses this value for Kibana's connection to Elasticsearch. Leave
+#   empty to keep the auto-generated password from initial_passwords.
+# @end
+kibana_system_password: ""
+
 # @var kibana_security:description: Enable security features (connect to Elasticsearch over HTTPS with authentication)
 kibana_security: true
 # @var kibana_tls:description: Enable TLS on the Kibana web interface itself (serve Kibana over HTTPS)

roles/kibana/tasks/kibana-security.yml

@@ -416,6 +416,48 @@
     _password_user: kibana_system
     _password_fact: kibana_password
 
+# -- Change kibana_system password if user defined one --
+#
+# When kibana_system_password is set, the role changes the auto-generated
+# password to the user-provided value via the Elasticsearch security API.
+# This runs once on the CA host, authenticating as the elastic superuser.
+# After the API call, kibana_password is updated so the rest of the role
+# (kibana.yml template, keystore) uses the new password transparently.
+- name: kibana-security | Change kibana_system password to user-defined value
+  when:
+    - inventory_hostname == elasticstack_ca_host
+    - kibana_system_password | default('') | length > 0
+    - kibana_password.stdout | default('') != kibana_system_password
+  block:
+    - name: kibana-security | Fetch elastic password for API authentication
+      ansible.builtin.include_tasks:
+        file: "{{ role_path }}/../elasticstack/tasks/fetch_password.yml"
+      vars:
+        _password_user: elastic
+        _password_fact: _kibana_elastic_password
+
+    - name: kibana-security | Change kibana_system password via Elasticsearch API
+      ansible.builtin.uri:
+        url: "{{ _kibana_es_protocol }}://{{ elasticsearch_api_host | default('localhost') }}:{{ elasticstack_elasticsearch_http_port }}/_security/user/kibana_system/_password"
+        method: POST
+        body_format: json
+        body:
+          password: "{{ kibana_system_password }}"
+        user: elastic
+        password: "{{ elasticstack_password.stdout | default(_kibana_elastic_password.stdout) }}"
+        force_basic_auth: true
+        validate_certs: false
+        status_code: 200
+      no_log: "{{ elasticstack_no_log }}"
+      vars:
+        _kibana_es_protocol: "{{ 'https' if kibana_security | bool else 'http' }}"
+
+    - name: kibana-security | Update kibana_password fact to user-defined value  # noqa: var-naming[no-role-prefix]
+      ansible.builtin.set_fact:
+        kibana_password:
+          stdout: "{{ kibana_system_password }}"
+      no_log: "{{ elasticstack_no_log }}"
+
 # -- Distribute CA certificate to Kibana nodes --
 - name: kibana-security | Distribute CA certificate to Kibana nodes
   ansible.builtin.include_tasks: