Álex Ruiz Becerra 93ef5f1f67
Add tooling and index definitions for Wazuh decoders (#581)
* Add initial WCS tooling for integrations and index definitions

* Add changelog and index creation

* Run mappings generator on pull request sync

* Fix mappings generator

* Bump versions

- ECS: v8.11.0 -> v9.1.0
- Python image: 3.10 -> 3.13-slim

* Apply rollover policy to wazuh-events indices

* Use short descriptions

* Add support for ECS v9.1.0

* Remove on_push trigger for the mappings generator Workflow

* Update generate_and_push_templates.sh to commit new (untracked) files

* Fix glob pattern

* Set checkout branch in mappings generator workflow

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Extend mappings limits

Remove synthetic_source_keep from the index templates

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Fix typo reading CSV column name

Was using Wazuh Type instead of 'Wazuh type', causing all custom fields to use the default data type 'keyword'

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Fix fortinet.yml and add the new indices to the setup plugin

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Update azure fields csv to trigger index creation

* Add azure template

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Create test tool

* Add root: true to ignore the integration name as part of the field names

* Add concurrency limit to the workflow to generate index templates

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Re-run with updated CSV

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Keep base fields for every stateless indices

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Upgrade test tool

* Delete extra field from f5 fields

* Update ECS templates for modified modules: stateless-amazon-security-lake stateless-apache_tomcat stateless-audit stateless-azure-app-service stateless-azure-metrics stateless-checkpoint stateless-cisco-asa stateless-cisco_umbrella stateless-f5-bigip stateless-fortinet stateless-gcp stateless-iis stateless-iptables stateless-microsoft-dhcp stateless-microsoft-dnsserver stateless-microsoft-exchange-server stateless-modsec stateless-oracle_weblogic stateless-pfsense stateless-snort stateless-spring_boot stateless-squid stateless-suricata stateless-template stateless-unifiedlogs stateless-websphere stateless-windows stateless-zeek

* Add missing iss index creation

* Add more integrations

Replace underscore with dash on integrations names

* Fix flaky integration tests

Replace custom code to check cluster is green with the OpenSearchIntegTestCase::ensureGreen() method, which fits this purpose

* Add more modules and enhance tooling

* Add back path output on navigate_to_project_root function

* Add ecs/scripts/module_list.txt to the files to commit

* Update the Wazuh Common Schema

* Re-structure tooling

* Fix links and filter of modified files

* Improve detecting of modified files to commit and push

* Add creation of more indices

Rename modsec to modsecurity

* Update the Wazuh Common Schema

* Fix Cisco IOS and AWS templates

* Increase limit of nested fields for the AWS indices

* Update the Wazuh Common Schema

* Add suggestions from code review

---------

Co-authored-by: Wazuh Indexer Bot <github_devel_xdrsiem_indexer@wazuh.com>
Co-authored-by: Jorge Sanchez <jorge.sanchez@wazuh.com>
2025-10-14 16:39:11 +02:00

4 lines
69 B
Plaintext

**/mappings
*.log
generatedData.json
**/wcs-test-tool.log.summary.csv