Elastic Platform Governance at Scale
Built distributed log management across the Elastic Stack, balancing source owner autonomy with security team control over the platform's CIA
The Problem
- Uncoordinated log ingestion made it hard and tedious for the security team to manage the SIEM and engineer their security rules
- Individual teams needed access to application and platform logs & dashboards but had no self-service capability
- Decentralized control risked the confidentiality of the platform, and the integrity and availability of the logs
The Solution
- Transitioned to managing the Elastic infrastructure as code
- Developed standardized CI/CD validation checks accelerating log ingestion while ensuring data quality
- Enforced access controls and resource limits over the ingested logs and dashboards, while preserving team autonomy
The Impact
- Teams self-service their logs and dashboards, without ticketing delays
- Reduced platform outages; enforced the confidentiality and integrity of the logs through centralized access control
- Storage costs decreased with standardized log ingestions and lifecycle policies
- Log and event quality improved through ECS-enforced parsing consistency
- Security team standardized their detection rule development and deployed across the organization
