Cooks and Chaos: Turning a Self-Inflicted P1 into a Culture Win

[Derek32]

With a customer events fresh from last week, the timing is relevant to share how P1s can be learning opportunities. It’s the dreaded P1 situation with war rooms, hotlines, and heated sentiment across the org. To make matters worse, it was a self-inflicted situation. The cause? A developer with admin access in production, making a system properties directly in PROD without following the standard process. Why the change and why directly in production? No one knew.

As a leader, the gut reaction is to add more policies, procedures, or technical controls to crack down on governance. While certainly understood and some of it warranted, simply adding more controls and processes does not address the human behavior that can bypass most of that if desired.

This is an excellent opportunity to address culture and coaching. Lead with genuine curiosity to get to the “Why.” Where did we fail or cause friction where making the update directly in PROD was the decision the developer made? Was it fear of forgetting to include a story in the release? Was testing taking too long? As a leader, creating a sense of vulnerability within the team will allow them the ability to openly share where we are failing. By addressing the root cause versus punishing the person, we create champions that understand why we do something and what the right thing to do is.

The other area of improvement was the complete lack of a unified process. There were leaders across the org pinging one another, the platform team, developers, architects, and anyone else who would listen. This created a tremendous amount of noise and prevented those with the expertise needed to remediate the room and ability to do so. In these mission critical moments, platform teams need to have guardrails and a defined process that is communicated out to the org. It’s the “too many cooks in the kitchen” message. If you are not actively engaged to fix the issue OR you are not communicating the fix, you are simply adding the problem. Stand down.

The benefit? Its situations like this that allow us to create a culture built on vulnerability and progress.