There are plenty of texts talking about expectations of engineers and best practices during an incident. What’s not mentioned as often, mainly because giving feedback “up” is hard, is expectations of senior management. In a recent discussion at work, I naively proposed the following 3 expectations / guidelines for senior managers and above.
Power Dynamics Link to heading
The voluntary involvement of senior management during an active incident is discouraged.
This is due to how easily power-dynamics add rpessure and confusion in regards to authority when dealing with an already stressful situation.
Performance Evaluations Link to heading
Being a leading actor in resolving an important production incident should be no more rewarded than quietly preventing one, and senior management should be conscious of this when doing performance evaluations.
This point applies to organizations large enough to have systems and processes around internal career progression.
Adoption Non-Reaction Link to heading
If, as a result of a team adopting best practices around incident management increases visibility around its incidents, then senior management should not react with any more urgency than prior to adoption, keeping the level of trust, oversight and attention as it was previously.
This is to not revert the adoption for said team, and to not dissuade other teams from adopting incident handling best practices.
Go on… Link to heading
I’m lucky enough to work with several engineering leaders I consider far senior to myself.
A VP pointed out that the temptation for senior management to get closer to the action is fuelled by inadequate upwards communication. Empathy goes both ways.
A principal engineer also shared an idea for certain Directors/VPs to be trained as “Incident Executives”. These are separate from “Incident Commander” (typically an individual contributor). The role of an Incident Executive is to be present on calls and communicate with other execs as well as provide shielding for the incident response team.