#2 Seatbelts, Sidewalk Salt, and Emergency Plans

What figure skaters and goats have in common

Nov 13, 2022

I thought I was going to write about performance reviews this week, but it’s been one of those weeks at work where everything feels scattered. The performance review piece wasn’t really speaking to me, but some other things definitely were. Performance reviews can wait, and so can the follow-up on prioritization, which will dig more into “what if other people are asking me to give them a roadmap?”. This week, I want to tell you a story.

It’s a hell of a lot more difficult and time-consuming to repair things than it is to break them in the first place.

When I was in college, I was on the executive board of a student organization. The outgoing board had gotten the organization into a bit of trouble: sanctions from the university, loss of trust within the organization, mismanagement of budget, the works. Nothing catastrophic, but they had clearly made a mess. Near the beginning of our term, the new president of the board got us all in a room and said “This is a building year. We’ve got some things to turn around, some wrongs to make right, and it’s probably going to feel unsatisfying. I know that we could have had more of an impact if the previous executive board hadn’t left things the way they did. But this is what we have to work with and it’s on us to make the best of it.”

I think about that experience often, because the lessons are so applicable. It’s hard to fix things. It’s frustrating to be able to see things as they could be if they hadn’t broken in the first place, and struggle through the act of repairing them. It can be demoralizing to work so hard on a problem, only to get things back to where they were before they got broken. Sometimes, you’re not able to get the thing back to the exact way it was before, and you have to mourn that fact alongside the frustrating work of rebuilding.

This is very, very tangible:

it’s harder to repair a bowl than it is to smash it (though, the repair can be quite beautiful)
it’s more difficult to reupholster a couch than it is to spill red wine all over it
it’s harder to put a jigsaw puzzle together than it is to break it apart
restoring production data is much more laborious than accidentally destroying it (which is why there are often protections against destroying data, and backups, and it’s why we test our code)
$8 sure was able to wipe out $15B sooner than that $15B is coming back
once you lose your team’s trust, you have to do a whole lot to gain it back. Similarly, once you lay most of your team off and then need them back, you don’t get to call them “weak, lazy, unmotivated”, Luke!
however much damage Twitter’s company culture and product have sustained from October 27th until today, it will take far, far longer to recover (if they do at all)

So, what are we to do? Breaking things and fixing them seems like a lot of work, can’t we just not break them?

There are at least two evergreen options:

#1 Reduce the risk of breaking things at all

This is striving towards risk elimination/avoidance. This is why we write tests for our code, why we go through code review, and why we try to avoid directly manipulating production data. It’s why we don’t leave candles unattended, or the oven on when we leave the house. It’s why we don’t ever stop a car on train tracks, why we salt sidewalks in the winter, and why we measure twice but cut once. It’s why we have multi-factor authentication.

#2 Break things as little as possible

This is risk mitigation. It’s why we wear helmets, and seatbelts, and have airbags. It’s why we teach figure skaters to fall safely, and why we graze goats as part of fire management. It’s why we have earthquake kits, and family emergency plans. It’s why we rarely give one person full, unilateral access to all production systems.

Besides risk mitigation, there’s also an aspect of this that boils down to “don’t do more harm than you need to”. If you shatter a plate, don’t throw the rest of the set on the ground. If you spill wine, don’t also pour out the rest of the bottle on the carpet. If you lay people off, a good risk mitigation strategy might be to not badmouth them in writing when you realize you need them back.

Bad things do happen. But we can reduce the likelihood that they will, the impact when they do, or both.

We’re past all that. We broke things anyway. We need to rebuild. How do we do that?

Remember at the beginning of this story, when I mentioned that this would be very hard and very slow?

First, accept that you’re rebuilding. You can’t recover from something that you don’t acknowledge. Get everybody aligned on the fact that you’re rebuilding. State your goals as clearly as you can, at every step of the process. A good first goal: triage the problem. Figure out how bad it is, whether it’s still getting worse, and who needs to be involved in working together to fix it.

Don’t make it worse. Once you triage the problem, don’t cause more damage by flailing around while looking for solutions. Move slowly, lead with curiosity, take responsibility for the cleanup (and the problem, if you had a hand in causing it). Know what actions are reversible, and which aren’t. Don’t take irreversible actions unless you’re sure about the after effects. Don’t lie. Don’t promise things you can't deliver.

Figure out where to start. In theory, there might be a best place to start, but you might not be able to see it right away, or ever. Start with something that you think is no-regret work. It’s almost never a bad idea to talk to people affected by and around the problem, to offer support and to begin gathering ideas for next steps. Helpful questions can include “could we have done anything to prevent this?”, “if something like this happens again, what are the earliest ways that we’ll hear?”, and “what tools do we wish we had right now to start fixing these problems?”.

Have patience, with yourself and with the process. Rebuilding is slow (and sometimes annoying) work. The fix won’t happen as quickly as the problem got created. Don’t expect it to, and don’t get discouraged. Pace yourself. If you can, try to identify some signals that will tell you if you’re on the right track, so you can celebrate gradual wins.

Work consistently and steadily. Be dependable throughout a chaotic process. When things go wrong, it’s common that there’s a degradation in trust somewhere. One good way to begin to rebuild trust is through transparency: if you’re clear about what you’re doing, what still needs more work, and what’s puzzling you, you’re more likely to get people on board with your plan.

Accept your limits. You might never be done with cleanup in the way you want to be done. There may be something (data, trust, money) that is lost forever, and you can’t bring it back no matter what you do. If you have a perfectionist streak, this will be difficult, so have some compassion for yourself. Do what you can, do what makes sense to do, but don’t lose sight of the forest for the trees. Don’t be so focused on fixing this one problem that you lose perspective on everything else.

The Second 90%

Discussion about this post