How To Convince Your Boss And Make Them Say Yes To Chaos Engineering

Christophe Rochefolle

Director Operational Excellence at OUI.sncf

Last Updated: March 21, 2018

Categories: Chaos Engineering

This is a guest article by Christophe Rochefolle, Director of Operational Excellence at OUI.sncf, France. It is based on a presentation (in French) he gave during the second Meetup of the Paris Chaos Engineering Community.

You may start by telling them you will break everything in production, and that will be fun! We call this in French “coming with big clogs — Gros Sabots”. They will see you coming from far, but except for noises, there is few chances you will hit the target.

They will probably ask for Return of Investment (R.O.I.). By arranging number of incidents, outage by year, cost of outage, and some hypothesis to reduce them, you may easily build a proposition with 2–3 years R.O.I.

However, between a project with new revenue and one which will maybe avoid losses, if they have to choose, the first one will be probably chosen first.

The rational approach is not the best one for a subject like Chaos Engineering. You must review your influence strategies.

Step 1: Make it familiar

Biologist Henri Laborit wrote in 1976: “Faced to unknown experience, man has only three choices: fight, do nothing or flee “. This subject is quite new, with few visibility, it is important to allow your boss to discover the concept at his speed, to avoid instinctive rejection.

You may start by sharing on internal or external social network some interesting papers on the subject:

• What Is Chaos Engineering? by Kolton Andrus (Answered on Quora)
• 4 Chaos Experiments to Start With by Matt Jacobs (Gremlin community)
• Why We Need More Chaos—Chaos Engineering, That Is by Nora Jones (AWS re:Invent 2017)
• The rise of Chaos Engineering by Danny Chrichton (TechCrunch)

Depending on tools used by your boss, you just need to find a way to send them those articles. I was lucky to be followed on twitter by many members of our executive board, I never hesitate to use it to make them aware on some subjects I will later discuss with them, including some post from executive influencers, like the CEO of a major bank Société Général:

Our first goal here is only to make Chaos Engineering sound familiar before going further.

Step 2: Identify your players

To be able to adjust your story to the different directors of your executive board, you need to identify the kind of players they are, among other things using sociodynamics:

From book : L’élan sociodynamique, Jean-Christian Fauvet You will be able to focus by adapting your strategy and identifying the way they move and act. Using your energy on the right priorities to make them allies.

What are their major challenges? What they may win or lose by collaborate with you? What are their influences on other players? Take time to identify the key issues at stake with decision-takers, as well as the brakes and leverages in relation to your subject. Concerns and objections are only excuses for your opponents to test you, but may also be real blocking points. You will win them by studying and including their views, but not necessarily during formal meeting…

Based on this map, you will be able to adapt your strategy:

• Friends (Zealots and Golden Triangles)—These are a dream as they are Promoters. They go out of their way to support the project. They are dedicated to their task, quick to provide resources and aid in issue resolution. It is important to know these individuals and maintain them as allies. Ensure you understand what is critical to them to make the delivery.·**
• Fence-sitters (Passives)—They define 40–80% of the stakeholders. Despite ones excellent inter personal skills you may not be able to reach all your stakeholders. A key asset in winning the masses is using influencers who run your campaign and positively influence the passives. As passives are impressionable they are easily swayed based on the feel in your project and it’s important they moved in your preferred direction. Winning the passives increase the synergy with your project team and increase the chances of project success. Other ways of wining Passives over is ensuring, these individuals feel part and parcel of the project i.e. are consulted and informed in what is going on.
• Foes (Mutineers, Moaners and Opponents)—They are a handful. However they have the ability to cause great damage to the synergy within the project team. Opponents and Mutineers are against your project for various reasons and they take any opportunity to antagonize the project delivery. This handful of individuals may rally those on the fence to be antagonistic towards the project. Established who they are, their concerns and evaluate if they can be resolved e.g. do they want to be more involved. If not, contain their impact on the passives as they would sway against you. Changing the global dynamic can take many forms. Sometimes it will be enough to change the views of some player on the subject … Be attentive to the evolution of the positions rather than the “photography” of these at a T time to mobilize the players who influence the dynamics towards the synergy.

Step 3: Tell the right story for your players

Once you have identified your players, you should be adapt your story to their concerns, questions or objections. Never hesitate to play with emotions, they are a major factor in decision making.

The most obvious emotion to play with is fear: fear of major outage. The one which will impact your revenue. For instance, 5-minute outage represent around one million dollars for Google/Alphabet, one hundred thousand dollars for Netflix and 3 million for Apple:

Outage Outrage: the True Cost of Tech Giant Downtime by Jolt

Moreover, it is during this type of incident that you should not hesitate to be opportunistic to advance your pawns, to propose new practices that will limit the impact in future incidents.

One of the best way is to speak about resilience - Ability to recover quickly from difficulties.

Werner Vogels, Vice President & Chief Technology Officer at Amazon, introduce this subject by explaining that with size and continuous evolution of our system: “Everything fails all the time”. The main point is no more to avoid failures, but to limit impact of those failures.

With you CIO/CTO, you may tell it is better to have an experiment in production during day time with everyone available, thus training teams and limit impact, instead of react during night outage with sleepy people and limited resources. The final goal of Chaos Engineering is to sleep like a log, without worrying about issues.

You may also point out links with actual practice. For business continuity plan, you already do experiments in production, disaster recovery testing. This type of experiment is at the heart of the Chaos Engineering approach, the main difference is that we don’t want to do it once, manually. We want to do it automatically and continuously.

Similarly, a Chaos Engineering experiment is about injecting perturbation and analyzing impact to detect weakness. This kind of analysis is very close to root cause analysis used during postmortem of outage. Regular practice during planned exercises, will allow your teams to improve their skills and to work more on prevention action than correcting issues. Moreover, it will test your monitoring and alerting system, a task rarely done in real life, even if detection is a major part of an effective system to limit outage impact.

You may also include you Chief Security as experiments will also help reinforce defense and security of your system: Security Chaos Engineering: A new paradigm for cybersecurity.

For the most concerned, do not hesitate to show that experiments of this type exist in more and more companies, as you can see in this mind map:

• Cloud and hosting companies: Amazon, Azure, Google, Digital Ocean, Pivotal Cloud Foundry, Twilio, Dropbox …
• Companies for final users: Netflix, Facebook, LinkedIn, Uber, OUI.sncf,…
• And even bank: Fidelity Investments. For the more adventurous, it is better to highlight the innovative side and precursors of these practices. Even if we are more and more, it is a discipline that remains to be explored.

In summary, as in the parable of the blind men and the elephant, every member of your executive board will see the point that will make them confident or even enthusiast:

Parable of the blind men and the elephant

This approach will allow you to build allies who will help you influence the system and launch Chaos Engineering in your company.

Still not convinced?

If you still think that R.O.I is the only way that will work for you, and that this article is just bullshit, please have a look at Gary Yukl “Leadership in Organization”:

Sources: Yukl, Lepsinger, & Lucia, 1992; Yukl, Chavez, & Seifert, 2005; Yukl, Seifert, & Chavez, 2008

Rational persuasion like R.O.I only provides 23% of engagement whereas inspiration tactics (90%), consultation (55%) and personal appeals (42%) are far more efficient.

In any case, buying a beer (31%) is always better than pressure (3%). In French, we say that we do not suffer pressure, we drink it, as pressure (“pression” in French) is a synonym for beer.

Please do share your tactics to convince your boss to do Chaos Engineering in comments below!

(french version)