Based on our presentation (in French) during the second meetup of Paris Chaos Engineering Community, also done during ChaosConf, San Francisco, in 2018.
You may start by telling them you will break everything in production, and that will be fun! We call this in French “coming with big clogs — Gros Sabots.” They will see you coming from far, but except for noises, there are few chances you will hit the target.
They will probably ask for the Return of Investment (R.O.I.). By arranging the number of incidents, outage by year, cost of an outage, and some hypothesis to reduce them, you may quickly build a proposition with 2–3 years R.O.I.
However, between a project with new revenue and one which will maybe avoid losses, if they have to choose, the first one will probably be selected first.
The rational approach is not the best one for a subject like Chaos Engineering. You must review your influence strategies.
Step 1: make it familiar
Biologist Henri Laborit wrote in 1976: “Faced to unknown experience, man has only three choices: fight, do nothing or flee. “ This subject is entirely new, with few visibility, it is essential to allow your boss to discover the concept at his speed, to avoid automatic rejection.
You may start by sharing on the internal or external social network some interesting papers on the subject:
- Chaos Engineering on ThoughtWorks Technology Radar
- What Is Chaos Engineering? Answer by Kolton Andrus, CEO of Gremlin, Chaos Engineering Expert
- 4 Chaos Experiments to Start With by Matt Jacobs
- AWS re:Invent 2017 — Nora Jones Describes Why We Need More Chaos — Chaos Engineering, That Is
- Tammy Butow, SRE Manager at Dropbox, on Chaos Engineering
- The rise of chaos engineering, by Danny Chrichton
Depending on the tools used by your boss, you just need to find a way to send them those articles. I was lucky to be followed on twitter by many members of our executive board; I never hesitate to use it to make them aware on some subjects I will later discuss with them, including some post from executive influencers, like the CEO of a significant bank Société Général:
Our first goal here is only to make Chaos Engineering sound familiar before going further.
Step 2: identify your players
To be able to adjust your story to the different directors of your executive board, you need to identify the kind of players they are, among other things using sociodynamics:
You will be able to focus by adapting your strategy and identifying the way they move and act. Using your energy on the right priorities to make them allies.
- What are their major challenges?
- What they may win or lose by collaborate with you?
- What are their influences on other players?
Take time to identify the critical issues at stake with decision-takers, as well as the brakes and leverages concerning your subject. Concerns and objections are only excuses for your opponents to test you, but may also be real blocking points. You will win them by studying and including their views, but not necessarily during formal meetings…
Based on this map, you will be able to adapt your strategy:
- Friends (Zealots and Golden Triangles) — These are a dream as they are Promoters. They go out of their way to support the project. They are dedicated to their task, quick to provide resources and aid in issue resolution. It is essential to know these individuals and maintain them as allies. Ensure you understand what is critical to them to make the delivery.
- Fence (Passives) — They define 40–80% of the stakeholders. Despite one’s excellent interpersonal skills, you may not be able to reach all your stakeholders. A key asset in winning the masses is using influencers who run your campaign and positively influence the passives. As passives are impressionable, they are easily swayed based on the feel in your project, and it’s crucial they moved in your preferred direction. Winning the passives increase the synergy with your project team and increase the chances of project success. Other ways of wining Passives over is ensuring, these individuals feel part and parcel of the project, i.e., are consulted and informed in what is going on.
- Foes (Mutineer, Moaners, and Opponents) -They are a handful. However, they can cause significant damage to the synergy within the project team. Opponents and Mutineers are against your project for various reasons, and they take any opportunity to antagonize the project delivery. This handful of individuals may rally those on the fence to be antagonistic towards the project. Established who they are, their concerns, and evaluate if they can be resolved, e.g., do they want to be more involved. If not, contain their impact on the passives as they would sway against you.
Changing the global dynamic can take many forms. Sometimes it will be enough to change the views of some players on the subject … Be attentive to the evolution of the positions rather than the “photography” of these at a T time to mobilize the players who influence the dynamics towards the synergy.
Step 3: the right story for your players
Once you have identified your players, you should be adapt your story to their concerns, questions, or objections. Never hesitate to play with emotions; they are a significant factor in decision making.
The most apparent emotion to play with is fear: fear of significant outage. The one which will impact your revenue. For instance, 5-minute outage represents around one million dollars for Google/Alphabet, one hundred thousand dollars for Netflix, and 3 million for Apple:
Moreover, it is during this type of incident that you should not hesitate to be opportunistic to advance your pawns, to propose new practices that will limit the impact in future events.
One of the best ways is to speak about resilience — Ability to recover quickly from difficulties.
Werner Vogels, Vice President & Chief Technology Officer at Amazon, introduce this subject by explaining that with the size and continuous evolution of our system: “Everything fails all the time.” The main point is no more to avoid failures but to limit the impact of those failures.
With your CIO/CTO, you may tell it is better to have an experiment in production during day time with everyone available, thus training teams and limit impact, instead of reacting during night outage with sleepy people and limited resources. The final goal of Chaos Engineering is to sleep like a log, without worrying about issues.
You may also point out links with actual practice. For a business continuity plan, you already do experiments in production, disaster recovery testing. This type of experiment is at the heart of the Chaos Engineering approach; the main difference is that we don’t want to do it once, manually. We want to do it automatically and continuously.
Similarly, a Chaos Engineering experiment is about injecting perturbation and analyzing the impact to detect weakness. This kind of analysis is very close to the root cause analysis used during the postmortem of an outage. Regular practice during planned exercises will allow your teams to improve their skills and to work more on prevention action than correcting issues. Moreover, it will test your monitoring and alerting system, a task rarely done in real life, even if detection is a significant part of an effective method to limit outage impact.
You may also include your Chief Security, as experiments will also help reinforce the defense and security of your system: Security Chaos Engineering: A new paradigm for cybersecurity.
For the most concerned, do not hesitate to show that experiments of this type exist in more and more companies, as you can see in this mind map:
- Cloud and hosting companies: Amazon, Azure, Google, Digital Ocean, Pivotal Cloud Foundry, Twillio, Dropbox …
- Companies for final users: Netflix, Facebook, LinkedIn, Uber, OUI.sncf,…
- And even bank: Fidelity Investments.
For the more adventurous, it is better to highlight the innovative side and precursors of these practices. Even if we are more and more, it is a discipline that remains to be explored.
In summary, as in the parable of the blind men and the elephant, every member of your executive board will see the point that will make them confident or even enthusiast:
This approach will allow you to build allies who will help you influence the system and launch Chaos Engineering in your company.
Still not convinced?
If you still think that R.O.I is the only way that will work for you, and that this article is just bullshit, please have a look at Gary Yukl “Leadership in Organization”:
Rational persuasion like R.O.I only provide 23% of engagement, whereas inspiration tactics (90%), consultation (55%), and personal appeals (42%) are far more efficient.
In any case, buying a beer (31%) is always better than pressure (3%). In French, we say that we do not suffer pressure, we drink it, as pressure (“pression” in French) is a synonym for beer.
Please do share your tactics to convince your boss to do Chaos Engineering in the comments below!