Skip to content

Christophe Rochefolle

Experienced IT executive providing tech & organization to improve #quality & #agility of IT systems, #ChaosEngineering fan ; #LGBTQ+ #DiversityInclusion

Menu
  • Home
  • About me
  • Values
  • Books
  • Conferences
  • Readings
  • Inspirational pictures
  • Contact
  • Français
Menu

#DevOps: What if we deploy on Friday?

Posted on 8 December 20198 December 2019 by Christophe ROCHEFOLLE

From FEAR culture to TRUST culture

Often, after reading a tweet from Charity Majors, my obsession to NOT follow rules only because « we’ve always done it that way » inspire me to write a post. Hope you’ll enjoy it.

Forbid to push in production on Friday is a #DevOps anti pattern  

More than just make Dev and Ops work better together, DevOps aims at delivering as soon as possible any developed and tested features to our users, get the expected value and having quick feedbacks. 

Forbid to deploy on Friday is to lose 20% of time to deliver this value to your users. Sometimes, we even forbid it on Thursday afternoon or before Bank Holidays, thus forbid more than 30% of time ! 

More than lose this important time, we give the impression that we are not confident of our CI/CD pipeline, of our tests, maybe not even trust our teams. If you are not confident to deploy on Friday, why are you to do it any other day? 

This rules implies than deploying is dangerous, risky. We should not FEAR  to deploy in production. 

If you believe you are #DevOps, you should be able to deploy any times, nights and days, even on Friday!

Why do we have this rules?

Doom-mongers will tell you that this rules was not set only to please the boss. That you need it if you want to protect your week-end. It’s common sense, precautionary principle…

And to be fair, we all have many anecdotes and experiences of bad weekend. So even if you agree that we should break this rules, it can be hard to make the move. 

Let’s step back a little bit. Why do we regret having deploy on Friday? 

1- Deployment went wrong and you have to stay late, even overnight to fix it.

Having the right level of DevOps maturity, deploying should be event less. You can build this with a CI/CD pipeline, automated test who provide the right level of confidence, a blue-green deployment for a instant rollback, … You must concentrate on building trust on your build process. A lot of books might help you, my favorite is Continuous Delivery by Jez Humble.

2- Incident some time after the deployment, managed by “On Call team”

The real subject here is why the team did not detect the issue before or at least right after the deployment? 

First of all, the team must monitor their application in production after the new release. It’s criminal to deploy on Friday at 5pm and then go have a beer. 

Then, do the team have the right level of observability to detect the issue? During a DevOps transformation, you must not only improve deployment pipeline and testing but also improve the way you monitor your systems. With increasing complexity, classic monitoring and alerting are no more enough to detect abnormal behavior of our systems. About Observability, I highly recommend this white paper by Charity Majors and Liz Fong-Jones, as well as this ebook from D2SI and my friend Benjamin Gakic.

Finally, why the incident is managed by a separate “On Call team”? A responsible team should assume their own actions, risk taken and obviously the consequence. Thus, the team will make everything to not lose nights and weekend, by building a safe process for their release. 

3- Unknown unknowns 

Charity Majors Chaos Engineering and Observability 

Last but not least, the unknowns. Deployment were seamless hundred times. But this time, something went wrong. A concurrent deployment do a change on the same file. A server with a different system date believe it’s already tomorrow… All these issues never happened before, and probably will never occur again. 

They are part of the risks, even if it’s more frustrating on Friday’s nights. But do we not go outside because a meteorite might fall on us? 

The best thing to do is to be prepared, and Chaos Engineering practice are a great way to do it.

To summarize, with the right level of DevOps maturity, this rules should not exist anymore.

How to move on?

You want to move on and allow your teams to deploy any times they can. But a small voice still say no (might be your boss or the Head of Operations). My proposition is to have a “licence to deploy”.

Principle is quite simple. If a team has the right level of maturity:

  • Great coverage of automated tests, 
  • CI/CD pipeline 
  • Feature flipping to activate/deactivate new features
  • Instant rollback capability, like blue-green deployment
  • Is On Call
  • …

This proposition should be review adapt to your context and needs.

In summary, instead of having a rules to forbid to deploy on Friday by default, let’s trust your team by default and deliver them a licence to deploy anytime.

You may also add some point than make you lose your licence until you did a new training or fix some technical debt:

  • Forget to adapt monitoring system during deployment and sending false alarm: -1pt
  • 3 consecutive deployment with rollback: -3pts
  • Launch a deployment and go for lunch or having a beer without checking the results: – 6pts
  • …

Deployment distribution graph is an interesting way of measuring your level of DevOps maturity

However, it’s still a number, if your whole strategy of change is based on it, teams might just wait for Friday for their deployment to please you. My favorite KPI is the level of code stock, that’s to say the code done (and tested) that is not in front of its public.

Thus, removing the rules forbidding any deployment on Friday, eventually with a licence to deploy, we move from a culture of FEAR/PROHIBITION to a culture of TRUST.

As a bonus, we all face the issue of recruiting Tech profiles, being able to tell that you trust your teams enough that deploying a Friday is part of your company culture might help attracting talents !

Share :

  • Click to share on Twitter (Opens in new window)
  • Click to share on Facebook (Opens in new window)

Articles similaires

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Follow me

Tweets

RT @SNCFConnectTech 📸 Retour en images sur notre #Hackathon. Des projets innovants pour enrichir et simplifier l'#expérienceclient et tous les déplacements ! Bravo 👏 aux 3 équipes gagnantes et à l’ensemble de nos collaborateurs mobilisés pendant près de 48h ! #DigitalMobilityChangers pic.twitter.com/As8Q…

About 16 hours ago from Christophe Rochefolle's Twitter via Twitter for iPhone

@FlutterFrance @FrancoisN0 @FlutterDev @SNCFConnect @Green_spector @FlutterGdOuest @SNCFConnectTech Bonjour - avec plaisir contactez-moi en MP

About 3 days ago from Christophe Rochefolle's Twitter via Twitter for iPhone

SI vous avez un peu plus d'une heure pour m'écouter partager mon parcours de #testeur à directeur technologies #cto #qualityengineering #agility #devops #chaosengineering podcast.ausha.co/cto… twitter.com/KaibeeIT…

About 5 days ago from Christophe Rochefolle's Twitter via Twitter for iPhone

C’est toujours un moment fantastique de créativité et d’entraides, des idées à foison, des premières démo épatantes dès la première journée quand on passe voir les #hackathon ien.ne.s : #innovation #DigitalMobilityChangers twitter.com/SNCFConn…

About 5 days ago from Christophe Rochefolle's Twitter via Twitter for iPhone

RT @SNCFConnectTech 🚀 Un #Hackathon 2022 qui invite nos collaborateurs à libérer leur créativité et replace l’#innovation et la cohésion d'équipe au coeur de notre démarche. 🆕 Des clients membres de notre communauté Connect & Vous viendront challenger les projets et aider à trancher ! 🏆 pic.twitter.com/hqKd…

About 6 days ago from Christophe Rochefolle's Twitter via Twitter Web App

RT @SNCFConnectTech Nos talents #tech sont prêts à hacker le futur des mobilités durables 💪. Les défis pour construire l’avenir sont nombreux et les services de demain se développent dès aujourd’hui. 3,2,1… Codez ! 👨‍💻👩‍💻 #Hackathon #Innovation pic.twitter.com/HX4s…

About 6 days ago from Christophe Rochefolle's Twitter via Twitter Web App

Dans un voyage ce n'est pas que la destination qui compte mais aussi le chemin parcouru. De même, la sobriété dans la conception d'applications est une démarche à mettre en œuvre tout au long du delivery. Nous sommes fiers aujourd'hui des premières étapes parcourues. #silverlabel twitter.com/SNCFConn…

About a week ago from Christophe Rochefolle's Twitter via Twitter for iPhone

RT @SNCFConnectTech L’application #SNCFConnect notée 75/100 par @Green_spector #Silver💪 De bonnes pratiques de conception alliant tech et design, sont un levier majeur pour réduire les impacts environnementaux🌍 et sociaux tout en conservant une qualité d’usage. #écodesign twitter.com/Green_sp…

About a week ago from Christophe Rochefolle's Twitter via Twitter for iPhone

@IgurumiP Je vois pas du tout ce que tu veux dire 😁 #ironic

About a week ago from Christophe Rochefolle's Twitter via Twitter for iPhone

@IgurumiP Oui, un excellent niveau. Pourquoi être surpris ? #DragRaceFrance

About a week ago from Christophe Rochefolle's Twitter via Twitter for iPhone

Posts

  • Chaos (6)
  • Conferences (1)
  • DevOps (3)
  • Management & Leadership (3)
  • Quality & Excellence (1)
  • Values (4)

Abonnez-vous

Join 3 other subscribers

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
© 2022 Christophe Rochefolle | Powered by Superbs Personal Blog theme