Progressive exposure using deployment rings and feature flags

Editor's Notes

• #3: Welcome. Today we’ll talk about progressive exposure using deployment rings and feature flags. Is everyone in the right session?
• #4: So, my name is Willy Schaub I’m a program manager with the Visual Studio Team Services and the ALM | DevOps Rangers teams. Contact me at willys@microsoft.com or give me a shout on twitter @wpschaub.
• #5: In case you're wondering about my accent, I was born in Basel Switzerland, I grew up in sunny Johannesburg South-Africa, and am currently working in Vancouver Canada, the city with the liquid sun.
• #6: Here are a few pictures showing my interests once I’m unplugged from digital world. I love to explore our oceans, I’m fascinated by aviation, and enjoy watching sunsets wherever I am.
• #7: I work on a thing called Visual Studio Team Services, or VSTS, and with the ALM | DevOps Rangers, a part-time community VSTS is our collaboration solution for engineering teams, allowing you to take an idea, plan, develop, test, release, and monitor your solutions. ​<CLICK> Whether you are targeting the cloud, hybrid cloud, or an on-prem environment, VSTS is our Azure DevOps solution, for any language, any platform. CLOUD – HYBRID CLOUD – ON-PREM
• #9: Here’s our very own definition of DevOps, which is … “the union of PEOPLE, PROCESS and PRODUCTS to enable the continuous delivery of value to our CUSTOMERS”. In this session we’ll focus on two strategies we use to continuously deliver value to our users. Ring deployments and feature flags.
• #10: Rings …
• #11: … as first discussed in Jez Humble’s book, make it possible to progressively deploy binary bits and have multiple production releases running in parallel. For example, when your developers complete a pull request with proposed changes to the master branch, <CLICK> (1) a continuous integration build performs the build, unit testing, and triggers an automatic release to the Canaries environment in production. When you're confident that the release is ready for user acceptance and exploratory testing in production … <CLICK> (2) you approve the release to the Early Adopters ring. <CLICK> Similarly, when you're confident that the release is ready for prime time, (3) you approve the release to the Users ring. The names and number of rings depends on your preferences, but it's important that all rings are using the same release and are all in your production environment.
• #12: Using a ring-deployment strategy … You can gather feedback to validate your hypothesis Decommission old releases, and Distribute new releases when you are confident that everything is working properly … … without the risk of affecting all users in PRODUCTION. It’s about progressively exposure your release to more and more users.
• #13: Feature Flags …
• #14: … popularised by Martin Fowler, decouple release deployment and feature exposure, down to the individual user. A typical feature flag implementation is based on <CLICK> (1) a management service that defines the flag and its false, true, or multi-value. <CLICK> (2) a run-time query to figure out the value of the flag, and <CLICK> (3) an if-else programming construct.
• #15: Using feature flags … You can hide incomplete, regional or unreleased features in production You can fine tune a release for all or selected users Emergency roll-back … feature does not work? Flip a flag and it’s gone.
• #16: Let’s look more closely at how we use rings and feature flags and what we learned to date.
• #17: For VSTS we use the ring-deployment model for our production environment, with multiple rings. The first ring is our canary ring. That's where we live, for example the VSTS team, the Early Adopters, and the ALM | DevOps Rangers. We’re going to be the first to experience pain if there’s an issue with the release … which is great. Then we release to and test the smallest data centre, then the largest data centre, the international data centres, and finally all the rest. TEST IN PROD – EVERYTHING IS IN PROD CANARY – SMALLEST – BIGGEST – LATENCY - MAINLAND
• #18: We ship features every 3-weeks and bug-fixes daily, sometimes multiple times a day, in production. We use a custom feature flags solution to progressively expose, test, enable or disable features, for everyone, selected users, or as selected by users.
• #19: The Ranger community also use the ring-deployment model for their production environment, with three rings. The first ring is our canary ring and includes all active Rangers. <CLICK> The second contains our early adopters who are willing to share the pain of evaluating preview bits and help us raise the quality bar. <CLICK> The third is the Users ring, which contains all the users who have access to the public marketplace. TEST IN PROD – EVERYTHING IS IN PROD
• #21: When we explored the use of feature flags for our VSTS Extensions, we reviewed custom, open source, and service-based solutions. <CLICK> We decided to keep it simple and opted for LaunchDarkly. It’s comprehensive, it’s simple, and they support the open source community. <CLICK> We also abstracted the feature flag logic to query and set feature flags in Azure Functions to keep our extension code as simple as possible. <CLICK> Both our extension and the Azure Function solutions are open source. You can find, use, and contribute to them on GitHub.
• #22: Here’s a quick comparison of how the ALM | DevOps Ranger community views rings and flags in the context of VSTS Extensions. Both support progressive exposure. For rings the primary cost is the maintenance of the production environment. For flags it’s the technical debt. We use rings to manage the release blast radius, and flags to fine tune the release. The blast radius for a ring includes all its users. For Feature flags we can target none, specific, or all users.
• #23: Caution …
• #24: … and be aware that progressive expose comes at a cost. By adding flags you’re adding technical debt to your code and increasing your code and test paths. It’s also important to understand the implications of flipping a feature flag. <CLICK> We had our Corporate Vice President, Brian Harry, on stage when we flipped a feature flag at a big marketing event. It didn’t go well. When we enabled the feature, everyone started using it, we experienced authentication failures, and the backend systems buckled under load. <CLICK> When we enabled feature flags in one of our most popular VSTS extensions, users experienced 503 errors, then severe performance issues as our Azure Functions failed under load. In both examples we did not anticipate the impact on our system. We investigated, mitigated, learned a lot, and shared the experience in two comprehensive blog posts that you should explore. It’s a common theme in DevOps. A journey of continuous learning and improvement, with a destination you never quite get to.
• #25: Finally, here are a few short links for reference information, including the two mentioned blog posts.
• #26: Thanks for listening! Any comments or questions?
• #35: Thanks for listening! Any comments or questions?