White Paper - ART of Federation

Creating a top performing and efficient engineering organisation for an enterprise is an interesting undertaking. There are multiple constraints at play simultaneously that are not always aligned. For instance, business needs significant amount of changes in order to respond to ever changing business environment, that often prompts to purchase of COTS product that gives a significant lead time as most functions are pre-build. However, the total cost of ownership (called TCO - since we all love abbreviating important phrases that contribute to decision making) multiplies to almost unacceptable levels if we buy COTS for every department.

What is the solution then? As Buddha said - find a middle path. In this context, it means purchase few "Generic COTS platform" (if there is such a thing) that would give you enough pre-built capabilities but also allow you to build individual variations for each department if you like. Lately, such platforms have been very successful like API Platform, App-Dev Platform, Reporting Platform that would abstract all the complexity of every domain leaving your engineering teams to focus just on the problem at hand. Some organization even go next level by creating own platform by assorted open source tools and creating an abstraction layer of consumption for rest of the engineering organization, giving a similar advantage at much cheaper price point. 

Now that we have found a way to a Tech nirvana, organizing our engineering teams is the next big challenge. The last wisdom tells us to organise cross functional tech teams closer to the Gemba - but that means engineers with different belonging are working on one Platform (Or few platform). That's like building a castle on someone else's kingdom, sooner or later there may be confrontation. There are for sure better ways of building a castle on a shared land in modern world - apparently we call them apartments or township, where there are rules of segregation (read "containerisation").

This paper discusses the thought process in building a technology platform that allows scaled delivery, various patterns of achieving that and considerations around implementation

Layers of Abstraction

Some of the low fidelity platforms like SharePoint, O365 tools give a flavour of what a federated mode of Application or capability development entails. For instance, share-Point completely abstracts the user from underlying technology nuisances at the cost of flexibility of what we can build on top of SHP. While we create stuff directly on SHP prod platform, that may not be possible for app platform with integrations. So we need to provide similar guardrails by other means like DevOps tooling, virtualized integration or if possible docker based full-stack containers.

The stronger the abstraction - higher the flexibility of change but lower flexibility of scope of change

For instance, a UI platform could provide full federated service for creation of UI or skins, where changes could be allowed to be made by UI developers siting with the business with little change control, but any changes to underlying business logic is handed over to a more specialised team working under relatively higher governance.

Such a platform should provide technology access control based guardrails to prevent accidental or intentional changes to the core engine of the software

Second layer of abstraction is the specialized team that maintains the background integrations, data flow of the application(s). They could be working in an assembly line controlled environment (Kanban or Scrum in software terms) - where they roll out changes needed by teams at higher abstraction layer at a relatively faster pace whilst still maintaining the sanctity of the technology patterns centrally.

 Such teams could be your API teams maintaining API(s) exposing features of core domains of your enterprise or your domain workflow (business logic) teams

Third layer of abstraction is where there is little opportunity of federation due to high risk of deviation from standards. For instance the teams creating and maintain your core platforms itself.

Scalability Paradox - Control what is allowed to scale how many are allowed

In order to help business respond to market changes faster, these days teams are organized around the business operations instead of tech capabilities. That means developers of a single team build apps on multiple platforms and link them together as per ever changing business needs. Normal logic would dictate that best way is to give them full control of underlying platform so that they can build what they want, but doesn’t really help at scale (Imagine if every service centre mechanic is allowed to change the piston of your BMW, you soon wouldn’t be driving a BMW).

To help find the right balance, we need to purposely build restriction to implement various levels of abstraction in the technology platform(s). This provides the guardrails that removes the need for governance forums as long as all adhere to their abstraction levels.

DevOps and its adoption

DevOps, these days is no longer just about automation your code promotion. It now serves as a mechanism to enforce the guardrails at every aspect of software creation. Pretty much the role automation plays in modern aircraft ensuring that all the heavy lifting is done by automation and pilot does the all critical just in time decision making.

Let's call it DevOps++.

If you have to let your decentralised developers do changes at will, you need to ensure that you let the tool do the heavy lifting and enforce standards by not allowing code commits, thereby removing the need for any manual policing. These areas could be

• Security testings (DevSecOps is the new jargon)
• Performance testing is automatically performed in your code promotion and code rejected if it fails
• Test Data and virtualisation as part of pipeline

This will help your federated teams to cruise through the "invisible" gates.

In order to Promote Leadership - Automate or delegate management

Yes, automation is fine but what about things we cannot automate !! The more subjective aspects of delivery. For instance, How to ensure Teams are following standard patterns that are often not enforceable by tools

Some options are

• Option 1 – Have a governance forum that enforces it !! ….Bottleneck !!!
• Option 2 – Encourage a Culture of Self Governance !! ……. Too Idealistic !!

 Again following the middle path. The mantra is

Decentralize low risk decision and centralize high risk ones

This can be implemented in technical decision making by having governance forum like Architecture approval etc purposely limited to high cost items like product choice, cloud strategy etc. Have a gradual pathway to shift or defer the ground level decisions - which by the way are not that costly but reap a huge benefits - to the teams. This will invariably seed confidence and train leaders from bottom up. 

Like Wise, same philosophy can be extended to management. For instance, Product Owner should have portion of enterprise maintenance and reliability budgets. Many organization these days centralize these funds employing a big red tape around spending such maintenance budget causing costly delays.

We can in fact learn a lot from the way our bodies work - most skills once mastered are delegated to the subconscious mind and arguably to each organ system. This frees up lot of conscious mind time in strategic thinking and things that matter most. Same we need to aim for an evolving enterprise, by continuously looking for opportunities to transfer accountability to the teams.

That doesn’t guarantee that the leaders will use the clawed back time in constructive things and organisations may still fail. This is similar to corresponding human analogy - having a free conscious mind doesn’t guarantee a successful life.

Financial Trade Off – Share the Cost vs Scale the Price

Now that we have enough wisdom about various permutations of federated delivery, the question is how do we decide what level of federation is appropriate for your enterprise. That decision is going to drive what COTS product or open source products you will buy - since at the end of the day you have to distribute the cost to justify TCO.

This dilemma is similar to the one you have when you are buying your home - apartment vs suburban house. Apartment allows you to have a CBD location but the deal is to share the cost of services and that brings in various restrictions on structural changes you can do. On the other hand, sub-urban house will give you flexibility on how you want to build and lesser governance and restrictions but you trade off no distance from CBD etc.

Depending on your personal situation, preferences and long term needs you decide on the kind of property you purchase. Likewise, in your enterprise you can decide and engineer the platform accordingly. If your organization is sufficiently large and tech department and business is ready for Significant federation, then you can invest in container based tech across your infrastructure with products that support containerized build and deployment.

More often than not, most organisations are littered with variety of COTS products that may or may not support Infra level containerisation. In such scenarios you can always achieve app layer segmentation by architecting the application platform with segmentation in mind. Your operational processes, hosting decisioning, operational process and automations needs to be prioritized accordingly.

The Art of federation is in understanding your organization and tailoring your own structure instead of lifting a killer model from the market