Scaling Up After Initial Success: A Balancing Act

After the initial success of our product, the demand for our team skyrocketed. While this was an exciting milestone, it also presented a tricky challenge: scaling up our software development team while maintaining productivity. Balancing productivity and team growth is no easy feat. It requires strategic planning, effective communication, and a keen eye on maintaining our company culture.

This problem is even more pronounced when developing a centralized data architecture platform, which is steered and developed by multiple teams.

Symptoms of Scaling Challenges

Unattended Client Demand in Backlog: More and more requirements are added to the backlog as stakeholders start pushing their use cases to the centralized platform.

Eager Teams, Limited Support: Many teams are eager to work on the platform, but the central team cannot provide sufficient help.

Productivity Dips: Adding new team members can initially cause a dip in productivity as existing members spend time on onboarding and training instead of focusing on completing stories. For example, increasing team size by 20% in a short time frame found to be more counterproductive from delivery speed perspective.

Build Failures: Quick onboarding of new teams without sufficient training can lead to build failures and integration issues.

Complex CI/CD Pipelines: Evolving CI/CD pipelines can also add complexity to the mix, making it harder to maintain smooth deployments.

Work Distribution Challenges: Structural problems in distributing work across teams, especially in specialized areas like data modeling and data ingestion.

Maintaining Culture: Ensuring that new hires align with your product culture and values can be challenging as the team grows.

Coordination Overhead: Increased coordination overhead as the number of teams grow, leading to potential inefficiencies.

A Solution: Squad topology aligned with underlying software architecture supported by robust platform engineering practices

To address the scalability challenges, we are employing various squad optimization techniques - opportunistically consolidating or distributing work based on underlying software architecture, refactoring software architecture, and improving platform engineering practices.

Establishing Team APIs: Team APIs provide a structured approach to organize teams, set up efficient interaction, and manage workloads. This leads to better collaboration and a more efficient way of working with minimal disruption to other teams. All teams working on the central data platform are instructed to operate like a well-functioning API ecosystem. e.g. standardized datasets are provided by central squad to all reporting teams.

Centralized Platform Management Team: By providing reusable components and services, platform engineering allows non-platform squads to focus on their mission statement and deliver business value as quickly as possible. While stream-aligned teams busy delivering business value, the platform squad takes care of infrastructure monitoring, costing, SLA management, refreshing IT controls over time, creating/upgrading/removing CI/CD pipelines, controlling product deployment, and participating in internal and external audits.

Consolidation of Data Modeling and Data Ingestion Work: This team is not easy to scale up due to the following reasons:

• Keeping consistency and standards over 8-10 squads is not possible logistically. Any misalignment can lead to data quality issues, integration issues, and data reload issues.
• The complexity of managing models becomes more difficult as the data volume grows.
• Specialized skills: The data modeling team has to be aware of both functional and technical domains. Finding such profiles can be very difficult.

We decided to keep data modelling and data ingestion in a centralized squad to maintain consistency and quality but at the cost of scalability and flexibility.

Setting up Stream-Aligned Teams for Delivering Business Value: These teams have specific knowledge required for delivering (regulatory or internal) reporting requirements on the data platform without disrupting other teams once the underlying data is available on the data platform. From our experience, it was observed that these teams are easier to scale up as there exists a well-defined boundary (in terms of scope, deliverables and autonomy) for such squads. In ideal scenario, if the underlying platform is robust, we should witness a linear productivity growth once team is onboarded.

Refactoring Software Architecture: By breaking down a monolithic data warehouse architecture into smaller, manageable pieces, we anticipate greater agility and scalability with respect to software development. We are finalizing the best technology stack for each module to deliver greater development speed. These smaller modules should facilitate autonomy and flexibility to develop, deploy, and scale independently.

Overall, adopting a squad optimization strategy based on the underlying software architecture and adopting best practices of platform engineering has already started showing results. From a product lead's perspective, balancing the delivery of business value along with these such feels overwhelming at times and seems like we are changing a car's wheel while it is moving.

Nonetheless, this also makes the job interesting :).

Personally, I would love to understand how such problems are being tackled in your organization.

#ProductDevelopment #TeamGrowth #ScalingUp #Leadership #TechIndustry #ModernDataPlatform #ProductManagement #DataPlatform #Datawarehouse #Datamodel #Teamstopology #TeamAPI #PlatformEngineering #squadoptimisation