Why Scrum Struggles with AI Projects: Rethinking Agile for Machine Learning Success

As an AI Project/Product Manager, I’ve witnessed firsthand the challenges of delivering successful AI/ML projects using traditional Agile frameworks like Scrum. While Scrum has revolutionized software development by providing structure and focus, its rigid, sprint-based approach often falters in the face of the complex, unpredictable nature of AI/ML projects.

The question isn’t whether Agile methodologies can be applied to AI projects—they can. The real question is: why does Scrum, in particular, struggle to deliver AI projects, and how can we rethink Agile to better suit machine learning success?

The Unique Nature of AI/ML Projects

AI projects differ from traditional software development in several critical ways:

1. Experimentation and Research: AI/ML development is inherently experimental. Unlike building an app feature where requirements are clear, AI/ML projects involve a lot of uncertainty. It’s more about discovering patterns in data, testing hypotheses, and refining models. This often means that the outcome is unknown, and the time to achieve it is hard to estimate.
2. Data-Dependent Cycles: AI/ML models rely heavily on high-quality data. Much of the work involves gathering, cleaning, and labeling data, tasks that can be unpredictable and time-consuming. Without good data, no amount of sprint planning can save the project. These activities don't always fit neatly into two-week sprints.
3. Non-linear Progress: AI development doesn't follow a linear path where each sprint builds on the last. There may be setbacks, like when a model underperforms or when the data reveals unforeseen issues. This non-linear, trial-and-error process makes it difficult to plan and deliver incrementally within the confines of Scrum’s sprint cycles.
4. Infrastructure Needs: AI models often require specialized infrastructure for training and deployment. Training a complex machine learning model might take days or even weeks, which can derail a sprint that expects continuous progress.

These factors combine to make AI/ML development feel more like research than software engineering. While Scrum works well in environments with predictable outcomes, AI/ML projects often unfold in ways that are hard to anticipate or control.

Why Scrum Struggles with AI/ML Projects

Scrum’s structured framework works brilliantly for projects where incremental progress is achievable, such as developing a new feature for an app or website. But the following characteristics of AI/ML projects can cause friction with Scrum’s core principles:

1. Sprint-Based Development Doesn’t Match AI Timelines: Scrum is designed around the idea of delivering functional software or product increments at the end of each sprint. In AI/ML, you might spend several sprints just cleaning data or tuning a model. This makes it difficult to show tangible progress or results at the end of each sprint.
2. Fixed Sprint Cycles Conflict with Iterative Research: AI/ML development requires flexibility and the ability to iterate as new insights emerge. Scrum’s rigid sprint structure, with predefined goals and backlogs, can be too restrictive when model experimentation is required. Teams may need to pivot or revise objectives mid-sprint, which doesn’t align well with Scrum’s commitment to delivering predefined tasks.
3. Definition of “Done” is Vague in AI: Scrum relies on clear definitions of "done" to ensure a task is completed by the end of a sprint. In AI, defining “done” is tricky. A model might work well in one context but fail in another. Furthermore, continuous improvement is a hallmark of AI models—they often require retraining as more data becomes available, which makes it difficult to establish when something is truly finished.
4. Model Deployment Delays: AI models often require significant testing before they can be deployed, and deployment itself is a non-trivial task. Unlike traditional software, where deployments can be part of regular sprint cycles, AI models might require specialized environments, adding another layer of complexity to timelines.

Rethinking Agile for AI/ML Success

Scrum may not be the best fit for AI projects, but Agile principles—flexibility, collaboration, and iteration—are still critical to success. The key is adapting Agile to fit the unique needs of AI/ML development. Here are a few approaches I’ve found effective:

1. Use Kanban for Flexibility: Kanban, another Agile framework, provides more flexibility than Scrum. By focusing on continuous delivery and allowing work to be pulled as capacity becomes available, Kanban can accommodate the research-driven, unpredictable nature of AI/ML projects. It enables teams to respond quickly to new findings without being tied to fixed-length sprints.
2. Incorporate Research Sprints or Discovery Phases: Instead of expecting functional increments from the start, organizations can incorporate dedicated "research sprints" or “discovery phases” into the AI/ML project lifecycle. These sprints focus solely on exploration—collecting data, tuning models, and testing hypotheses—without the pressure of delivering a finished product at the end.
3. Adapt the Definition of Done: For AI/ML projects, the definition of “done” should be more flexible and reflective of progress rather than finality. For instance, a model might be considered “done” when it reaches a certain performance threshold in testing, even if it’s not fully production-ready. Acknowledge that AI models are never truly finished—they can always improve with more data or better tuning.
4. Invest in AI/ML Infrastructure Early: Infrastructure is a critical piece often overlooked in the early stages of AI/ML projects. Building infrastructure to handle data pipelines, model training, and deployment early on can reduce delays later in the project. Integrating tools that support continuous deployment of AI models can also help streamline the process.
5. Encourage Cross-Functional Collaboration: Successful AI/ML projects require input from data scientists, engineers, product managers, and domain experts. Ensuring open communication and continuous collaboration throughout the project is vital to ensuring everyone is aligned on goals and expectations, even when things don’t go as planned.

Conclusion

Scrum has proven effective in many software projects, but its sprint-based, structured approach is often too rigid for the experimental, data-driven nature of AI/ML development. By recognizing the unique challenges of AI/ML projects—uncertainty, non-linear progress, and data dependency—we can adapt Agile principles to better suit these projects. Whether by adopting Kanban, redefining “done,” or investing in infrastructure, organizations can set themselves up for AI/ML success.

In the end, it’s about embracing flexibility and creating a development process that accommodates the iterative, exploratory nature of AI. With the right approach, we can bridge the gap between Scrum and AI, ensuring that AI projects deliver on their enormous potential.