What is PACE? Your Starting Point to Building a Culture of Experimentation
A few successful tests don’t establish a true culture of experimentation.
Building such a culture requires embracing the spirit of experimentation. This means letting data guide whether a hypothesis is validated, rather than focusing solely on winning as the goal of a test.
An organization that values experimentation understands that success isn’t guaranteed. Hypotheses can be proven right or wrong. The goal is to avoid launching experiences without proper testing, knowing the risks of releasing untested ones.
Would you prefer to launch a test, find that the control performs better, and save hundreds of dollars? Or skip testing, implement a new UX change with confidence, and risk a significant drop in revenue that could take a long time to recover?
This understanding must be shared by everyone in the company, not just top management.
But how do you build such a culture? It can’t be achieved overnight or with just a few steps.
In our interview with Ngo Wei Kang Gladwin , VP of Growth at Crimson Education, Singapore, he shared how he fostered a pro-experimentation culture using the acronym PACE. We’re excited to share these insights and our own with you.
P - Preserve knowledge by creating a centralized test bank. This repository records all tests conducted, allowing team members to learn from past experiments and find inspiration for new tests.
We couldn’t agree more.
Centralizing test data reduces redundancy and increases efficiency. Teams do not need to reinvent the wheel with each new experiment. Instead, they can refer to the test bank for relevant information and avoid repeating tests that have already been conducted. This leads to more consistent experimentation practices and a more efficient use of resources.
A centralized bank that is accessible to all ensures that valuable insights and learning are not confined to a select group of individuals or teams. Every member of the organization has the opportunity to learn from previous tests. This democratization of knowledge fosters a more inclusive environment where everyone can contribute to and leverage the shared expertise and insights of the entire organization. This way, experimentation becomes more systematic and inclusive, ensuring growth and innovation in the organization.
VWO Plan, our free built-in project management tool, serves as a centralized repository for your insights, past test records, and comments. It provides clear visibility into the status of tests, including those in progress and the backlog. This tool streamlines and structures the experimentation process, ensuring everyone stays informed and aligned without getting lost in the ocean of data and information.
A - Assure quality by standardizing the end-to-end experimentation process. This includes comprehensive test plans that cover ideation, hypothesis generation, execution, tracking, and auditing of results.
We are game for processes.
You don’t create great by planning abruptly or sprinkling surprise elements now and then. Processes, consistency, and standardization might seem like the same old boring terms, but they have their blessings. These bring consistency in what you do, especially when you are running tests because there are many steps involved that you might overlook or forget easily if you don’t have them accepted as a standard.
This commitment leads to quality tests - tests with either winners or learners. Either way, you know the resources are not invested without any reason. Standardizing the end-to-end experimentation process—including ideation, hypothesis generation, execution, tracking, and analysis—creates a structured framework that fosters a culture of experimentation.
By having clear guidelines for each phase, teams can consistently generate and evaluate new ideas, formulate data-driven hypotheses, and execute tests uniformly. This approach ensures that all experiments are conducted methodically, results are tracked accurately, and insights are analyzed systematically.
The VWO platform integrates all the capabilities needed for an end-to-end experimentation process in one place. From research and hypothesis building to testing and post-test analysis with segmentation, VWO covers it all. The success of your experimentation process now depends on your commitment to leveraging these resources effectively.
C - Create a scoring system for each experiment based on its potential impact—high, medium, or low. Each team member is required to achieve a set number of points as part of our team OKRs, ensuring everyone is actively engaged in meaningful experimentation.
It’s like hitting two birds with one stone: first, the scoring system helps prioritize ideas when you have too many tests in the pipeline. Second, it encourages team members to actively participate in experiments, giving them a clear target tied to their OKRs, which fosters a sense of ownership.
Ranking experiments ensures that the team's efforts are directed toward the most valuable experiments, streamlining the decision-making process. This system aligns individual and team goals with the broader organizational objectives, as team members are motivated to engage in experiments that have the potential to make a significant impact.
By incorporating these scoring metrics into team OKRs (Objectives and Key Results), you ensure that everyone is actively contributing to and investing in meaningful experimentation. This approach fosters a competitive yet collaborative environment where experimentation is a shared responsibility across the organization.
Some might question why to use a scoring system when we can simply discuss and decide on priorities. However, relying on discussion alone can lead to subjective opinions about what’s most important. For example, you might prioritize implementing a countdown banner for a sale, while someone else might consider fixing a navigation glitch more urgent. A scoring system provides a consistent way to determine priorities, reducing the scope of personal biases.
E - Encourage innovation by developing AI prompts to assist team members in generating new test ideas and reviewing their test plans. This fosters creativity and efficiency in our experimentation process.
Running a large number of tests is better than running a small number because that way, you can screen every change through testing before implementing it. However, running a large number of tests is no joke, especially when we consider the standard process behind every test that we discussed earlier.
So, what do you do? Not run enough tests because you don’t have time? No. That feeling is a thing of the past. With generative AI making waves in CRO, many aspects of the process have become streamlined.
If there’s something AI can do better or faster than you, let it. You’ll still find that your role is as important as ever. Gladwin encourages his teammates to use AI prompts to generate test ideas and refine test plans, which is commendable. This approach fosters fresh, out-of-the-box thinking, frequent testing, momentum, and a faster experimentation cycle. These benefits help build confidence in experimentation as a core practice.
VWO, our experience optimization platform, provides you with AI-generated personalized testing ideas based on the goals you want to achieve on your webpages. You can select from one of the goals that VWO shows you or enter the goal you want to generate ideas for.
Not only are ideas furnished in a matter of seconds, but they are backed with hypotheses, scientific principles, and actionable steps to take. This feature is your savior when you hit a creative roadblock but need to maintain the momentum of testing. With these quick yet personalized suggestions, you can save time and focus on other key aspects like strategy and implementation.
We are confident that the steps Gladwin shared will be effective. However, we have a few additional points to consider:
Win over top management
Without the support of top management, building a culture of experimentation is challenging. Securing budgets for experiments is part of it, but having management’s backing means you can approach experimentation without fear of failure. Management understands that experimentation’s value lies in discovering unknowns and learning from results, which is why testing is essential.
If you are a mid-level manager or specialist, communicate this to top management. With their support, the process becomes much easier. As Jeff Bezos put it, Amazon’s success comes down to the number of experiments they run per year, per month, per day
Win your team’s trust
What if you are in top management and need to earn your team’s trust? Beyond setting up processes, understand the psychological roadblocks your team might face. People naturally prefer the status quo, and your employees are no different. Motivate them to embrace experimentation by showcasing successful companies like Amazon, Netflix, and Booking.com.
Demonstrate how changes based on user behavior can lead to improved results and encourage a positive attitude towards learning from unsuccessful tests. Introduce processes gradually and provide training if needed.
Choose the right tool
The choice of experimentation tool can significantly impact how well the team adopts the process. A tool with a steep learning curve can discourage users from engaging. Ensure the tool integrates seamlessly with your existing tech stack to avoid complications. Ideally, find a platform that consolidates research, testing, and other functions (like VWO) into one place, so your team doesn’t need to be trained on multiple tools for different purposes.
Way forward
The path to excellence in experimentation isn't built overnight, but with the right approach, tools, and mindset, every organization can develop a thriving culture of experimentation that drives innovation and growth.
Start with PACE, add the essential elements of leadership support and team engagement, and watch as your organization transforms into one where experimentation becomes second nature. For more insights about building a culture of experimentation, read our interview with Gladwin where he shares his experience and practical tips for CRO success.