5 metrics to help you measure the success of a pilot

Testing new technologies in a low-risk environment -- think test lab or sandbox -- before moving to full scale is a wise move.

But how to plan your pilot and run it effectively?

Just like any other “traditional” project, some may say. You need a clear timeline and detailed deliverables. You also need to identify risks, determine a methodology, allocate resources, assign roles, define QA protocols... But is it enough?

Let’s assume it is. Let’s also assume that, as the pilot goes, the pilot team delivers on time and within the allocated budget.

Now comes the most important question; was the pilot successful?

Well, from a project management perspective, it was. But was the pilot itself, or the solution you are testing, a real success? What was its impact on your organization? Do you have all the information you need to deploy it at full scale?

Well, not so sure.

That’s where performance indicators (aka KPIs for Key Performance Indicators) come into play. Setting specific and measurable targets early on, at the planning stage, in order to define what success (or failure!) looks like, will reduce uncertainty and help inform the decision process.

But what type of KPIs should you use to measure the performance of a new solution that has never been tested before?

To help you answer this question, I collected information and identified best practices developed by some of the most innovative cities in the country to build 5 KPI templates.

1- Set a survey target to measure users satisfaction

Build a survey and set a "satisfaction target" to measure the satisfaction of the future potential users of the product or service your organization is testing.

Keep it simple - “how likely are you to recommend this new product or service?” - using a scale of 1 to 5. Make sure to survey enough people to get statistically significant results, and don’t forget to use the survey to identify the detractors (score of 1 or 2) or the promoters (score of 4 or 5) of the project.

2- Set a numeric goal to track usage

Setting a numeric goal to track usage, such as a target number of users or of deliverables - eg. 100 residents will sign up for the new service or 5 sensors will be installed - is very helpful, especially when rolling out a new product or a new service that did not exist before. Try to get information from your partners about similar projects to help you set an ambitious yet achievable target.

3- Set a percentage target to measure engagement

Just like numeric targets, setting a goal in percentage, such as a target share of users or a target percentage of usage - eg. 10% of the employees will use the new service or the service will be accessible to 80% of the residents - is a good place to start with a new technology that hasn’t been implemented before.

4- Set a savings target to measure efficiency

Savings targets are critical to measure the expected economy made or reduction in resources (time and money!) resulting from the implementation of a new technology - eg. 20% reduction in water usage or 15% reduction in waiting time.

Make sure you have data about the current status or situation, to determine a baseline!

5- Set a growth goal to assess impact

Just like savings, growth is the best way to measure the expected impact and the improvement generated by the implementation of a new service or product - eg. 15% increase in ridership or 20% increase in user satisfaction.

As a rule of thumb, try to tie your pilot goals with the strategic priorities of your organization. If your organization has set a goal to decrease energy use per capita by 25%, use this as a reference to set performance metrics for all the pilots related to that specific initiative. After all, those experiments are not implemented in a void. Always try to put pilots into the context of your organization’s higher goals. This is the best source for KPIs!

Jeremy is VP of Customer Success & Operations at UrbanLeap. He’s passionate about urban innovation and urban development projects.