Setting the right expectations for a product evaluation
Data Extraction and Automation have taken off in a significant manner in recent years. As more and more businesses realize the value of information being trapped in unorganized document formats, the segment has been getting more and more attention. The result is a whole host of products that address these challenges, using a variety of approaches, and providing a diverse experience for their clients.
This presents both an opportunity and a dilemma to organizations looking to benefit from these technologies. Their main challenge then comes down to how to evaluate these disparate tools in the most efficient manner. And right here, setting the right expectations for a product evaluation is the key.
First let's cover some pitfalls that can be avoided in setting the right expectations. And in the interest of keeping things a little interesting, let's use titles as an allegory rather than industry terms.
Sand Castles
Adding too many parts to a single evaluation exercise, is like building a sand castle. True, with enough sand, and water, you can build fairly impressive castles. But the first wave of reality will break it.
Keep complexity in terms of the number of features being evaluated to a manageable minimum. Typically, choose a maximum of about 3 primary and 5 desirable features to be evaluated. Keep everything else as a bonus delight factor. Then things can be kept simple. Increasing to more features can lead to blind spots about inter dependencies or gaps in the selection that may tend to become deal breakers later.
A good idea is to list out all the features that you want to evaluate and do an exercise as follows. Keep a maximum of 100 points that can be distributed among all features and assign the points to each feature according to priority. Ensure all stakeholders are part of this exercise. And remember that the sum of all points assigned to the set of features cannot exceed 100. You might have to iterate this process a bit to get the final numbers. Once you have it, set a threshold and pick only the features that cross the threshold.
Alchemy
Alchemy is a quest that many people embark on, trying to achieve the effect of being able to turn everything to gold. Although it is a great dream to have, it is not a good investment of your life. On the other hand, seeing the value that each element brings on its own merit sometimes gives us better value.
Businesses sometimes are tempted to set up a "golden data output" expectation. While this may be a possibility in many situations, we have to be careful to know if our particular use case lends itself to such expectations.
What does the "golden data output" expectation really mean? Any data coming out of the product must be perfect and if it has even a single flaw, we will consider it invalid. It implies that if a record has been extracted from the document and a single field is missing a single word, the entire record is discarded as erroneous.
The issue here is that if the record had 10 fields and only one was wrong, it was still 90% effective in getting the data but we used an all or nothing rule to ignore it. Value of doing 90% of the work is then lost due to a 10% miss rate. But to be really honest, there is a key factor to be taken into account here. What is the cost of detecting the 10% miss and correcting it? If this cost is higher than just doing the 100% manual extraction, then an "all or nothing" approach is the right one.
However in most cases, there are ways to detect misses or correct the issues either with partial automation using business rules or through substantially lesser manual effort than 100% manual extraction. So therein lies the engineering decision to take.
In scenarios where the layouts or structure or context is highly standardized or regulated, the all or none approach is really one worth considering in the interest of better automation.
On the other hand, where data is very diverse (varying in layout, structure, context, etc), and can have new language or terminology creeping in frequently, overall reduction of manual effort rather than 100% perfection is the more useful expectation level to set .
So the objective here is to identify whether the use case falls in the precision or effort based category.
Morning Mist
Anyone who has been to the hills in winter has seen the beautiful morning mist that covers the scenery. It's great to watch and not exciting to navigate or walk/drive through.
So ensure that the required level of clarity is already established for a use case before beginning the evaluation. In other words, a use case that has just seen the light of day for a business, may not be the best choice for a product evaluation. Instead, a midday kind of use case, that has been around long enough, for the teams working on it, to be able to easily articulate, in great detail, the challenges they face, is a better choice.
In short, a familiar and well established use case is better than one that is still in an exploration stage.
Rope Effect
Anyone who has thrown a bunch or ropes, threads or wires together would have experienced the way they tend to get tangled up in some very surprisingly convoluted ways. And the pain of having to untangle them. The way to avoid it is simple. Keep only one rope around (or as few as possible), keep it short, keep it tied up neatly.
Keep the number of use cases you want to evaluate to one ideally. This way you know for sure that the effort of dealing with it will be limited. Having more can tangle up the way the evaluation is to be assessed.
Even if more than one use case needs to be part of the evaluation, at least ensure that the features selected for all of them are more or less the same. Avoid clutter. Another aspect is the choice of metrics for the use cases. They need to be the same or else it becomes difficult to find comparability with the results. If you have more than one use case, you will also have to put additional thought into how you will compare the results from the different tests to come up with the final assessment report.
Keep your side of the effort simple, clear, uncluttered and avoid tangling up your plans in trying to load up too much into a single Trial/PoC.
Essay Writing in a Pop Quiz
Evaluations are like tests. You would not ask someone to write a thesis on quantum physics for a physics test right? Or write an essay for a Pop Quiz. You don't want to see so much information. What you want is only key points to see if the other person matches up with overall expectations .
That is what is needed with an evaluation of products too. Here we are talking about keeping the other side of the equation simple for the product vendor. Note down the key objectives you want to address, keep them simple, keep them clear, keep them straightforward and most importantly keep them SMART (Specific, Measurable, Achievable, Realistic, and Timely).
Pushing the effort of figuring these things out to the product vendor may skew things either in their favor or against them. Either way, this does not lead to a good evaluation process.
Thermometer to check Water Level
Well, the title says it all right? Use the right tool to check the right things.
We are talking about metrics here and typical first thoughts are "That's easy, it's high accuracy we are looking for, so high accuracy is what we should use". However it is not that simple. When you use accuracy as just a description for your expectations, it is different from using the mathematical term for accuracy. This one is a bit deceptively complicated and can throw you off track easily. For more details, please read this blog.
In reality, depending on business needs, the metrics that become most important may change. Explore the options available and understand how they connect to your goals. Keep things simple and clear. If something appears to be too convoluted and does not lend itself to clear interpretation and deduction, avoid it.
Woodcutters for Carpentry Work
Sometimes we think certain roles are relevant to some kinds of work by virtue of dealing with the same stuff, but the truth could be stranger than fiction.
Identifying the right people who work with the data and have the right depth of knowledge about challenges faced is the key here. Without talking about designations, people who have spent ample time going through documents to find the correct data and people who clarified the correct data from the wrong data are the right people to include. In addition, if there are internal technical personnel who have tried to address the extraction challenge, they will be the right people to point out key areas of focus from a technical perspective.
Sometimes it may seem like the right people to include are the ones who use the extracted data, i.e. the consumers of the data. This is good to an extent of creating expectations. But in order to go deeper with the evaluation, the people who have experienced the difficulty of actual extraction are the best.
Recommended by LinkedIn
The Pilotless Plane
A plane that is in the air may have all the necessary equipment to keep flying, but if the pilot got dropped off somewhere, the journey becomes a little more adventurous than we would care to ask for.
Even if you get all the ideas and plans in place and involve all the right people to get the use case properly assessed, there is still the most important aspect of getting the final decision makers involved in the process. Now, the critical thing here is to know where to get them involved, how much of their time to ask for and how deep to take them into the process. These are all the factors that will enable quick decision making once the work is completed.
Sometimes, companies follow the process of having decision makers involved in either the kick off meeting or the final evaluation meeting and the actual process itself is left to the team. This may be okay for short and simple evaluation exercises. However, for work that has a larger impact on the company as a whole, it is important that the decision makers are aware of the major checkpoints in the plan and are updated on the status and impact of the same.
Typical things to ensure a decision maker is aware of are:
The Speeding Bulldozer
In spite of having all the above points addressed, you may still find that some evaluations are engagement heavy by virtue of the challenges presented by the use case itself. In such cases, avoid setting aggressive timelines that will just force shortcuts into the process from either side and run the risk of reducing reliability of results.
Know that a heavy engagement evaluation can present unforeseen challenges and provide extra buffer time wherever needed. Such heavy work can only run at a certain pace and we have to respect that to get the best results. Even taking into account the fact that the market pressures may be high and other things are lined up in waiting, we need to remember this; a failure in this critical step can lead to higher cost of opportunity, or risk of loss later, than a potential delay in the earlier stage of evaluation.
It is very important to know what a heavy engagement means. It could be factored into complexity of the use case, number of evaluation features, complexity of metrics, the diversity of the data or just plain lack of clarity on the challenges. Any of these factors could lead to heavy engagement. Understanding and accepting these factors could save on time, cost and even prevent premature failure of evaluation.
So ask yourself these questions:
Remember there are limits in reality for what can be achieved in a given amount of time. If we ignore this, the only ones being affected are just about everyone involved or connected with such activity. Unreal expectations on timelines can be a detrimental factor in achieving proper results. And note that the opposite is also true, that if you take too long, the energy put into the work will die down and the results will suffer. So find that balance. Put across your expectations, and listen to what the vendor has to say. Then, both teams can arrive at a fairly comfortably acceptable time-frame.
Swimming on Land
Make sure you have mapped out the path from evaluation to production in a way where it makes sense. If we do evaluations in a way which does not reflect production conditions, it is like learning to swim on land and hoping you will not drown when you fall into deep water.
If you hold back on critical aspects of information or documents in the name of discretion or security, these will not get validated in the process. Deal with the security aspects in whichever way possible and keep the conversations with your vendors as transparent and expansive as possible.
Vendors who ask more questions are better than the ones who make assumptions (if they haven't dealt with this domain before). And sometimes, vendors may not be aware of what the right questions to ask are. They are not the domain experts compared to the business that is dealing with the data. So actively identify gaps in communication and address them with the vendor and document them for future engagements.
Don't avoid the difficult questions, even if you may not expect answers immediately. It is always better to leave future challenges on the table as a reference.
Getting up with a triple flip somersault
We've all seen those Kung Fu movies where the hero always gets up or sits down with a little flip and tumble. Nice to watch Jackie Chan do it but it would be quite exhausting to do in real life and quite counter productive to the elegant art of just getting up.
So it is to keep the spirit of product evaluation away from unnecessary elements that may be redundant or inconsequential in the final production run. For example, if there is meta data being collected with the documents that are being sourced for extraction, refrain from asking the vendor to extract the same information from the document itself. A rather more useful task would be to see if the product can utilize this already available metadata to speed up learning or improve prediction.
Additionally, the choice of data to be extracted in some scenarios matters. Sometimes a simple filter at export would do the job better. For example, if there is data for multiple periods to be extracted, expecting the product to know which year to select might be an overkill. Instead letting the product extract data for all the periods, and applying a filter at the export level is a simpler approach.
Knowing what is really needed and what is avoidable is the key point here.
Put on your mask first
On this little flight to success, remember to take the use case you and your team are familiar with, as the first one to use. You can keep the rest of the use cases for a later time. Once you are through with your needs, you will be in a much better position to guide other teams regarding the application of the product for their needs.
Jumping too fast too quickly will leave us too far from where we wanted to land. Keep perspective on what you are closest to and aim for the best shot.
Shopping with a Blindfold
When you are looking to identify vendors and products to evaluate, make sure to look in a bit more detail. Making assumptions that something you are looking at is very similar to something you have already seen, can lead to blind-spots. Instead, ask the vendor to tell you the differentiating factors.
Remember, when a certain problem is being addressed by different teams, a certain portion of it may end up being similar looking by virtue of the considerations taken. And the main thing that pulls up one solution against another may be hidden behind the scenes. Tell them what you have already seen and ask them if your understanding of what you are looking at is the same as you think. They may be able to shed more light on that aspect.
The Monk who drove the Ferrari
After all that build up and multiple plot lines, it is time to come to the climax of this movie. Here's where the world is saved by the incandescent hero coming to the rescue!
In the end, after all these things we needed to avoid, there is only one thing you really need to keep in the center of your cross-hairs; the actual working of the product.
So keep your focus, listen calmly, follow meticulously and have a laser focus on the road ahead. The monk who drives the Ferrari is not there thinking that he will buy or not buy it. He is just in the seat of power to engage all the features and to give his full attention to everything the car is doing. The car will tell him the decision, if he listens carefully (no, not a typo, a pun...).
The Evaluation is a test run of your production experience and the more attention you pay to what is happening, the clearer it will be as to how your future engagement is going to look like. If something breaks, watch how well the vendor responds. If nothing breaks, pay more attention to what you are watching, and see if these are the right things to watch or if something was missed.
#uq #uniqreate #productevaluation #expectations #dataextraction #evaluationmetrics #product #automation #investment #evaluate #smartgoals #productcomparison #customers #businesses #usecase #managingcomplexity #pilot #poc #vendorevaluation #roi #timeandeffort #documentdataextraction #intelligentdocumentprocessing #keepitsimple #engagement #challenges