Giving AIs the (pattern) recognition that they deserve.

How do AI’s help us when they find patterns in massive datasets and how do these patterns actually help our businesses? Clarifying this helps us to design better AIs and to use them more successfully.

The job of an AI is to find useful patterns

AIs just find patterns, that’s it. We call them “models” because they are simplified models of whatever the dataset is describing, e.g. customer buying patterns, faces to recognise or human speech to understand.

The number of new data types, tools, Machine Learning techniques and AI applications are multiplying very fast. But it’s all about finding useful patterns.

One type of pattern connects many inputs with many outputs, e.g. the pattern describing the types of customers who tend to buy a lot of cheese. This is found using Supervised Learning techniques. Another type of pattern organises individual elements into higher level groups, e.g. a market full of customers grouped into customer segments. This is found using Unsupervised Learning techniques. A third main type is Reinforcement Learning. But this is more about using trial and error to find and improve a pattern rather than the found pattern itself.

Why do we need AIs?

This question may seem a bit weird since we’ve got AIs everywhere and many more on the way. But the more clarity you have about the purpose of an AI project then the better you will be at designing AI models and then utilising them.

The bottom line is that we need AIs because humans have very limited information processing capacity. The Nobel Prize winner Herbert Simon called it Bounded Rationality. Bounded Rationality is the limit on our brain’s ability to get, store and process information.

And these limits are naturally passed on to the organisation that we build. Companies, governments, social networks and society as a whole all have limits on the information that they can handle, and the speed with which they do it.

Even though human organisations are themselves constructed as an attempt to push back these limits.

AI-generated patterns help us to organise massive datasets

So, if humans and the firm which we work in have “information overload” limits then how do AI generated patterns help with that?

The answer is that these patterns are just ways of organising massive datasets, so that humans can handle them and use them.

For example, customer loyalty datasets hold data which describes what millions of customers buy every day. Popular websites, like Amazon, can potentially record every click, keystroke and mouse move from an even greater number of people.

Finding which customers to target with specific offers or interesting product recommendations would be impossible without a pattern to guide us. A pattern is a “short cut” which connects specific offers or interesting products, with the people who have shown some interest in them in the past. Or a pattern that classifies groups of people to support the prioritisation of resources.

How does a pattern actually help?

Patterns are “cheat sheets”.

They are “bridges” which link the data in incredibly complicated datasets with insights that we want to find.

Limitations on human information processing mean that we have to break large problems into smaller chunks and tackle them one chunk at a time.

Our minds need problems (in this case, searching through datasets) to be organised hierarchically. Hierarchies are tree structures, like a firm’s hierarchy of job titles and positions. Hierarchies are the “knives” which cut big problems into smaller chunks.

Modularity and object-oriented programming both use hierarchies to break large problems into different levels, with loosely coupled chunks on each level. Each “chunk” is a module with some connection to the other modules, but with enough looseness allow each module to be usefully looked at in isolation.

People need different things from a dataset, so there are many different patterns in a single dataset. Which is why its important to get your analytical objectives very clear at the start of any project.

Patterns are just the rules for dividing a problem up into loosely coupled modules. A pattern for a massive dataset is like an index or an address book of where to find what you need. And each pattern comes from finding the underlying structures in the problem.

I used to teach modular hierarchies by talking about carving up a roast chicken for dinner. The underlying structure of the chicken’s joints makes it easy to carve it up in a certain way. Follow the pattern of the joints, don’t try to saw through the bones.

But one MBA student was an expert in testing chickens. It turns out that if your objective is testing a chicken for diseases rather than eating it for dinner, then sawing through the bones is a better way to divide up the chicken.

So, think of the AI’s job as finding patterns, think of a pattern as one “cheat sheet” for a dataset and use your analytical objectives to make sure that you get the right “cheat sheet”.

Designing better AIs and using them more successfully

There are two implications from understanding how patterns help our limited human brains to use massively complicated datasets. They help us to design and use the AI’s which find the patterns. And to design and use the patterns themselves.

First, start by thinking about the desired pattern, not the algorithm, because the Machine Learning algorithm is the means not the end. If a pattern’s role is to follow an underlying structure in a dataset, to simplify that data (but still keep it relevant), then think about all the different structural arrangements in that dataset. Which are the most relevant to your project? Consider the trade-offs between relevance, accuracy and simplicity. Do not get side-tracked (or even fooled) by non-relevant patterns.

Second, think about the process of searching for patterns. If an AI’s role is to find patterns, then think about how it goes about looking for and testing potential patterns. And think about maintaining these patterns and the associated model retraining costs. Given your understanding of what the desired pattern might look like, what is the most efficient and effective AI technology to use to search for it? What sort of dataset might record the real-life phenomena that you are interested in?

Most business leaders and MBAs will employ a specialist AI firm like Filament to use world-class AI tech from Google, IBM Watson and other sources. But it’s worth thinking through the Qualitative Strategy of what you are trying to accomplish, and how it all works together.

Duncan is a lecturer at Nottingham University Business School. He also advises organisations on creating value with digital data and he writes in his own blog.

Connect with me on LinkedIn at www.linkedin.com/in/duncan-r-shaw-7717538.