Artificial Intelligence: Hiding in Plain Sight

Before we can talk about where we find artificial intelligence (AI), we need to define the term “AI” itself. This turns out to be more difficult than it first seems. Most of us would agree that we can invoke the term AI with something less than robots that can propel themselves around the world and discuss philosophical questions with us. Beyond this, it appears that no clear consensus exists.

A reporter went to Alphabet (formerly Google), the epicentre of AI, and asked people working there for an explanation. Here are some of the answers:

 “I would definitely interview someone else.”

“No thanks. Sorry. Good luck.”

“I don’t know. I’ll pass.”

 “It’s machine learning.”

 “I work at Yahoo. . .”

In the very first chapter of Artificial Intelligence Marketing and Predicting Consumer Choice, I propose a working, if not comprehensive, definition. AI broadly means anything a machine does to respond to its environment to maximize its chances of success. The machines we use are computers and we set their goal as detecting complex patterns we cannot, to aid our making better decisions. AI is not having a robot that does all our work for us (sorry, not yet); humans still need to decide how to use the information the machines provide.

There is no question that AI and machine learning overlap substantially. This of course raises the question of what machine learning means.

This definition also varies depending on who you ask. An online article purporting to teach about machine learning included regression and clustering among advanced machine learning methods. These are two of the most august and long-standing of analytical approaches. Regression was widely used well before computers existed.

Even taking a less expansive definition, machine learning has been with us for decades. It has been working in the background as often as in the foreground, solving problems that would have been impossible to approach without it. And indeed, artificial intelligence, pure and unadulterated, has entered into our daily work more than we suspect.

For instance, classification tree analysis routinely uses algorithms for testing statistical significance, which are so sophisticated they have been classed as artificial intelligence. Classification trees, as a reminder, repeatedly split a sample into smaller subgroups. At each splitting point, scores on some predictor variable lead to groups with higher and lower scores on some target or outcome variable.

To make this more concrete, suppose you wanted to know which households, based on demographic characteristics, consumed the most of a particular breakfast cereal. You have consumption levels for several hundred thousand households, and 46 demographic characteristics of these households, including factors like location, number and ages of children, household income, ethnicity, and so on. (You can readily buy hundreds of demographic items like this for every household, at least in the UK and US, and add these to any data set.)

The classification tree routine would look at all these variables and the ways they could be used to split the sample into smaller groups. Suppose that overall, households’ averaged 21 boxes of this cereal. The tree procedure, after scanning all possible variables, would then return the answer that type of town is the best variable for finding differences—and further, that if you put everybody living in cities and rural areas into one group and those in the suburbs in another, you have the strongest contrast. Those in the suburbs average 28 boxes a year and those elsewhere, 16. That is, the suburban group consumes some 175% as much as the others.

At first, reaching this answer may not seem like anything impressive. After all, there are only four different ways to arrange three groups into different configurations. (That is, using, the initials, we have s vs. c and r combined, c vs. s and r combined, s vs. c and r combined, and s. vs. c. vs. r.) However, when we consider that all the other variables were being scanned also—and that these included some like number of children, which could be anywhere from one to 15—the number of possible solutions become astronomical.

It also emerges that it is a fiendishly difficult task to compare the significance of differences for variables that can be split in many ways and variables that can be split in just one or a few. (A variable that can be split in just one way would be, for instance, whether the person was retired or not.)

The difficulty of doing these comparisons completely sunk the earliest form of classification tree analysis, AID. It always picked the variables that could be split in more ways as the best. This weakness was so salient that for years after the newer algorithms solved this problem, classification trees were often dismissed as a weak and unreliable method.

The first really strong new algorithm was presented at a conference on artificial intelligence in 1991. Since then, more advances in intelligent tree construction have made these methods stronger still.

Other already powerful methods have been enhanced by machine learning approaches. For instance, discrete choice modeling and conjoint analysis have been substantially extended by use of a machine learning method called Hierarchical Bayesian Analysis. This is a highly computer-intensive application, that still can require hours to reach a solution. (As a reminder, these methods allow us to forecast responses to hundreds or thousands of alternative products, services, or messages.)

We now can solve much larger problems and in more detail than was ever possible before. We explain how this happens in Artificial Intelligence Marketing and Predicting Consumer Choice. All these methods, and others described in the book, show that artificial intelligence and/or machine learning have been serving useful assistive roles in discerning patterns in data, so that we can make better strategic decisions. These day-to-day uses, where we define problems and set the scope of investigation and then let the computers do the heavy lifting, could provide the best template for further developments in applying artificial intelligence.