The classic Business Intelligence tool presents a summary view of data along many dimensions to the user, but stops short of suggesting the best course of action. The next generation of the data-driven economy elevates the art to a new level, where predictive models automatically make optimal choices on behalf of the user.
The most commonly known approach to Predictive Modeling is linear regression, wherein a prediction is made from one or more predictor variables weighted by constant coefficients. This is known as the Maximum Likelihood approach and has several downsides. Often in business, it is not sufficient to answer what’s the best prediction, but also how much better is this prediction compared to an alternative prediction? For example, if a coin is biased, it is more important to find out the chances of heads vs. tails than simply calling heads or tails for the next flip. In other words, the complete predictive model should yield a distribution of predictions, not a single answer.
This brings us to the famous Bayesian statistics.
In the Bayesian philosophy, all events are governed by some underlying statistical model. The goal of the inferencing process is to determine which of many possible models is the most accurate by drawing samples from experiments. Keep in mind that the model itself is invisible, so only experimental data can be observed directly.
Likelihood
Prior
Posterior
Evidence
This innocuous-looking formula triggered an entire field of research that consumed some of the best minds for centuries with the single intent of solving for the posterior either formulaically, numerically, or by means of computer simulations. With the posterior distribution solved, we can make predictions and derive any other metric of interest, such as the expectation value of some indicator and the margin of error of any given predictions.
From a practical point of view, the business advantage of adopting the Bayesian approach is plainly obvious: the model keeps getting better as one gets more feedback.
With each iteration, the posterior reflects the real-world situation more closely and leads to more accurate predictions of the outcome.
Having addressed the need to consider individual preferences, we face the next hurdle in that a single consumer is still not of a consistent mindset when it comes to making a purchase. The marketing concept of a Funnel State captures how a consumer moves from unaware, to uninterested, to actively searching, to comparison shopping, and finally to conversion. Successful messaging to individual consumers must take into account the state of buyer readiness which, of course, is not directly observable. All marketing automation systems have this type of mechanism built in, but they depend on human judgment supported by manual scoring.
The branch of Machine Learning devoted to the detection of hidden states is called Hidden Markov Model (HMM). The accurate estimation of a consumer’s psychological state is a powerful way of determining the Next Best Action (NBA), where the seller offers the most optimal stimulus (ads, coupons, brochures, etc.) to reach a consumer based on their mental state in addition to the overall profile. In fact, the idea of taking proactive actions to affect the outcome moves the bar beyond Predictive Modeling to Prescriptive Modeling.
Advanced HMM involving this type of stimulus-driven state transition is called Non-Homogeneous HMM. This level of complexity seems foreboding, but is very much within the capability of the MCMC technique. The combination of HMM and HBM provides the ultimate Fine-Grained Personalization solution for Marketing efforts.