Encoding Likelihoods


To use Bayes' theorem we need two things: prior probabilities and likelihoods. "Likelihoods" is the name given to the reliability statistics of the source of additional information. In other words, the likelihoods tell us how likely it is for a forecast or indicator to be correct or incorrect. It is based on the track record of the information source on previous forecasts. If in the past the source correctly predicted a given event 70% of the time, then the probability of correctly forecasting the next instance of that event will be assumed to be 70%, the same as that of its track record. That makes sense.

What You Need to Know: Cost and Likelihoods
To illustrate how to encode likelihoods for use in Bayes' theorem, let's go back to Goldie Lockes and ACME's decision problem. Lockes knows that she can improve on the accuracy of her prior probability assessments regarding market demand for roadrunner traps were she to have access to reliable market information. So Wolfgang Cactus suggested they call Beep-Beep Associates, the renowned marketing consultants. BBA claimed that if there was a demand for the traps, their research would surely detect it. But Lockes knew all about these much ballyhooed "sure" events and requested more precise (i.e., quantitative) information about the reliability of BBA's research.
 
BBA cheerfully complied. According to their experience, on previous occasions when market demand for newfangled products turned out to be medium, they forecasted it correctly 80% of the time. The incorrect forecasts calling for high demand and low demand were split just about evenly. When market demand was actually high BBA predicted it 70% of the time, while incorrect medium-demand forecasts outnumbered low-demand forecasts by a margin of two to one. But forecasting low demand is trickier, according to the analysts at BBA: market signals are harder to interpret correctly. In the few cases they were consulted under low-demand market conditions, they were correct only half of the time. When asked about the other half, BBA estimated a slight advantage for erroneous medium-demand forecasts, but the number of consulting engagements was too small to provide a reliable reckoning. Lockes then inquired about the cost of conducting a market study for ACME. $400,000 was BBA's reply.
 
Should ACME reject BBA's offer?
BBA gave Lockes the two things she needs to determine if the market study would be of value to ACME: the cost of the study and its quality (or reliability) indicators. The first thing Lockes must decide is whether or not to reject  BBA's proposal due to a lack of cost-effectiveness. This can easily be done by comparing the cost of the market study to the expected value of perfect information (EVPI), which is $1.5 million. The EVPI, remember, means that no information —irrespective of its quality— is worth more than $1.5 million to ACME. Since BBA's quote is for $0.4 million, their proposal should be taken into consideration. But that is all Lockes can say at this point: it is still too early to tell if BBA should be hired. In order to determine that, she would have to assess if the (imperfect) information BBA provides is worth at least $0.4 million to ACME. This is where Bayes' Theorem comes in. The EVPI test serves only to show that Lockes should not disqualify BBA's proposal outright. But she must do further analysis to ascertain if they should be contracted.
 
Encoding Likelihoods
Since Lockes knows BBA's track record in newfangled-product demand forecasting and, moreover, believes it can serve as a reasonable indicator of the likelihood of a successful forecast for ACME's present problem, she can use Bayes' Theorem to revise her prior probabilities. If, for some reason, the assumption that the historical record can serve as a likelihood indicator for future events cannot be sustained, Bayesian revision is not justified. Revisor beware.

In order to use Bayes' Theorem, Lockes needs two things: her prior probability distribution and the reliability indicators (likelihoods) of BBA's previous market studies. Now, for the set of states of nature S = {H, M, W }, the probabilities Cactus and Locke had agreed upon are:
P( H ) = 0.1          P( M ) = 0.6          P( W ) = 0.3
which is her prior probability distribution. To encode the likelihoods, she must carefully analyze each statement about BBA's forecasting record.
 
1.  "When market demand for newfangled products turned out to be medium, BBA forecasted it correctly 80% of the time. The incorrect forecasts calling for high demand and low demand were split just about evenly." Notice that the given condition «actual demand is medium» holds for all three forecasting events. Consequently:
P( FM | M ) = 0.8          P( FH | M ) = 0.1          P( FW | M ) = 0.1
where FX  stands for the event "forecast predicts state of nature X." Since the sum of the conditional probabilities over the given event must equal 1 and the proportion of incorrect forecasts was "just about" evenly split, the last two probabilities must be 0.1 each.
 
2.  "When market demand was actually high BBA predicted it 70% of the time, while incorrect medium-demand forecasts outnumbered low-demand forecasts by a margin of two to one." The given condition now is «actual demand is high». Thus:
P( FH | H ) = 0.7          P( FM | H ) = 0.2          P( FW | H ) = 0.1
3.  "Under low-demand market conditions, BBA was correct only half of the time. When asked about the other half, BBA estimated a slight advantage for erroneous medium-demand forecasts, but the number of consulting engagements was too small to provide a reliable reckoning." Up to now, the likelihoods were based on factually derived (statistically observed) conditional probabilities. Remember, in general, if objective probabilities are available, one should seriously consider using them. This is especially important when revising subjective probability assessments, for it adds an element of hard-nosed objectivity to personal assessments. However, if no objective probabilities are available, remember Laplace's dictum: Go Bayesian! It is preferable to rely on subjective probabilities than to have no probabilities at all. Lockes interpreted BBA's assertion "about the other half ... a slight advantage for erroneous medium-demand forecasts... " as meaning "a proportion of three to two." Therefore:
P( FW | W ) = 0.5          P( FM | W ) = 0.3          P( FH | W ) = 0.2
Comments: Why not interpret the "slight advantage" as, say, a proportion of 2.6 : 2.4 instead of 3:2, one might ask. The short answer is, because Lockes has no basis on which to justify such a degree of precision. The obligatory retort would be: Why then not 2.5001 : 2.4999 or any other even finer discrimination? Unless Lockes had sufficient reason to extend the precision of her assessment, she is not warranted in doing so. She would be modeling the problem incorrectly by injecting unsupportable assumptions. In general, when dealing with subjective probabilities, be wary of going beyond one decimal digit in your appraisals. Do as the weather forecasters, who normally keep their predictions to increments of 10% (hurricane advisories excepted). Of course, some assesments like 0.75 and 0.3333 are entirely appropriate because they refer to simple fractions, 3/4 and 1/3, respectively, which are intuitively familiar quantities to most people. The rule is: avoid spurious precision. At the other end of the argument, it should be clear that a proportion of 4:1 instead of 3:2 can no longer be deemed to be "slight."


With priors and likelihoods properly encoded, one can proceed to Bayesian revision. Direct use of the formula is not recommended for error-prone, carbon-based life forms laboring with pencil and paper. (Silicon-based reasoning architectures are free to do as they wish.) A more enlightened form of manual computation is discussed next.

Make a Free Website with Yola.