By Simon Firth, July 2006
Predicting the future isn't just the province of fortune
tellers or crystal-ball gazers. A team of researchers
at HP Labs is doing just that – with the tools
of economics.
Labs’ most advanced project in this area is called BRAIN.
“That stands for Behaviorally Robust Aggregation of Information in Networks,” says
researcher Leslie Fine, an experimental economist at HP Labs and part of a team
that is using laboratory experiments with human subjects to evaluate economic
propositions. The program, founded in 1994 by researcher Kay-Yut Chen, has been
nationally recognized in applying this kind of research to the problems faced
by corporations.
Fine, Chen and HP Senior Fellow Bernardo Huberman developed BRAIN to tackle
a problem that troubles the corporate world – how to extract accurate
information about future events (such as predicted quarterly revenues or
expected demand for a product) from small teams of knowledgeable workers.
 |
|
In the wider world the best mechanism for predicting
the future is to run a market. When large numbers of
people speculate on stocks or commodities, for example,
their prices reflect everything that is currently known
about how they might perform in the future.
The University of Iowa’s famous Iowa Electronics
Market has shown that large groups of people can also
predict electoral results and the future direction of
major economic indicators with almost unrivaled accuracy.
Markets don't work well with small groups of people,
though. It’s not that such groups aren’t
knowledgeable, but outcomes from small markets can easily
be skewed thanks to the low number of players offering
bids, or through deliberate manipulation.
But without markets, the decisions made by small groups
are equally open to manipulation. Take an IT procurement
team trying to predict the future price of a key component.
Typically, Fine explains, they’ll do this by “sitting
around a room once a month and yelling at each other
until a number comes out.”
As a result, she says, “he who yells loudest wins.” Quieter
but more knowledgeable team members -- or team members
too intimidated to contradict the boss -- don’t
get heard and the predictions suffer as a result.
BRAIN gets around these problems in three ways: It makes
the prediction process anonymous; it asks people to back
up their predictions with real money -- essentially,
to make bets on where they think the numbers will land;
and it makes the whole exercise a game.
If you are asking people to make bets, you need to know
their appetite for risk in order to makes sense of the
bets they place.
So BRAIN first has the team play a market game which
requires them to put stakes on various imaginary outcomes.
From that, researchers calculate a behavioral coefficient
for each player that summarizes their risk attitude and
their predictive power.
Then players are asked to make bets on the real question
in play (such as 'how many handheld devices will we sell
this month?'). According to how accurate their bets turn
out to be, the players receive a payoff -- typically
in tens rather than hundred of dollars.
Meanwhile, says Fine, “we weight the bets by the
coefficients we have for each player, aggregate them,
and pass back the prediction.”
Such a mechanism, it turns out, works extremely well.
Because it gets people to divulge what they know, BRAIN’s
predictions consistently beat forecasts generated in
more traditional ways. It helps that people generally
enjoy playing BRAIN, too. It means they'll go back and
use the tool again.
The work that went into BRAIN also suggests ways to
mitigate against other business problems -- such as sandbagging
-- where individuals underestimate their future performance
so as not to lose out on their pre-agreed bonuses.
In the face of that, says Chen, a manager could simply
pay a bonus to people who made accurate forecasts of
their performance. But offer such a deal to some people,
he adds, “and you'll get a very accurate forecast,
but they won't sell anything!”
To solve this, researchers have proposed presenting
salespeople with the option of receiving more of their
pay as fixed compensation and less as bonus, or vice
versa.
“So then you say to the salesperson,” says
Huberman, “How would you like to get paid?”
“If he knows he’s going to sell everything,” Huberman
suggests, “he’ll want all bonus. If he knows
he won’t sell anything, he’ll want to bring
a check home. So by the way he chooses, you can figure
a probability that he will actually sell a certain number
of pieces.”
As a result, the boss knows the true sales outlook and
the salesperson is still well compensated for being good
at his or her job.
Huberman and Chen were able to successfully test this
idea in their experimental economics lab, where people
played a series of games that modeled such a sales situation.
It’s unusual for a corporation to have such a
research facility, but very useful.
Several groups within HP have already engaged Chen to
design games to help them reformulate the contracts they
sign to realign them in the interests of both signatory
parties.
“Contracts are a bit like the rules of a game,” says
Chen. “If you change the rules of the game, people
will play differently. And once you understand how, you
can write better contracts.”
BRAIN has shown that economic games have predictive
power outside the lab, says Fine, who ran a successful
field test with a financing forecast team in HP Services.
“At the beginning of each month, we would predict
operating profits and revenues for the end of that month,” she
says. “And we blew their numbers out of the water.”
Fine is running a pilot of BRAIN with its first external
customer. The service is also attracting interest from
major corporate players in communications, software and
energy exploration.
BRAIN can be useful, says Fine, “any place you
have a small team that has some kind of specialized knowledge
that you want to tap, and where you can take what you
want to know and rephrase it as a bet.”
Shorter time scales work best. “No one wants to
place bets on things they’re going to find out
about in ten years,” Fine notes. It’s also
important, she says, that “there’s no ability
to lose money here. So we’re not gambling.”
"This effort," says Huberman, "is really
about harvesting implicit knowledge in organizations." He
has pioneered mechanisms for identifying the best set
of people to poll for such knowledge -- groups that may
not always echo formal organization charts.
Researchers have developed other tools for prediction
as well. One technique is aimed at helping risk-averse
business managers make decisions and another helps businesses
better predict demand. (See 'Afraid
to make risky business decisions?'
and 'Riding
the peaks'.)
Tools like these are growing increasingly important,
Huberman argues, as we're faced with burgeoning amounts
of information.
“It’s very hard to attend to what is important
and what is salient or not," he says. "This
offers us a way to know.”
Simon Firth is a writer and television producer living
in Silicon Valley.
|
Printable version
_files/s.gif){kind=small}
