13 Ways Wine Industry Insight advertising is safer, more effective & convenient for you CLICK HERE

FREE! Subscribe to News Fetch, THE daily wine industry briefing - Click Here

Beverage Trade Network
Beverage Trade Network

Rating The Rating Systems – The One Place Where Words Can Actually Work

This is part of a series on flaws in current wine recommendation systems originally published at Wine Industry Insights:

For much more on recommendation systems, please visit: Recommendation Insights

How you ask a question can determine the answer you get.

The concept lies at the heart of journalists, scientists, those who interview job candidates, police detectives, opinion pollsters … and wine ratings.

In many settings — especially those where the vino-cognoscenti gather — the question, “What did you think of that wine?,” will frequently elicit a number between 70 and 100.

But that number, even if everyone were calibrated and always consistent, does not answer something even more basic to the average wine consumer, “would you buy it again.”

The gulf between the two question types is even wider than the 100-point scale’s credibility and consistency gap.


Numerous scholarly article as well as those in the general media have pointed out that, even if the 100-point scale were a good numerical system, the actual ratings are so inconsistent among the best experts that the numbers expounded are mostly worthless.

The best round-up of these flaws was recently published in The Guardian: “Wine-tasting: it’s junk science” which notes that, “Every year Robert Hodgson selects the finest wines from his small California winery and puts them into competitions around the state.

“And in most years, the results are surprisingly inconsistent: some whites rated as gold medallists in one contest do badly in another. Reds adored by some panels are dismissed by others. Over the decades Hodgson, a softly spoken retired oceanographer, became curious. Judging wines is by its nature subjective, but the awards appeared to be handed out at random.”

Hodgson’s most recent study (as are his oprevious ones) was published in the Cambridge University Press’s scholarly publication, Journal of Wine Economics An Examination of Judge Reliability at a major U.S. Wine Competition


The 100-point wine-rating scale also carries psychological baggage from school where below 70 is an F and 100 is an A+.

Those rating/grade associations solidify the false perception that a rating is an objective, unassailable judgement on quality.

Beyond that, what is the “meaning” of an 83 versus 85 or 87? Is the “meaning” biased by personal expectations? Grades in school or performance at work?? What a wine “deserves?”


Alternative scales — the substitution of 10 or 5 point scales as well as stars and other icons — have found widespread acceptance in wine as well as in other consumer products. But they also have psychological biases, possibilities for misinterpretation (“what does it mean?”) .

Research shows that when confronted with an odd-numbered scale, respondents tend to cluster around the neutral point with a bias to the positive. This reflects potential anxiety over extreme positions and a tendency to avoid those.


Odd-numbered scales allow for the expression of neutral opinions because if offers a middle point — neither negative or positive — that amounts to “undecided.”

Even-numbered scales force a choice into negative or positive territory because they have no “undecided” position.

Offering neutral or “undecided” positions serves a purpose in political polls and other surveys. But wishy-washy has little value in the market place where the decision is “buy or not buy?”

Thus, scales with an even number of choices provide the best, most valuable actionable data in a purchase decision environment.


The expression of personal preference in purchase decisions can be most accurately measured by a four-point, semantic scale involving common consumer actions.

For example:

  • I would serve this for a special occasion.
  • I would buy this again
  • I would not buy this again.
  • I would tell friends not to buy this.

The word choices reflect easily understood consumer purchasing and recommendation actions: Buy (or not) recommend (or not).

These sorts of distinctions between rating and buying can also be found in politics and elections where approval ratings for an elected official are less than 50% and yet a majority of the same people polled would vote for (buy) that candidate again.


In the quest for solid purchase information, the existence of expert ratings which are provably inconsistent, is of little help.  However, data that indicates that people would buy the product again or recommend it to friends (or not) offers helpful, actionable data.

The need, therefore, is the system that can accurately gather and dispense that actionable buying information.