FREE! Subscribe to News Fetch, THE daily wine industry briefing - Click Here

2019 Unified Wine and Grape Symposium

How to tell if a survey is credible or just fake data

Surveys are a dime a dozen. There’s even an app for that (Survey Monkey).

However, most surveys cannot be trusted because they fail to meet standards for transparency.


Standards for assessing data validity (see below for links) require that a study reveal:

  • The number of subjects in a study,
  • How the were selected,
  • How the questions were asked,
  • The exact wording of the questions,
  • When and where the questions were asked.

All of those affect the validity of the results and should be available whenever results are presented as valid.

If a study does not publish that information along with its results, there is no way to tell if it’s fake data or not.

Trustworthy survey: case in study

Screen Shot 2017-07-05 at 7.49.05 AM

Properly done polls — such as this recent one from Gallup always — ALWAYS — include the following specific information. Polls which do not include this should not be trusted.

Survey Methods

“Results for this Gallup poll are based on telephone interviews conducted June 7-11, 2017, with a random sample of 1,009 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia. For results based on the total sample of national adults, the margin of sampling error is ±4 percentage points at the 95% confidence level. All reported margins of sampling error include computed design effects for weighting.

Each sample of national adults includes a minimum quota of 70% cellphone respondents and 30% landline respondents, with additional minimum quotas by time zone within region. Landline and cellular telephone numbers are selected using random-digit-dial methods.”

In addition, trustworthy surveys, will include details allowing a close examination for validity: Gallup: View survey methodology, complete question responses and trends.

Again, surveys that do not include that degree of detail are always suspect.

Self-selected participant surveys among condemned practices

Screen Shot 2017-07-05 at 7.49.45 AM

Condemned Survey Practices — This link is from the American Association for Public Opinion Research (AAPOR) which sets basic standards for surveys.

The following link notes the following AAPOR-condemned practice that is the foundation of the Eaze poll:

“4. Representing the results of a self-selected “poll” as if they were the outcome of legitimate research.

Write-in, call-in, and interactive polls have become increasingly common. These “polls” report the opinions of only those people who called in, and not those of the general public. AAPOR believes that any publicizing or promotion of such activities not only damages legitimate market and survey research, but can be very misleading when used to influence public policy or simply to disseminate information about the general public.”

Other sources that should be taken into account when examining the validity of poll data: