And if these polls rely on party weighting to achieve the level of accuracy that they do, they might be even less effective during a primary season, for instance, when everyone is in the same party, or in a changing partisan environment, when public polling is most valuable. In recent weeks, the online pollsters have shown widely divergent results on the Democratic race.
Either way, it is a plain step back from the historical level of accuracy of the so-called gold standard polls that have adhered to traditional best practices. This is not an exhaustive list, but the pollsters that have used those practices include ABC/Washington Post, CBS/NYT, NBC/WSJ, Pew Research, the Selzer poll, Monmouth University or the Upshot/Siena polls. Those pollsters, typically with brand names, have had an average error, combined, of 3.5 points in nonpresidential general elections since 1998.
But traditional live-interview polls have also had struggles in recent cycles, particularly in state polling. In 2018, for instance, live-interview surveys had an average error of four points. That’s better than the online polls, on average, but not by so much. (FiveThirtyEight has a listing of all polls and whether they’re online-only or not.)
What does it mean for poll watchers?
In general, the simple solution for these unproven online polls is to average them, rather than pick apart the methodological details of individual surveys. But in some cases, people will have reason to consider the merits of a single poll.
One important consideration is what kind of error is tolerable for the question at hand.
It probably doesn’t make a huge difference, for instance, whether 60 percent or 64 percent of Americans think the country is on the right track. A five-plus-point survey error could be quite acceptable for that purpose.
But a five-plus-point error might be intolerably high for many kinds of election analyses, particularly in close races. Here it’s important to focus on an average, or at least on higher-quality polls.
What helps distinguish the higher-quality online polls? In general, it is reasonable to look for a transparent methodology with signs of a more sophisticated weighting or sampling approach, like YouGov’s synthetic sampling, rather than a simple draw of nonprobability sample weighted by census demographics. It is also reasonable to look for a sample drawn from a panel with diverse sources, rather than from a single place on the internet. But this will not be easy for most poll consumers to discern, and even the experts struggle to make sense of it.