Catalan elections 2012: evaluating electoral forecasting

Author: Xavier Fernández-i-Marín
November 27, 2012 - 5 minutes
Catalonia Electoral forecast

I have been tempted to use “The magnitude of the tragedy” as a title for the post, quoting Quim Monzó, a well know catalan writter. Mostly because the impression, the perception, is that nobody predicted such results.

So the elections have produced a new Catalan government with less seats for the previous winning party (CiU), but substantially more seats for the two left-wing parties that also support a referendum on Catalan sovereignty -ERC and ICV-). The difference in the interpretation of the results between Spanish and International press is astonishing, with Catalan press more alinged with the interpretation of International press. See Divisive Election in Spain’s Catalonia Gives Win to Separatist Parties at the New York Times, or this editorial from the Financial Times, Barcelona’s draw, that emphasizes that two thirds of the parliament supports a referendum.


The following figure shows the results of the elections in vote share for each party (black dot and discontinuous black line), along with the prediction of the pooling the polls model and its uncertainty, as well as the point predictions and errors associated with each of the individual polls.


It is worth mentioning that, except for CiU, all parties have at least one poll that predicts their actual result within the error margin.

The model, also, is capable of guessing the tendency of the polls.

An evaluation of the behaviour of the “pooling the polls” model

As for how well it behaves the pooling the polls model, I have done some comparisons using two different measures of precision of the polls:

The function in R to compute those values is:

mnd <- function(pred, res) mean(abs(1-(pred/res)), na.rm=TRUE)
sqd <- function(pred, res) sqrt(sum(((res-pred)^2), na.rm=TRUE))

The following figure shows the values of the mean of the normalized differences for all polls with a fieldwork in the last two months before the elections, as well as the relationships between the MND and the sample size of the poll and its date of fieldwork.


The figure shows that the model does not do a very good job, or at least only slightly better than a simple average of the polls. This is because no penalization is done on larger differences.

When the sum of the squares of the differences is used, the results change substantively. The model is, by far, the best prediction. This is due to the penalization for larger differences. So the model seems to be the best overall compromise for all parties.


It is also worth mentioning that the two largest surveys (the CIS and the CEO, the two official survey bodies from Spain and Catalonia, respectively) have very different performance. The CIS does a very good job (as it is traditional), while the CEO is performing really bad. It is worth mentioning that this is the third time that the CEO does an estimation of the final vote share. So “cooking” the plain results has to be substantively improved at CEO.

It is also important to notice that as the date of the fieldwork is approaching the election date the surveys tend to be more accurate. So, once again, it does not seem to be very reasonable to have this law that does not allow to publish survey results since a week before the election day.

Translation from vote share to seats

The performance of the translation of the vote share to seats has proved to be quite good. So my fears and doubts about it have diminished, but not faded away absolutely.

To sum up

The combination of the MND and the SQD suggest that the model performs quite well for big parties and less well for small parties. Also, that the model is an evident improvement of a simple plain mean.

Let me emphasize again this point: pooling the polls does not do wizardry with the predictions. It is only a way to take a sophisticated mean of the polls. But obviously all its virtues rely only on the quality of the sources: the polls. If no polls were indicating evidence of less support for the winning party, the sophistication of the model can’t compensate for it.

Bavarian state elections 2018 - Evaluation of the forecasts

Author: Xavier Fernández-i-Marín
October 17, 2018 - 3 minutes
Comparison of the electoral results and forecasts for Bavarian elections 2018
Electoral forecast Bayesian Data visualization

Bavarian state elections 2018 - Forecasting

Author: Xavier Fernández-i-Marín
October 16, 2018 - 9 minutes
Electoral forecast for Bavarian elections 2018
Electoral forecast Bayesian Data visualization

Predicció probabilística pel resultat de la consulta d'independència del 2014 a Catalunya

Author: Xavier Fernández-i-Marín
January 13, 2013 - 7 minutes
Electoral forecast
comments powered by Disqus