Back

Note: This article represents the view of the author and not the University of Hong Kong.

Followed by the final stage of the Legislative Council Election, the pre-election rolling surveys conducted by the author have almost come to an end as well. The general public and the media should have already had some knowledge on these surveys. However, according to the observations of the author, some scholars, experts, and even those in charge of the policy research, still have some misunderstandings towards them. Thus, on the last day of the release of this year's Legislative Council Election rolling survey result, the author would like to introduce some basic knowledge of rolling polls to all, so as to contribute to the media and civic education. Due to the space limitation, this article will concentrate on addressing several common misunderstandings.

Sample size of rolling poll

When the result of the rolling survey was first released on August 8 this year, the author has already openly explained that around 334 cases with the registered voters was expected to be completed every day, which meant over 1,000 successful cases would be accumulated in 3 days to form a complete territory-wide survey sample size. Figures in individual constituencies would only roll once every 5 days, and the sample size would gradually increase when the election was approaching.

The real situation was that, on August 4, the first day of the survey, 374 cases have been completed. The sample size has been gradually increased, and on September 9, the last day of the survey, we have completed 944 cases on one day. The whole survey process has lasted for 37 days and the daily sample size has increased for 2.5 times, while a total of 16,785 successful cases have been accumulated. Our interviewers and colleagues have paid painstaking efforts in each case.

Some criticisms suggested that our sample size was too small and the proportion of old figures was too high, thus we needed to aggregate data to increase the sample size. Rather than going into the details of the concerned viewpoints, let me introduce a foreign example here. In the United Kingdom's 2001 General Election, the University of Essex, which is famous of its election study, was responsible for the design and the execution of the British Election Study. The Study has a long history of 40 years and is viewed by the academics with great importance. During the 30-day election period, the research team selected 150 voters by random sampling to conduct telephone interviews each day. Figures were rolled on a 7-day basis and released through the internet on a daily basis, with over 1,000 people in each sample size. In the next General Election, the University of Essex plans to increase the daily sample size to 200 cases, which means a total of 6,000 people will be interviewed in 30 days. Compared with the top research in the United Kingdom's academic sector, our rolling surveys conducted since 1995 are really so poor?

The question here is that, excluding those criticisms with hidden political agenda and ungrounded criticisms, some local experts and directors who have participated in the discussion seemingly do not have a good understanding on the sampling error of the survey.

Correct use of sampling error

In some people's view, the sampling error for 1,000 cases is plus/minus 3 percentage points, 600 cases is plus/minus 4 percentage points and 300 cases is plus/minus 6 percentage points. Basing on this, if in a 300-case sample, a certain candidate's support rating is 5%, his actual support rating should then lie from a negative value to 11%? This is a serious misunderstanding.

In fact, under the 95% confidence level, the formula for calculating sampling error is 1.96 √[(p(1-p)*(N-n))/(n*(N-1))]. In this formula, p represents the percentage of a certain survey figure, N is the size of the target population and n is the sample size. The magic here is that, while the sample size remains constant, the smaller the p, the smaller the error is as well. The author would like to illustrate this with Table 1, while assuming that N is extremely large.

Table 1: Changes in sampling error

Sample size	Percentage (p value) together with sampling error
1,000	5% +/- 1.4%	20% +/- 2.5%	50% +/- 3.1%	80% +/- 2.5%	95% +/- 1.4%
800	5% +/- 1.5%	20% +/- 2.8%	50% +/- 3.5%	80% +/- 2.8%	95% +/- 1.5%
500	5% +/- 1.9%	20% +/- 3.5%	50% +/- 4.4%	80% +/- 3.5%	95% +/- 1.9%
300	5% +/- 2.5%	20% +/- 4.5%	50% +/- 5.7%	80% +/- 4.5%	95% +/- 2.5%
100	5% +/- 4.3%	20% +/- 7.8%	50% +/- 9.8%	80% +/- 7.8%	95% +/- 4.3%

The table reveals that, even if the sample size falls to 300, the sampling error of a candidate with a support rating of 5% in fact lies between 2.5% and 7.5%. Therefore, if one says the sampling error for 300 cases is plus/minus 6 percentage points, in fact it only refers to the largest error value when p=50%. While the concept of "margin of error" is commonly used by the Western media, it is in fact a simplified concept and only the approximation of the largest error is released always. When applying to rather large or small percentages, appropriate adjustments are necessary. Many commentators only know one side, but not the other side of the story.

When one has understood this, many criticisms will not be able to stand automatically. Whether the sample size is large enough depends on whether the survey result can accurately differentiate the two groups of figures, or the changes of figures within the same group in different surveys. The sample size of the international public opinion surveys generally will exceed 1000 cases. But when comes to cross-tabulation analysis with the demographics, comparison among sub-samples of smaller size will be necessary. Sometimes, only around 100 or 200 basic analyzing units are involved.

Concept of overlapping sample

Some critics suggested that the proportion of old figures was too high in our rolling survey and thus we were just repeating figures. For easy explanation, we can use the real figures in Table 2. According to the table, basing on a 3-day rolling basis, 2 days' figures, i.e. figures on September 2 and 3, have been overlapped in Sample 2 and Sample 1. By the same token, 2 days' figures have been overlapped in Sample 3 and Sample 2, and so on. Reader X, who does not love reading repeated figures, in fact can choose to read the results of Sample 1 and Sample 4 only, since there is completely no overlap between them. However, another Reader Y who does not love reading repeated figures can choose to read Sample 2 and Sample 5, since there is also no overlap between the two samples. Similarly, Reader Z can choose to read Sample 3 and Sample 6 only. Readers X, Y and Z can all say, in fact there are only two independent analyses in the 6 samples while the other 4 are just products of repeated figures. Yet, what is special here is that, all 3 readers can each view two different scenarios of "reality". While this has met the requirements of a large sample, figures have not overlapped.

However, based on the "Proportion of Opinion A", both Reader X and Y can view that the figures have increased by one percentage point, while Reader Z thinks there is no change in the figure and it remains at 59%. In fact, the figure has experienced a fluctuation and resumed to its original position, while Z cannot see that. Based on the "Proportion of Opinion B", Reader X can see no change and the figure stays at 83%. Reader Y can see that the figure has experienced a rise of one percentage point, while Reader Z can see a rise of three percentage points. This reveals that, even based on the non-repeated samples, more than two sets of observations can be obtained from the 6-day rolling surveys. This is the preciousness of the rolling poll: differences can only be detected through careful observation.

Table 2: Combination of rolling samples

Sample number	1	2	3	4	5	6
Date of survey	1-3/9	2-4/9	3-5/9	4-6/9	5-7/9	6-8/9
Sample size	1,559	1,669	1,784	1,902	1,978	2,121
Proportion of Opinion A	59%	59%	59%	60%	60%	59%
Proportion of Opinion B	83%	83%	82%	83%	84%	85%

There are also criticisms suggesting that, "While the data is collected every day, why not release the result every day as well? Is the rolling survey covering up the problem of inadequate sample size?" Putting aside the statistical theories, it is still easy to respond to these commentaries. Readers can, starting from today, take note of the release of result of all the professional surveys and see which surveys have a completion of 1,000 interviews in one day, and which have an accumulation of samples in several days or weeks. It can be expected that, the latter is the majority. Then, why those critics did not demand all surveys to list the data on a daily basis as well?

In fact, the number of days to be rotated for depends mainly on whether a micro- or macro-scale analysis is conducted. The former will have great fluctuations, but high sensitivity. One can see the whole structure in the latter, but will easily ignore the minor details. A total of 16,785 cases have bee accumulated by the author this year to analyze the structural variables. Thus, a fine analysis could be expected.

Other misunderstandings

Due to the limitation of space, the author could not go into the details of other unnecessary misunderstandings and unreasonable charges. The author would only like to add one more point here. Rolling poll is not a new thing. Starting from 1995, the author has started to conduct rolling surveys. This year's operation in fact does not differ much from the previous ones. We have fully publicized our research design and results over the years in our website. The sponsorship from media bodies and the Civic Exchange has never affected our independent operation.

Many years ago, some newspapers have doubted and asked us, "how come the public opinion poll could use several hundreds or a thousand cases to represent more than 6 millions of Hong Kong population?" At that time, I was childish enough to respond to them by explaining some basic statistical knowledge. However, later I discovered that those enquiries were sent not for the sake of seeking for the truth, and my reply was fruitless as well. When all are conducting public opinion surveys either publicly or privately nowadays, these points have gradually disappeared and are replaced by other unreasonable arguments.

It is hoped that after many years when the public wisdom has been improved, one can just laugh away these criticisms when looking back.