Introduction
"There are three kinds of lies: lies, damned lies, and statistics” first proclaimed the British Prime Minister Benjamin Disraeli before the sentence was popularised by Mark Twain. Indeed, statistics frequently had itself discredited because of misunderstanding or mistrust. A lack of statistical literacy can easily lead to “misunderstandings, misperceptions, mistrusts and misgivings about the value of statistics for guidance in public and private choices” (Wallman, 1993). In today’s complexities of our information society, an understanding of statistical information and techniques has become essential both for everyday life and effective participation in the workplace, leading to calls for an increased attention to statistics and statistical literacy (see Shaughnessy and Pfannkuch, 2004; Shaughnessy, 2007; Makar and Rubin, 2009). The quality of available statistics can vary considerably so that an understanding of sampling techniques and sources of bias can help to first assess what has been done and second adopt a critical stance on statistics. Increasing the public awareness regarding the quality of the information consumed from television or newspapers is crucial due to the “overwhelming amount of unregulated, unrestricted information being thrust upon a public that is generally ill equipped to consume the information” (Rumsey, 2002 p.33). Indeed, this current phenomenon is dual: progress in the use of statistics goes hand in hand with an increase in the misuses and statistical fallacies (Hooke, 1983). A large body of literature, built by teachers, education researchers, statisticians and professional organisations thus calls for improving and measuring statistical literacy, with a special focus on the student population. Begg et al. (2004), for example, underlined the societal motive behind the call for a greater emphasis on statistical literacy in school curriculum, being that students can become active and critical citizens. Callingham (2007) stressed the importance for students to adopt a critical stance about data, referred to as applying statistical literacy.
The call for statistical literacy has recently been echoed by the international community. In their “A World That Counts” report, the United Nations Secretary General’s appointed Independent Expert Advisory Group (IEAG) on the 'Data Revolution for Sustainable Development' recommended that more is done to increase global literacy. Specifically, the group called for “A proposal for a special investment to increase global data literacy. To close the gap between people able to benefit from data and those who cannot, in 2015 the UN should work with other organisations to develop an education program and promote new learning approaches to improve peoples’, infomediaries’ and public servants’ data literacy. Special efforts should be made to reach people living in poverty through dedicated programmes.” The Synthesis Report of the UN Secretary-General on the Post-2015 Agenda, “The Road to Dignity by 2030”, called itself for a transformative agenda where we “base our analysis in credible data and evidence, enhancing data capacity, availability, disaggregation, literacy and sharing”. It stressed that “the world must acquire a new ‘data literacy’ in order to be equipped with the tools, methodologies, capacities, and information necessary to shine a light on the challenges of responding to the new agenda”.
To inform this debate, the PARIS21 Secretariat established a task team for the purpose of developing and reporting on a global indicator to measure the current state and future progress in global statistical literacy. The paper, published in 2017, presents the outcome of this consultative process and presents a novel measure of statistical literacy based on the use of and critical engagement with statistics in national newspapers. The use of text mining techniques bridges current data gaps in this area and allows the assessment of statistical literacy of an the adult population on a day-to-day basis in more than one hundred developing and developed countries.
This post contains the update on the statistical literacy index with new information collected until 2023.
Methodology
Data sources
To measure statistical literacy empirically, we turn to references to statistics and statistical fallacies in national newspaper articles that are accessible online, in line with the work done by Watson and Callingham (2003) in terms of scaling. This is essentially for three reasons.
First, and foremost, while there is some gap between journalists’ perception of statistics, which is reflected by statistics reported in news articles, and the demand for statistics in the audience, the writing of journalists can be seen as an image for a nation's demand for statistical facts as well as the depth of critical analysis. In any case, in most parts of the world, it largely reflects the nation’s consumption of statistical facts as well as the level of critical analysis of statistics offered to a country’s population.
Second, newspaper articles are generally available, most of them online, which makes them representative for a country's literate population and easily accessible for text analysis.
Lastly, alternative data sources are either not representative (e.g. Google Trends searches related to statistics; downloads of statistical software packages) or are reported infrequently and/or not comparable across countries (e.g. job categories related to statistics; regional numeracy assessments).
The indicator used is a three-dimensional composite indicator of the equally weighted percentages of national newspaper articles that contain references to statistics at statistical literacy level 1, 2 or 3, respectively, following the scale defined in Table 1. The three levels are not mutually exclusive. For each of the three levels, we obtain the share of documents that match the classification, country per country. An overall measure for statistical literacy is then obtained as the sum over the three shares. Specifically, the methodology classifies keywords used in each article into literacy levels 1 to 3 based on three corresponding keyword lists, so that for each of the 3 levels, there is a different denominator of newspaper articles that is analyzed (see below for a precise description of the keyword analysis). Each keyword list contains different terms referring to statistics and statistical fallacies, and the use of one precise category of keywords by one newspaper article allows for defining one level of statistical literacy.
The empirical instrument is still under construction and the preliminary results described here are helpful to improve the quality of measurement. To establish the validity of the measure, the classification of articles will be further validated by analysts at National Statistical Offices (NSOs).
Text mining techniques
This subsection summarises the keywords used in the analysis, and the sources used to define the appropriate keywords. It also provides examples of keywords defined for each level of statistical literacy. Keywords are derived from major statistical data sources and refer to wide categories of indicator, based on standard internationally adopted by NSOs, International Organisations, books, articles and glossaries specialized in statistics and statistical fallacies (examples are the OECD Glossary of Statistical Fallacies or the Glossary of Statistical Terms by the University of California, Berkeley, for English keywords; or the Glossário Inglês-Português de Estatística for Portuguese keywords). The detailed list of keywords used in the analysis, data source and preliminary results are available from in Appendix A.
The study further used the World Bank's WDI database (World Development Indicator) to extend the initial keywords list and added a blacklist of keywords to disentangle ambiguous meaning of acronyms (such as IPC for instance, which stands for both 'indice des prix à la consommation' and 'International Paralympic Committee). The reliability and validity of the keyword lists will be further tested during the implementation of validity checks (see below).
Note: Keywords have been translated in all four languages used for the indicator. Text mining techniques, as word stemming, were applied to all keyword lists and news articles before proceeding with the analysis. For articles, stop words were removed and characters are converted to lower case.
Level 1: Consistent, Non-Critical Use of Statistics
Data source: Daily, top 100 news articles from Google News for publishers who
- have registered their RSS feeds with this service,
- publish in either English, French, Spanish or Portuguese, and
- use the country's top-level domain, e.g. '.sn' for Senegal, for their website.
Keywords: articles are considered a good fit for this category if they contain words from one of the following lists:
- Keywords indicating data sources
- word sequences of length two, derived from list of all NSO names worldwide
- main statistical data sources, such as 'population census', 'household survey', 'geospatial data', etc (cf. Espey et al., 2015)
- Keywords indicating a statistical indicator: GDP, CPI, etc. based on the World Development Indicator database’s ‘Economy and Growth’ category. This list is currently being extended using additional keywords from other categories.
- Keyword list from statistical capacity building projects
Levels 2 and 3: Critical engagement with Statistics
Data source: Daily, top 100 news articles from a Google News search for either: 'statistics', 'data', 'study', 'research', 'report'. For publishers who
- have registered their RSS feeds with this service,
- publish in either English, French, Spanish or Portuguese, and
- use the country's top-level domain, e.g. '.sn' for Senegal, for their website.
Keywords: articles are considered a good fit for this category if they contain words from one of the following lists:
- Critical mathematical engagement: List of statistical fallacies based on books, articles and websites that discuss statistical biases and fallacies
- Critical non-mathematical engagement: List of adjectives to assess the quality of research studies based on synonyms and antonyms for 'accuracy', 'reliability' and 'validity' ( Pierce, 2008)
The data source has several limitations that are usefully addressed. First, and foremost, our hierarchy of statistical thinking into three stages of skills (progression of non-rigid levels of statistical understanding based on the SOLO taxonomy) creates a scale that has widely been validated empirically as a measure of statistical literacy. Nevertheless, the indicator is measuring a count of terms specifically referring to each level of literacy, whereas literacy would also need to be tested against the “appropriateness” of the terms used, in context. Therefore, the measure is conditional on the assumption that statistical terms are appropriate for the context they are used in. This assumption is essential to a fully automated process allowing a daily collection and analysis of newspapers articles.
Second, the current implementation is limited to the four most widely spoken languages globally (English, French, Spanish and Portuguese) and thereby ignores local languages. Extending the analysis would require software that allows word stemming and stop word removal in these local languages. An initial analysis of newspapers coverage nevertheless reveals that a vast majority of countries have national newspapers available through their RSS feeds and written in one or several of these four languages.
Third, newspapers and blogs are only a subset of national media. Radio and TV, however, cannot easily be captured in machine readable format. New promising tools, as the Radio Analysis tools developed by Pulse Lab Kampala and the United Nations in Uganda, could maybe fill this gap in the coming years. Radio data could for instance be useful in the future to do a robustness check to see how the use of statistics differs in urban areas – that have access to (online) newspapers – from that in rural areas and illiterate populations. Moreover, automated text analysis does not cover visualized data, such as graphics and tables, an important way of presenting statistics in news media.
Finally, while based on high-level glossaries and internationally acknowledged statistical data sources, the keyword lists used for the analysis are subjective.
RESULTS AND ANALYSIS
The purpose of the indicator is to set and monitor targets and report on them annually. Target countries comprise all International Development Association (IDA) borrower countries, of which 65 countries were analysed this year. From April 2022 to March 2023, a total of 89,000 articles were analysed for the use of statistics. This corresponds to an average of 1300 articles per country for the period. Since 2006, over 600,000 articles have been analysed using a consistent methodology.
The aggregation score for each country is simply the sum over the three dimensions (ranging from 0 to 300): three-dimensional composite indicator of the equally weighted percentages of national newspaper articles that contain references to statistics at statistical literacy level 1, 2 or 3, respectively, following the scale defined in Table 1. For each of the three levels of statistical literacy, the resulting score gives the percentage of articles that contain at least one search term from the keyword lists defined previously. The score for each level thus ranges between 0 and 100 and the maximum total score over all three levels is 300. The results in Figure 1 are presented by language groups to allow for a direct comparison between countries for which the same keyword list was applied.
There are 4871 general news articles (corresponding to 5.47 percent of all articles) that cite statistics (Level 1) and the 1123 research-related articles (equivalent to 1.26 percent of all articles) that demonstrate a critical engagement with statistics (Level 2 and 3). The global distribution is visualised in Figure 1 below.
The use of data in news articles increased slighted across regions in 2022. Although this level is lower than the peak in 2020, when the use of statistics was driven up significantly by the intensive reporting on the COVID-19 pandemic, this number is still the second highest for all regions since the inception of the indicator. It is an encouraging sign for data-driven storytelling.
Since 2017, statistical literacy score increased by 27% among IDA countries. The increase in using and referring to data and statistical concepts in news reporting is built on the improved availability and accessibility of better data, the proliferation of support to data journalism, and the global public’s need for better evidence-based stories.
One can also argue that the pandemic, along with other recent global and local events, incentivised data users, especially journalists, to increase their ability to understand and interpret data more effectively. The increase in data-related content also reflected a stronger demand for data from the media consumers. Although many the improvements in the scores during the pandemic were transitory, the demand for data remained at a higher level than pre-pandemic.