Defiling the Rank: How Useful are the OECD League Tables?

Posted on September 29, 2007
By Dr Bronwyn Howell

Using OECD rankings as either a measure of a country’s performance or as a target to justify adopting a particular policy has become popular amongst the member states in recent years. Policies benchmarked using rankings are simple concepts to market to voters and appeal to a sense of nationalistic pride: ‘winning’ is important, but if you can’t win, then at least you want to be seen to be outranking your fiercest national rival (e.g. Australia if you are New Zealand; Sweden if you are Finland; the United States if you are Canada).

In the rankings races, better-ranked countries quickly become paragons, placed upon pedestals as a plethora of analysts scour their policy environments to determine the secrets of their ‘successes’. The resultant readings are peddled as recipes for improvement to lower-ranked countries eager to scale the heights of the league tables. Minute movements in the rankings following the adoption of specific policies quickly become offered as ‘evidence’ of the policies’ effect, often despite an absence of any statistically significant causal relationships in the empirical analyses¹. Poorer-ranked countries risk being deemed ‘failures’ for deigning to rank below the magical median place number 15 or the equally mystical ‘OECD average’. Countries eschewing the proffered policies popularly presumed to be bolstering the high rankings of the ‘winners’ risk international condemnation for ‘refusing to adopt best practice’, even though the efficacy of the policies is often more an article of faith than a conclusion following rigorous analysis.

New Zealand is no stranger to the promulgation of popularly-acclaimed policies in the pursuit of OECD league table success. The Labour-led government swept into office in 1999 promising to return New Zealand’s income (GDP per capita) to the top half of the OECD with its policies recently re-branded as ‘economic transformation’. The Ministry of Economic Development-led 2006 ‘Stocktake’ of the telecommunications industry that resulted in the unbundling of Telecom’s local loop and operational separation of the company was predicated largely upon the pursuit of a top quartile OECD ranking in broadband connections per capita. Unbundling was deemed the appropriate policy to adopt because most other OECD countries had already adopted it, and it was presumed to lead to a more competitive telecommunications environment. ‘Competition’ was deemed to be a New Zealand ‘problem’ impeding achievement of the broadband target because, when ranking the share of broadband customers serviced by competitors to the incumbent telecommunications companies in the top quartile (8 countries) of the OECD in broadband connections per capita and New Zealand, New Zealand ranked 9th. In recent weeks, the mobile telephony market came under scrutiny when the Commerce Commission’s modifications to OECD price rankings for standard calling bundles indicated New Zealand prices consistently in the bottom quartile ².

How useful are the OECD rankings, as either measures of, or targets for, policies? Whilst there is always some substance underlying the reported numbers, and the OECD goes to some lengths to ensure that the numbers reported are accurate, from reputable sources (e.g. national statistical agencies) and broadly comparable (e.g. purchasing power parity is used in preference to simple currency conversions as this reflects the different purchasing capacity of citizens in countries with vastly different incomes, spending priorities and choices), they must be treated with caution.

OECD rankings have severe limitations in policy comparisons because they typically report raw data. Differences in the raw scores between countries can be due to a vast array of factors, some of which may be a consequence of policy differences, but many of which are due to factors which the proffered policies are largely impotent to influence (e.g. geographic factors such as distance to markets, physical terrain, etc.). To attribute all of the differences between countries to different policies is overly simplistic. To base substantial, potentially risky, and costly changes in policy direction upon the pursuit of ranking goals may be foolish, misguided, and doomed never to succeed simply because the policy-makers and the protagonists of their policies fail to understand exactly what the statistics underpinning the ranking are actually telling them.

Take, for example, broadband connections per capita. Ranking in this statistic has become the ‘gold standard’ in the policy competition measuring who is ‘winning’ the ‘information economy’ stakes (Table 1). Korea ’s early success led to substantial analysis of the contribution made by its government in subsidising the deployment of infrastructure³. The United Kingdom’s more recent rise up the rankings has been linked to its ‘success’ in introducing competition via unbundling and structural separation of BT. The United States’ fall from 5th in 2001 to 10th in 2003 and 15th in 2006 is lamented as a catastrophe for the country whose president famously stated in 2004 that “tenth is ten spots too low as far as I’m concerned”⁴, and has spurred a flurry of research and inquiry seeking the source of what has come to be termed the ‘United States Broadband Problem’⁵.

Yet, as the House of Representatives Energy and Commerce Committee on the Digital Future of the United States was graphically informed by George S. Ford in his testimony to them in April 2007, in OECD Broadband Nirvana if every household and every business had a broadband connection, the United States could at best aspire to a rank of only 20th in broadband connections per capita (Table 2). This occurs simply because, as households and businesses rather than individuals purchase the types of broadband connection counted by the OECD⁶, at full diffusion countries with small average household and business size will naturally outrank those with larger households and businesses in the per-capita rankings.

As Ford and his co-authors note, “by today’s rhetorical standards, United States policy makers” faced with Broadband Nirvana “would continue to lament the fact that the country has sunk to 20th among the OECD and, no doubt, commission studies about what policies Sweden and the Czech Republic have utilized to achieve such a high rank”. Moreover, given such an environment, the only possibly successful strategy to push the country up the rankings would be to “kick teenagers out of their parent’s basement, which would lower the relative household size … and, consequently, increase subscriptions on a per capita basis”⁷. Ironically, New Zealand ’s mid-ranking position of 16th in Broadband Nirvana is so high principally because of our comparatively small average business size. Yet, most of New Zealand’s 300,000-plus significant businesses are ‘micro-businesses’, run from home and (likely) sharing the residential broadband connection⁸, leading to Table 2 substantially overestimating achievable maximum diffusion levels. Thus, even Broadband Nirvana rankings provide a poor benchmark for policy development – to reach this level of diffusion, it would be necessary for New Zealand to adopt policies preventing business and residential broadband connection sharing.

The ‘Broadband Nirvana’ example highlights flaws in policies promoting ranking outcomes in isolation from the crucial demographic differences that underpin the statistic. Failure to take account of other factors, such as economic differences, may also lead to significant policy errors. For example, the poor are less likely to own computers, much less purchase broadband, meaning income levels are important differentiators. This is confirmed by the OECD’s own analysis that shows (Figure 1) 62% of the difference between countries’ broadband uptake can be explained by differences in national income (GDP per capita). A large number of studies⁹ attempting to quantify the effect of a variety of factors on broadband uptake confirm that “consumers purchase broadband not out of national pride but based upon a standard set of factors that are involved in purchasing any product or service, including availability, price, consumer income, and other demographics and market conditions. Consequently, … it makes little sense to compare broadband subscription rates across countries without considering the role of relevant economic and demographic factors”¹⁰.

It is apposite, then, in light of the preceding discussion, to examine exactly what the Commerce Commission/OECD price rankings for New Zealand mobile telephone services actually tell us about comparative performance of the New Zealand mobile market. Is it possible to conclude from the report that New Zealand really does have some of the most expensive mobile phone services in the OECD? Or are there factors underlying the construction of the statistics that limit their usefulness in comparing New Zealand market performance with that in other countries? And importantly, do the figures substantiate evidence of a ‘problem’ that can reasonably be addressed by a policy change (e.g. increased regulation of mobile products and services)?

A reasonable analysis requires an understanding of how the statistics are created, and relevant economic, geographic and demographic information that might affect New Zealand ’s ranking. Taking the latter matter first, New Zealand ’s low population density, degree of population dispersion, mountainous terrain, extensive coastline and long, narrow topography all pose challenges in the construction of mobile telephony networks that lead to relatively higher investment costs per connection than in countries with more favourable demographic and geographic environments. It is a legitimate expectation that New Zealand mobile charges would, all other things being equal, be more expensive than countries with more beneficent geographic and demographic characteristics¹¹. A bottom-half price ranking, therefore, should not be a surprise¹².

Now to the statistics themselves. The OECD comparisons are provided by Teligen¹³, and are based upon a set of representative ‘baskets’ of calls made, and text and multimedia messages sent, by ‘light’, ‘medium’ and ‘heavy’ mobile phone users, to different destinations (local, national, international, fixed, mobile networks) at different times of the day and week (peak, off-peak, weekday, weekend) over a year. Monthly fixed charges are included and allowances made for the collection of bundled handset payments for post-paid account services¹⁴, and adjustments made for the amortisation of handset and SIM card purchase costs for prepaid account services. A large number (but not all) plans available in OECD member countries are compared. Prices for each basket for all plans identified are calculated in United States dollars, adjusted for purchasing power parity. The lowest price for each basket in each country is then selected for the rankings. Teligen updates the comparison quarterly, adding and deleting plans, recording price changes and adjusting currency and purchasing power parity conversion rates. The August 2006 Low User basket result is shown in Figure 2.

Price benchmarking is extremely problematic given the wide range of plans available and the variety of different usage patterns. The OECD representative ‘baskets’ create one standardised way of comparing prices. The OECD acknowledges that “for a certain country the prices may appear more competitive in one basket than in another. This is commonly the result of offers tailored to specific national calling patterns that may mimic the composition of a certain basket more closely than others”¹⁵. This acknowledgement highlights the problem of using the OECD baskets when evaluating policies or market performance in a given country – the OECD/Teligen comparison is limited in its usefulness by how closely the baskets chosen mimic actual usage patterns in the country examined. If the basket is a close match, then the comparisons may be helpful; if it is not, the comparison can be highly misleading if applied without understanding the crucial differences. For this reason, reliable benchmarking of a given country’s performance should be undertaken by also assembling a characteristic basket from the country in question and using the Teligen database to compare the foreign offers on the basis of this basket. This is the methodology I have used consistently in benchmarking broadband prices in New Zealand and the rest of the OECD. It is also the methodology used by the New Zealand Institute for Economic Research in its fixed and mobile telephony benchmarking¹⁶.

In its June 2007 report, the Commission used the OECD baskets with no alterations for actual New Zealand usage patterns (even though the Teligen database allows for up to three user-defined baskets to be analysed). The cheapest plan for all other 29 countries was established. Selected New Zealand plans were then chosen as the New Zealand ‘representative tariff’ and ranked against the cheapest plans from elsewhere. Unsurprisingly, the more expensive plans, when selected as the New Zealand ‘representative’, ranked poorly. The comparison was unfair by its very construction. By this methodology, one could benchmark 1500cc 5-door hatchbacks in the OECD, pick the cheapest in each country (likely a Korean or Eastern European marque) and then compare it to the price of a Daihatsu, a Holden, a top-range Nissan sports hatchback and a BMW in New Zealand in order to ‘prove’ how expensive New Zealand cars are.

When comparing like with like – the cheapest in each country – the New Zealand plans fared comparatively well given the dubious comparability of the OECD baskets to New Zealand usage and the geographic and demographic challenges that would bias New Zealand away from OECD ‘leadership’ in this statistic. The ‘low-user basket’, performed best (17th out of 30, down from 15th in February; 103% (101%) of the OECD average), and the worst ‘best rank’ was 24th (high user). The movement in ranking between quarters, however, has to be interpreted with some degree of caution because changing exchange rates relative to the United States dollar can result in substantial changes in ranking even with no change in the posted prices for the plans. In a variation of Broadband Nirvana, if the February plans were benchmarked using the May PPP conversion factors, or vice versa, it is highly unlikely that either the rankings or the percentage variations from the OECD average would be identical in the two periods, simply as a consequence of uncontrollable variation in underlying economic conditions (exchange rates) that are invariant to the actions of either telecommunications policy-makers or the management of mobile telecommunications firms.

When comparing the OECD baskets with actual New Zealand usage patterns, it quickly becomes obvious that none of the baskets adequately capture actual New Zealand usage, especially in respect of text and multimedia message use. By OECD standards, New Zealanders are prolific text users (approximately 1600 per year per subscriber, compared to 500 per year per subscriber in Finland), but rather more modest voice call users, possibly due to the relatively low cost of calls on the (free per call) local fixed-line residential voice network¹⁷. The OECD baskets compare bundles with 396, 600 and 660 text messages per year. Thus, none adequately replicates the average text usage component of the New Zealand mobile market. The effect of the extremely popular New Zealand ‘$10 text’ bundles offering up to 2000 ¹⁸ texts per month is therefore poorly captured in OECD/Commerce Commission benchmarking, resulting in a highly misleading set of price comparisons being reported.

The importance of texting combines in the New Zealand statistics with the very large (by international standards) share of prepaid subscriptions – over 60%. Prepaid subscriptions incur no fixed monthly charge (apart from the amortisation of the handset and SIM card) but have higher per-call rates. Individuals who anticipate making very few voice calls are the customers who prefer these subscriptions. Nonetheless, New Zealand prepaid subscribers can access the ‘$10 text’ packages (something very uncommon in countries such as Finland, where such plans are available only to individuals purchasing post-paid subscriptions with attendant cancellation penalties and other services in the bundle¹⁹). Telecom’s average revenue per subscriber (ARPU) is $11 per month for its 976,000 prepaid subscribers²⁰ (slightly less than one quarter of the New Zealand mobile market). Combining this information with the average number of texts in New Zealand appears to confirm that for a substantial proportion of the New Zealand market, texting rather than voice calling is the predominant use of a mobile phone. The OECD/Commerce Commission baskets prioritising voice calling and mixed use therefore fail to fairly compare the price of actual use for large portions of the New Zealand market.

The Commerce Commission finds that Vodafone plans are consistently cheaper across all baskets. This has been used by some to support claims that Telecom has been especially avaricious in its pricing behaviour. However, this ‘finding’ is predominantly an artefact of the baskets used. That Vodafone consistently scores better in the Commerce Commission rankings is simply because it has the plans that are the best value for the OECD basket consumer, but not necessarily the average New Zealand mobile telephony consumer.

As the OECD baskets place a high weight on the use of a mobile phone for voice calls, especially for the ‘heavy user’ basket, a New Zealand plan with lower per call costs but a high per-text charge will likely rank as cheaper than one with lower text costs and higher per-call charges. Research by Eugenio Miravete (recently in New Zealand as an S. T. Lee Fellow at the New Zealand Institute for the Study of Competition and Regulation) shows that consumers faced with multiple tariffs for telephony services do a remarkably good job of selecting the plan that gives them the lowest total cost for their personal usage level²¹. Again, a little more insight into New Zealand usage patterns reinforces that New Zealanders are smart in picking their plans and providers to get the best value given their individual requirements. Whilst Telecom’s ARPU for prepaid callers is very low, Vodafone’s is more than twice as high ($22.50 per month)²², even though the market shares are very similar²³. Assuming that $10 texts appeal to Vodafone’s customers as well as Telecom’s, then much of the difference is likely due to more voice calls being made by Vodafone’s prepay customers. Smart prepay customers have worked out which plans are cheaper for their needs – those making more voice calls appear to prefer the cheaper (for them) Vodafone prepay plans²⁴.

Similar bifurcation of the market also carries through to the post-paid plans – Telecom’s ARPU ($61.70) is less than half that of Vodafone ($125.30). That Telecom has a similar market share to Vodafone, despite the ranking differences, is probably because its plans are the best value for its customers’ individual demands, which are different from those used in the OECD bundles and the Vodafone plans. The higher ARPU for Vodafone appears to confirm that its plans are more attractive for higher-volume voice users, whereas Telecom’s appeal to lower-volume voice users with higher cross-network texting requirements, which are not well-reflected in the OECD baskets.

In summary, therefore, the OECD/Commerce Commission price ranking exercise as it stands provides a very poor basis for either policy-making or policy assessment in the New Zealand mobile telephony market. The promulgation of its findings as ‘evidence’ of a New Zealand ‘pricing problem’ without caveats on the inherent methodological limitations is irresponsible. It is dangerous if the uncaveated findings become used subsequently as justification for increased regulatory intervention²⁵. The issues underpinning the statistics are highly complex. What we should be looking for from our policy-makers is reasoned analysis of the underpinning issues and policy predicated upon full economic, demographic and geographic understandings. Whilst the nationalistic competitions that ensue from simplistic ‘rankings races’ might make good politics, improvements in national wellbeing rest ultimately on good policies. Blind pursuit of top rankings is seldom the basis of good policy.