IMPACT OF METHODOLOGICAL CHOICES ON ROAD SAFETY RANKING

E. Hermans*, F. Van den Bossche, G. Wets
Transportation Research Institute – Hasselt University, Belgium
elke.hermans@uhasselt.be

In recent years, the interest in and the use of indicators and indexes are rapidly increasing. Their usefulness for policymakers and communication purposes is a key advantage. Trends can be identified, problems predicted, policy targets and priorities set, the impact of measures assessed, attention achieved, … [1,2]. In domains like economy (the human development index), technology (the technology achievement index) and sustainability (the environmental sustainability index) the aggregation of indicators in one index is common [3]. 

In this research we focus on the road safety domain which is currently showing much interest in the use of indicators. For the most influencing road safety risk factors, indicators are presently being developed and data collected on the European level [4,5]. Although countries are often compared on their level of road safety by means of the number of traffic deaths per million inhabitants, the development of a road safety performance index will provide valuable insights. There are a number of disadvantages linked to accident data, like for example the lack of uniformity in definitions and the problem of under-registration [6]. However, the most important drawback is that knowledge of the number of accidents and casualties in a country is insufficient to understand the processes that lead to these accidents. If one wants to enhance the road safety level in a country, a set of measures has to be identified, able to tackle the real problem. 

Road safety indicators can help in this respect. Based on a set of carefully selected indicators, the safety conditions can be reflected, the impact of safety interventions can be measured and the safety performance of different countries can be compared [5]. Not only will an insight be gained into the domains that need additional efforts from policymakers, the aggregation of useful information into one road safety index will be a valuable tool for the road safety domain. A sound methodology for constructing a road safety performance index is however prerequisite for its use. To this end, a composite indicator methodology has to be elaborated.  

In the past, limited attention has been paid to the construction of a road safety index, and we believe that a methodologically valid composite indicator approach is a new, challenging and necessary matter in road safety. In order to develop an acceptable index, the subjective choices involved in the process of developing a road safety index need to be justified and their impact on the end result quantified. As there is no agreement or a priori knowledge on the best or ideal method to be used in the steps defined in [1], several possible methods need to be tested. Sensitivity and uncertainty analysis is a requirement for composite indicators. The end result – for example the ranking of countries based on their road safety index score – can be heavily influenced by the choices made in the index construction process. As stated in [7] the iterative use of uncertainty and sensitivity analysis contributes to the well-structuring of the composite indicators, provides information concerning the robustness of the countries’ ranking and identifies ways to reduce the uncertainty in the ranking for a better monitoring and policy. 

In the study at hand, various methodological aspects used in other composite indicator studies [1,3] are investigated and adapted to the specific context of road safety. We will illustrate some methodological topics on a dataset consisting of 7 road safety indicators (related to the domain of alcohol, speed, protective systems, visibility, vehicle, infrastructure and trauma management) and 18 countries. In the literature [8], these 7 domains are generally agreed to be very important road safety risk factors. For now, for each domain only one indicator was defined based on policy relevance, clarity and data availability [2]. 

As part of the development of a road safety index, we will study the impact of the weighting method, expert selection and indicator selection on the average shift in rank of the 18 countries in our dataset. More specifically, we will test two commonly used weighting methods [3] both based on expert opinions, namely the Analytic Hierarchy Process (AHP) and Budget Allocation (BA). The results of comparisons in pairs of the road safety indicators as well as the allocation of a budget over the indicator set were obtained from 9 road safety experts from different European countries. The average indicator weights over the experts are often used. However, we will assess the impact of selecting the weights from one particular expert. In addition, the change in rank will be studied in case one of the seven indicators is no longer included in the road safety index. The result of the analysis will indicate how robust the ranking is, which of these methodological choices has the largest impact on average rank shift and which input factor needs special effort in order to reduce the output variance. It can also be shown which countries are favoured under a particular set of assumptions. 

The output variable of interest is the average change in the country rankings for all possible scenarios. Due to the specific nature of this composite indicator we will compare the road safety ranking against the reference ranking based on the number of traffic deaths per million inhabitants. 

First, a probability distribution function is assigned to each input factor from which values are drawn for each sample. Multiple evaluations of the model with randomly selected input factors will be performed, considering simultaneously all sources of uncertainty. For non-linear models (which is the case here) variance-based techniques for sensitivity analysis are the most appropriate [9]. These techniques are model independent (thus suitable for non-linear and non-additive models), they capture interaction effects apart from the fractional contribution of input factor xi to the variance of the model output y, and they explore the whole range of variation of each factor.

The largest shift in country ranking occurs when the weights of expert 6 from the budget allocation method are used and the infrastructure indicator is no longer part of the dataset. In addition, the first-order indexes Si – which capture the fractional contribution to the model output variance due to the uncertainty in xi – and the total effect indices Sti – which concentrate all the interactions involving factor xi in one single term – are calculated for each input factor. These indices give us insight in the amount of output variance that is explained by the input factors singularly, indicate which factors are mostly involved in interactions with other factors, which factors can be fixed without a significant impact on the output and which factor could reduce the output variance most if more information was found. This information needs to be considered in the building process of a composite road safety indicator. 

To conclude, the development of a road safety index is a challenging and necessary task. Uncertainty and sensitivity analyses are a prerequisite to develop a methodologically sound index with a large acceptance. In this study, a first attempt was described in which the impact of three aspects on the country ranking was the focus: the selection of the weighting scheme, the expert and the indicators. In the future, the impact of several aggregation methods, normalisation techniques and imputation practices for missing data will be incorporated. Finally, the development process of a road safety index needs more elaboration and the analysis of uncertainty and sensitivity is indispensable for obtaining a robust road safety index in the end. 

References

[1] Nardo M, Saisana M, Saltelli A, Tarantola S, Hoffman A, Giovanni E: Handbook on constructing composite indicators: methodology and user guide. OECD, 2005.

[2] Litman T: Developing indicators for comprehensive and sustainable transport planning. In: Proc Transportation Research Board. Washington D.C., 2007.

[3] Saisana M, Tarantola S: State-of-the-art report on current methodologies and practices for composite indicator development. JRC, 2002.

[4] European Transport Safety Council: Transport safety performance indicators. ETSC, 2001.

[5] SafetyNet: State-of-the-art report on road safety performance indicators. 2005.

[6] Evans L: Traffic safety. Michigan: Bloomfield Hills, 2004.

[7] Nardo M, Saisana M, Saltelli A, Tarantola S: State-of-the-art report on simulation and indicators. JRC, 2005.

[8] World Health Organization: World report on road traffic injury prevention. Retrieved March 20, 2004 from http://www.who.int/world-health-day/2004/infomaterials/world_report/en.

[9] Saltelli A, Tarantola S, Campolongo F, Ratto M: Sensitivity analysis in practice: a guide to assessing scientific models. West Sussex: John Wiley & Sons Ltd, 2005.