Calculating first order sensitivity measures: a benchmark of some recent methodologies

S. Tarantola1, D. Gatelli1*, S. Kucherenko2, A. Pagano1, M. Ratto1 
1Joint Research Centre, ITALY; 2Imperial College London, UK; 
stefano.tarantola@jrc.it

This work compares some recent techniques to execute sensitivity analysis of model output. We highlight strengths and weaknesses of each technique in terms of efficiency and computational cost, thus enabling the user to choose the more suitable method depending on the computational model analysed. 
The paper focuses on three recently developed techniques, which are not yet commonly known in the literature: the State Dependent Parameter modeling (SDP) [1], the Random Balance Designs (RBD) [2], and the formula for estimating small sensitivity indices proposed by Kucherenko and Mauntz [3]. 

The SDP technique is a full meta-modelling approach, which is based on recursive filtering and smoothing estimation, to estimate the truncated high-dimensional model representation (HDMR) expansion up to the third order. The method is theoretically simple and all measures of interest are computed by means of a single set of model runs. As it can be applied using any kind of Monte Carlo sampling technique (or even using a sample coming from an experiment), the technique is very flexible. As other meta-modelling approaches in the literature ([4, 5] or the Gaussian emulator by Oakley and O’Hagan [6]; see also [7], for a review of smoothing approaches to sensitivity analysis), it is very efficient and allows for a significant reduction in the cost of the analysis. When coupled with low discrepancy sampling methods (e.g. quasi-random sequences), the efficiency of SDP estimates can be optimally exploited.

The RBD procedure combines Satterthwaite’s random balance designs [8] with the Fourier Amplitude Sensitivity Test (FAST) (see [9]). Contrarily to FAST, in RBD the input space is explored by using one single frequency: this has the advantage to make the computational cost independent from the number of input factors. Therefore RBD remains significantly cheap even for models with several parameters. Furthermore, the method is easy to implement and handy to use. 

The approach of Kucherenko and Mauntz is a variant of the method of Sobol’. The approach yields more accurate estimation formulas with a lower computational cost. Improved formulas are based on the extended version of Sobol’ presented in [10]. All comparison studies show a significant improvement in the accuracy for improved formulas especially in the case of small sensitivity indices.

The benchmark will be carried out on two test models. The first one is the Oakley and O’Hagan analytic function; it is composed of 15 input factors, of which five are very influential on the output variance, five are relatively influential and the remaining five are almost non influential [6].

The second test case is the Level E model. The model predicts the radiological dose to humans over geological time scales due to the underground migration of radionuclides from a nuclear waste disposal site through a system of natural and engineered barriers. We consider 12 independent uncertain parameters that can influence the radiological dose predictions. The core of the model is a set of partial differential equations which describe the nuclide migration in the geosphere. The model is time dependent: the simulated time frame ranges from  to  years. Of the 12 parameters, we concentrate on the two most influential ones and on a non influential one, so that to highlight different behaviours of the methods with respect to the parameters’ importance.

For both test models we will execute the three sensitivity methods at similar number of model evaluations, and at increasing number of model evaluations, in order to investigate their relative accuracy. The relative merits of the methods are also discussed depending on the degree of importance of the input factors. Graphs with confidence bounds for the sensitivity indices are provided to assess the robustness of the sensitivity estimates.

Acknowledgement:

One of the authors (SK) gratefully acknowledges the financial support by the EPSRC grant EP/D506743/1.

References

[1] Ratto M, Saltelli A, Tarantola S, Young P: Improved and accelerated sensitivity analysis using State Dependent Parameter models. EUR 22251 EN, JRC report (2006)
[2] Tarantola S, Gatelli, D, Mara TA: Random balance designs for the estimation of first order global sensitivity indices. Reliability Engineering and System Safety 91(l), 717–727, (2006)
[3] Kucherenko S, Mauntz W: Application of global sensitivity indexes for measuring the effectiveness of quasi-Monte Carlo method. Submitted to Monte Carlo Methods and Simulation (2007).
[4] Li G, Wang SW, Rabitz H: Practical approaches to construct RS-HDMR component functions. Journal of Physical Chemistry, 106:8721-8733, (2002).
[5] Li G, Hu J, Wang SW, Georgopoulos PG, Schoendorf J, Rabitz H: Random Sampling-High Dimensional Model Representation (RS-HDMR) and orthogonality of its different order component functions. Journal of Physical Chemistry A, 110:2474-2485, (2006).
[6] Oakley J, O'Hagan A: Probabilistic sensitivity analysis of complex models: a Bayesian approach. Journal of the Royal Statistical Society Series B, 66, 751-769, (2004)
[7] Storlie CB, Helton JC: Multiple predictor smoothing methods for sensitivity analysis, SANDIA Report, SAND2006-4693, (2006)

[8] Satterthwaite FE: Random balance experimentation. Technometrics,1(2), (1959)
[9] Cukier RI, Levine HB, Shuler KE: Nonlinear sensitivity analysis of multiparameter model systems. J Comp Phys;26:1–42, (1978)
[10] Saltelli A: Making best use of model evaluations to compute sensitivity indices. Computer Phys.Commumications, 2002, v.145, p.280-297.