Proceedings
NCSU-USDA Workshop on Sensitivity Analysis Methods
Results of Workshop Proceedings
The following white-paper links represent the results of the NCSU/USDA Workshop on Sensitivity Analysis Methods held on June 11 through 12, 2001, at North Carolina State University.
Clicking on one of the links will take you to a web page where you will be able to view the abstract and table of contents and download the portable document file (pdf) of the white paper you selected. You will need Adobe® Acrobat® Reader to view and download the papers.
In addition to appearing on this web site, all of the papers have been published in the journal Risk Analysis as a special section on the topic of sensitivity analysis. The special section appears in the June 2002 Volume 22, Number 3 issue of Risk Analysis.
INTRODUCTION TO SPECIAL SECTION ON SENSITIVITY ANALYSIS AND SUMMARY OF NCSU/USDA WORKSHOP OF SENSITIVITY ANALYSIS
H. Christopher Frey, Ph.D.
Department of Civil Engineering
North Carolina State University
Raleigh, NC 27695-7908
ABSTRACT
This guest editorial is a summary of the NCSU/USDA Workshop on Sensitivity Analysis held June 11-12, 2001 at North Carolina State University and sponsored by the U.S. Department of Agricuture’s Office of Risk Assessment and Cost Benefit Analysis. The objective of the workshop was to learn across disciplines in identifying, evaluating, and recommending sensitivity analysis methods and practices for application to food safety process risk models. The workshop included presentations regarding the Hazard Assessment and Critical Control Points (HACCP) framework used in food safety risk assessment, a survey of sensitivity analysis methods, invited white papers on sensitivity analysis, and invited case studies regarding risk assessment of microbial pathogens in food. Based upon the sharing of interdisciplinary information represented by the presentations, the workshop participants divided into breakout sessions responded to three trigger questions: What are the key criteria for sensitivity analysis methods applied to food safety risk assessment?; What sensitivity analysis methods are most promising for application to food safety and risk assessment?; and What are the key needs for implementation and demonstration of such methods? The workshop produced agreement regarding key criteria for sensitivity analysis methods and the need to use two or more methods to try to obtain robust insights. Recommendations were made regarding a guideline document to assist practitioners in selecting, applying, interpreting, and reporting the results of sensitivity analysis.
Key Words: Sensitivity Analysis, Food Safety, Uncertainty, Variability, Modeling.
Contents:
- Introduction
- Papers in the Special Section
- Challenges for Sensitivity Analysis Applied to Food-Safety Risk Assessment
- Recommendations Regarding Sensitivity Analysis and Food-Safety Risk Assessment Modeling
View or download portable document file (pdf) of Dr. Frey’s summary paper.
HAZARD AND CRITICAL CONTROL POINT (HACCP) HISTORY AND CONCEPTUAL OVERVIEW
Karen L. Hulebak, Sc.D.
U.S. Department of Agriculture
Food Safety and Inspection Service
Office of Public Health and Science
Washington, DC
Wayne Schlosser, D.V.M.
U.S. Department of Agriculture
Food Safety and Inspection Service
Office of Public Health and Science
Crystal Park Plaza Suite 3000
2700 Ear; Rudder Freeway South
College Station, TX 77845
ABSTRACT
HACCP is a system that enables the production of safe meat and poultry products through the thorough analysis of production processes, identification of all hazards that are likely to occur in the production establishment, the identification of critical points in the process at which these hazards may be introduced into product and therefore should be controlled, the establishment of critical limits for control at those points, the verification of these prescribed steps, and the methods by which the processing establishment and the regulatory authority can monitor how well process control through the HACCP plan is working. The history of the development of HACCP is reviewed, and examples of practical applications of HACCP are described.
Key Words: HACCP, critical control points, CCPs, critical limits, beef slaughter
Contents:
- The History of Meat Inspection
- HACCP Core Concepts
- Rationale for Adopting HACCP
- Overview of HACCP Principles
- Applying HACCP to Beef Slaughter: An Illustration
View or download portable document file (pdf) of Drs. Huleback and Schlosser’s white paper.
IDENTIFICATION AND REVIEW OF SENSITIVITY ANALYSIS METHOD
H. Christopher Frey, Ph.D.
Civil Engineering Department
North Carolina State University
Raleigh, NC
Sumeet R. Patil
Civil Engineering Department
North Carolina State University
Raleigh, NC
ABSTRACT
Identification and qualitative comparison of sensitivity analysis methods that have been used across various disciplines, and that merit consideration for application to food safety risk assessment models, are presented in this paper. Sensitivity analysis can help in identifying critical control points, prioritizing additional data collection or research, and verifying and validating a model. Ten sensitivity analysis methods, including four mathematical methods, five statistical methods and one graphical method, are identified. The selected methods are compared on the basis of their applicability to different types of models, computational issues such as initial data requirement and complexity of their application, representation of the sensitivity, and the specific uses of these methods. Applications of these methods are illustrated with examples from various fields. No one method is clearly best for food safety risk models. In general, use of two or more methods, preferably with dissimilar theoretical foundations, may be needed to increase confidence in the ranking of key inputs.
Key Words: sensitivity analysis methods, food safety, microbial risk assessment, critical control points
Contents:
- Introduction
- Objectives and Motivation
- Importance of Risk Assessment in Food Safety
- Risk Assessment Framework
- Important Issues in Food Safety Modeling
- Sensitivity Analysis Methods
- Organization of the Report
- Sensitivity Analysis Methods
- Nominal Range Sensitivity
- Difference in Log-Odds Ratio
- Break-Even Analysis
- Automatic Differentiation Technique
- Regression Analysis
- Analysis of Variance
- Response Surface Method
- Fourier Amplitude Sensitivity Test
- Mutual Information Index
- Scatter Plotts
- Comparison of Methods
- Applicability for Different Types of Models
- Computational Issues
- Ease and Clarity in Representation of Sensitivity
- Purpose of the Analysis
- Conclusions
- Acknowledgement
- References
View or download portable document file (pdf) of Dr. Frey and Mr. Patil’s white paper.
ILLUSTRATION OF SAMPLING-BASED METHODS FOR UNCERTAINTY AND SENSITIVITY ANALYSIS
J. C. Helton
Department of Mathematics
Arizona State University
Tempe, AZ 85287-1804 USA
F. J. Davis
Sandia National Laboratories
Albuquerque, NM 87185-0779 USA
ABSTRACT
A sequence of linear, monotonic, and nonmonotonic test problems is used to illustrate sampling-based uncertainty and sensitivity analysis procedures. Uncertainty results obtained with replicated random and Latin hypercube samples are compared, with the Latin hypercube samples tending to produce more stable results than the random samples. Sensitivity results obtained with the following procedures and/or measures are illustrated and compared: correlation coefficients (CCs), rank correlation coefficients (RCCs), common means (CMNs), common locations (CLs), common medians (CMDs), statistical independence (SI), standardized regression coefficients (SRCs), partial correlation coefficients (PCCs), standardized rank regression coefficients (SRRCs), partial rank correlation coefficients (PRCCs), stepwise regression analysis with raw and rank-transformed data, and examination of scatterplots. The effectiveness of a given procedure and/or measure depends on the characteristics of the individual test problems, with (i) linear measures (i.e., CCs, PCCs, SRCs) performing well on the linear test problems, (ii) measures based on rank transforms (i.e., RCCs, PRCCs, SRRCs) performing well on the monotonic test problems, and (iii) measures predicated on searches for nonrandom patterns (i.e., CMNs, CLs, CMDs, SI) performing well on the nonmonotonic test problems.
Key Words: Chi-square, common mean, common median, correlation coefficient, epistemic uncertainty, Kruskal-Wallis, Latin hypercube sampling, Monte Carlo, partial correlation coefficient, random sampling, rank transform, regression analysis, replicated sampling, scatterplot, sensitivity analysis, standardized regression coefficient, statistical independence, stepwise regression, subjective uncertainty, uncertainty analysis.
Contents:
- Introduction
- Uncertainty and Sensitivity Analysis Procedures
- Linear Test Problems
- Monotonic Test Problems
- Nonmonotonic Test Problems
- Discussion
View or download portable document file (pdf) of Drs. Helton and Davis’ white paper.
USE OF SENSITIVITY ANALYSIS TO ASSESS RELIABILITY OF MATHEMATICAL MODELS
Michael C. Kohn
Laboratory of Computational Biology and Risk Analysis
National Institute of Environmental Health Sciences
PO Box 12233, MD A3-06
Research Triangle Park, NC 27709
Contents:
- Definitions
- System Sensitivity Theory
- Sensitivity Analysis of Empirical Models
- Sensitivity in Metabolic Networks
- Examples from Physiological Modeling
- Conclusions
- References
View or download portable document file (pdf) of Dr. Kohn’s white paper.
RISK AND UNCERTAINTY ANALYSIS IN GOVERNMENT SAFETY DECISIONS
Elisabeth Pate-Cornell
Department of Management Science and Engineering
Stanford University
ABSTRACT
Probabilistic risk analysis (PRA) can be an effective tool to assess risks and uncertainties and to set priorities among safety policy options. Based on systems analysis and on Bayesian probability, PRA has been applied to a wide range of cases, three of which are briefly presented here: the maintenance of the tiles of the space shuttle, the management of patient risk in anesthesia, and the choice of seismic provisions of building codes for the San Francisco Bay Area. In the quantification of a risk, a number of problems arise in the public sector where multiple stakeholders are involved. In this paper, I describe different approaches to the treatments of uncertainties in risk analysis, their implications for risk ranking, and the role of risk analysis results in the context of a safety decision process. I also discuss the implications of adopting conservative hypotheses before proceeding to what is, in essence, a conditional uncertainty analysis, and I explore some implications of different levels of “conservatism” for the ranking of risk mitigation measures.
Key Words: risk, uncertainty, probability, conservatism, ranking
Contents:
- Introduction
- Three Examples of Engineering Risk Analysis: Finding and Fixing System Weaknesses
- Seismic Risk Analysis for the San Francisco Bay Area
- The Tiles of the Space Shuttle
- Anesthesia Patient Risks in Modern Western Hospitals
- Risk Analysis Results as Input to an Acceptable Decision Process
- Different Approaches to the Treatment of Uncertainties in Risk Analysis
- Risk Comparison and the Relevance of Different Risk Analysis Methods
- The Problems of Conditional Uncertainty Analysis
- Conclusions
- References
- Appendix: Ambiguity Aversion vs. Risk Aversion
View or download portable document file (pdf) of Dr. Pate-Cornell’s white paper.
SENSITIVITY ANALYSIS FOR IMPORTANCE ASSESSMENT
Andrea Saltelli
Joint Research Centre of the European Communities in Ispra (I)
andrea.saltelli@jrc.it
ABSTRACT
We review briefly some examples that would support an extended role for quantitative sensitivity analysis in the context of model-based analysis (Section 1). We then review what features a quantitative sensitivity analysis should have to play such a role (Section 2). The methods that meet these requirements are described in Section 3. An example is given in Section 4 along with some pointers to further research in Section 5.
Key Words: uncertainty analysis, quantitative sensitivity analysis, computational models, assessment of importance, risk analysis
Contents:
- Introduction
- Desired Properties and Settings
- Properties
- Settings
- Methods
- A Worked Example
- Cases
- Final Remarks
- Acknowledgements
- References
View or download portable document file (pdf) of Dr. Saltelli’s white paper.
VARIABILITY AND UNCERTAINTY MEET RISK MANAGEMENT AND RISK COMMUNICATION
Kimberly M. Thompson, Sc.D.
Assistant Professor of Risk Analysis and Decision Science
Harvard School of Public Health
718 Huntington Ave.
Boston, MA 02115
(617) 432-4285
kimt@hsph.harvard.edu
ABSTRACT
In the past decade, the use of probabilistic risk analysis techniques to quantitatively address variability and uncertainty in risks increased in popularity as recommended by the 1994 National Research Council that wrote Science and Judgment in Risk Assessment. Under the 1996 Food Quality Protection Act, for example, the U.S. EPA supported the development of tools that produce distributions of risk demonstrating the variability and/or uncertainty in the results. This paradigm shift away from the use of point estimates creates new challenges for risk managers, who now struggle with decisions about how to use distributions in decision-making. The challenges for risk communication, however, have only been minimally explored. This presentation uses the case studies of variability in the risks of dying on the ground from a crashing airplane and from the deployment of motor vehicle airbags to demonstrate how better characterization of variability and uncertainty in the risk assessment lead to better risk communication. Analogies to food safety and environmental risks are also discussed. This presentation demonstrates that probabilistic risk assessment impacts both risk management and risk communication, and highlights remaining research issues associated with using improved sensitivity and uncertainty analyses in risk assessment.
Key Words: variability, uncertainty, risk communication, risk management, probabilistic risk assessment
Contents:
- Introduction
- Learning from the Past
- The Risk to Groundlings from Crashing Airplane
- The Risks and Benefits of Airbags
- Discussion
- References
- Acknowledgements
View or download portable document file (pdf) of Dr. Thompson’s white paper.