SOFTWARE SYSTEM SAFETY HANDBOOK Joint Software System Safety Committee A Technical & Managerial Team Approach December 1999 This Handbook was funded and developed by the Joint Services Computer Resources Management Group, U.S. Navy, U.S. Army, and the U.S. Air Force Under the direction and guidance of the Joint Services Software Safety Committee of the Joint Services System Safety Panel and the Electronic Industries Association, G-48 Committee AUTHORS AUTHORS David Alberico Contributing (Former Chairman) John Bozarth Contributing Michael Brown Contributing (Current Chairman) Janet Gill Contributing Steven Mattern Contributing and Integrating Arch McKinlay VI Contributing This Handbook represents the cumulative effort of many people. It underwent several reviews by the technical community that resulted in numerous changes to the original draft. Therefore, the contributors are too numerous to list. However, the Joint Services Software System Safety Committee wishes to acknowledge the contributions of the contributing authors to the Handbook. Special thanks to Lt. Col. David Alberico , USAF (RET), Air Force Safety Center, Chair- person of the JSSSSC, from 1995 to 1998, for his initial guidance and contributions in the development of the Handbook. The following authors wrote significant portions of the current Handbook: John Bozarth , CSP, EG&G Technical Services, Dahlgren, VA Michael Brown, Naval Surface Warfare Center, Dahlgren Division, (Chairperson, JSSSSC, 1998 to Present) Janet Gill, Naval Air Warfare Center, Aircraft Division, Patuxent River, MD Steven Mattern, Science and Engineering Associates, Albuquerque, NM Archibald McKinlay, Booz-Allen and Hamilton, St. Louis, MO Other contributing authors: Brenda Hyland, Naval Air Warfare Center, Aircraft Division, Patuxent River, MD Lenny Russo , U.S. Army Communication & Engineering Command, Ft. Monmouth, NJ The committee would also like to thank the following individuals for their specific contributions: Edward Kratovil, Naval Ordnance Safety and Security Activity, Indian Head, MD Craig Schilders , Naval Facilities Command, Washington, DC Benny Smith , U.S. Coast Guard, Washington, DC Steve Smith , Federal Aviation Administration, Washington, DC Lud Sorrentino, Booz-Allen and Hamilton, Dahlgren, VA Norma Stopyra, Naval Space and Warfare Systems Command, San Diego, CA Dennis Rilling, Naval Space and Warfare Systems Command, San Diego, CA Benny White , National Aeronautics and Space Administration, Washington, DC Martin Sullivan, EG&G Technical Services, Dahlgren, VA This Handbook is the result of the contributions of the above mentioned individuals and the extensive review comments from many others. The committee thanks all of the authors and the contributors for their assistance in the development of this Handbook. Software System Safety Handbook Table of Contents i TABLE OF CONTENTS 1. EXECUTIVE OVERVIEW 1–1 2. INTRODUCTION TO THE HANDBOOK 2–1 2.1 Introduction 2–1 2.2 Purpose 2–2 2.3 Scope 2–2 2.4 Authority/Standards 2–3 2.4.1 Department of Defense 2–3 2.4.1.1 DODD 5000.1 2–3 2.4.1.2 DOD 5000.2R 2–4 2.4.1.3 Military Standards 2–4 2.4.2 Other Government Agencies 2–8 2.4.2.1 Department of Transportation 2–8 2.4.2.2 National Aeronautics and Space Administration 2–11 2.4.3 Commercial 2–11 2.4.3.1 Institute of Electrical and Electronic Engineering 2–12 2.4.3.2 Electronic Industries Association 2–12 2.4.3.3 International Electrotechnical Commission 2–12 2.5 International Standards 2–13 2.5.1 Australian Defense Standard DEF(AUST) 5679 2–13 2.5.2 United Kingdom Defense Standard 00-55 & 00-54 2–14 2.5.3 United Kingdom Defense Standard 00-56 2–14 2.6 Handbook Overview 2–15 2.6.1 Historical Background 2–15 2.6.2 Problem Identification 2–15 2.6.2.1 Within System Safety 2–16 2.6.2.2 Within Software Development 2–17 2.6.3 Management Responsibilities 2–18 2.6.4 Introduction to the “Systems” Approach 2–18 2.6.4.1 The Hardware Development Life Cycle 2–19 2.6.4.2 The Software Development Life Cycle 2–20 2.6.4.3 The Integration of Hardware and Software Life Cycles 2–24 2.6.5 A “Team” Solution 2–25 2.7 Handbook Organization 2–26 2.7.1 Planning and Management 2–28 2.7.2 Task Implementation 2–28 2.7.3 Software Risk Assessment and Acceptance 2–29 2.7.4 Supplementary Appendices 2–29 3. INTRODUCTION TO RISK MANAGEMENT AND SYSTEM SAFETY 3–1 3.1 Introduction 3–1 3.2 A Discussion of Risk 3–1 Software System Safety Handbook Table of Contents ii 3.3 Types of Risk 3–2 3.4 Areas of Program Risk 3–3 3.4.1 Schedule Risk 3–5 3.4.2 Budget Risk 3–6 3.4.3 Sociopolitical Risk 3–7 3.4.4 Technical Risk 3–7 3.5 System Safety Engineering 3–8 3.6 Safety Risk Management 3–11 3.6.1 Initial Safety Risk Assessment 3–12 3.6.1.1 Hazard and Failure Mode Identification 3–12 3.6.1.2 Hazard Severity 3–12 3.6.1.3 Hazard Probability 3–13 3.6.1.4 HRI Matrix 3–14 3.6.2 Safety Order of Precedence 3–15 3.6.3 Elimination or Risk Reduction 3–16 3.6.4 Quantification of Residual Safety Risk 3–17 3.6.5 Managing and Assuming Residual Safety Risk 3–18 4. SOFTWARE SAFETY ENGINEERING 4–1 4.1 Introduction 4–1 4.1.1 Section 4 Format 4–3 4.1.2 Process Charts 4–3 4.1.3 Software Safety Engineering Products 4–5 4.2 Software Safety Planning Management 4–5 4.2.1 Planning 4–6 4.2.1.1 Establish the System Safety Program 4–10 4.2.1.2 Defining Acceptable Levels of Risk 4–11 4.2.1.3 Program Interfaces 4–12 4.2.1.4 Contract Deliverables 4–16 4.2.1.5 Develop Software Hazard Criticality Matrix 4–17 4.2.2 Management 4–21 4.3 Software Safety Task Implementation 4–25 4.3.1 Software Safety Program Milestones 4–26 4.3.1 Preliminary Hazard List Development 4–28 4.3.2 Tailoring Generic Safety-Critical Requirements 4–31 4.3.3 Preliminary Hazard Analysis Development 4–33 4.3.4 Derive System Safety-Critical Software Requirements 4–37 4.3.4.1 Preliminary Software Safety Requirements 4–39 4.3.4.2 Matured Software Safety Requirements 4–40 4.3.4.3 Documenting Software Safety Requirements 4–40 4.3.4.4 Software Analysis Folders 4–41 4.3.5 Preliminary Software Design, Subsystem Hazard Analysis 4–42 4.3.5.1 Module Safety-Criticality Analysis 4–45 4.3.5.2 Program Structure Analysis 4–45 4.3.5.3 Traceability Analysis 4–46 Software System Safety Handbook Table of Contents iii 4.3.6 Detailed Software Design, Subsystem Hazard Analysis 4–47 4.3.6.1 Participate in Software Design Maturation 4–48 4.3.6.2 Detailed Design Software Safety Analysis 4–49 4.3.6.3 Detailed Design Analysis Related Sub-processes 4–53 4.3.7 System Hazard Analysis 4–60 4.4 Software Safety Testing & Risk Assessment 4–63 4.4.1 Software Safety Test Planning 4–63 4.4.2 Software Safety Test Analysis 4–65 4.4.3 Software Standards and Criteria Assessment 4–69 4.4.4 Software Safety Residual Risk Assessment 4–71 4.5 Safety Assessment Report 4–73 4.5.1 Safety Assessment Report Table of Contents 4–74 A. DEFINITION OF TERMS A.1 Acronyms A-1 A.2 Definitions A-5 B. REFERENCES B.1 Government References B-1 B.2 Commericial References B-1 B.3 Individual References B-2 B.4 Other References B-3 C. HANDBOOK SUPPLEMENTAL INFORMATION C.1 Proposed Contents of the System Safety Data Library C-1 C.1.1 System Safety Program Plan C-1 C.1.2 Software Safety Program Plan C-2 C.1.3 Preliminary Hazard List C-3 C.1.4 Safety-Critical Functions List C-4 C.1.5 Preliminary Hazard Analysis C-5 C.1.6 Subsystem Hazard Analysis C-6 C.1.7 System Hazard Analysis C-6 C.1.8 Safety Requirements Criteria Analysis C-7 C.1.9 Safety Requirements Verification Report C-8 C.1.10 Safety Assessment Report C-9 C.2 Contractual Documentation C-10 C.2.1 Statement of Operational Need C-10 C.2.2 Request For Proposal C-10 C.2.3 Contract C-11 C.2.4 Statement of Work C-11 C.2.5 System and Product Specification C-13 C.2.6 System and Subsystem Requirements C-14 C.3 Planning Interfaces C-14 C.3.1 Engineering Management C-14 C.3.2 Design Engineering C-14 C.3.3 Systems Engineering C-15 Software System Safety Handbook Table of Contents iv C.3.4 Software Development C-16 C.3.5 Integrated Logistics Support C-16 C.3.6 Other Engineering Support C-17 C.4 Meetings and Reviews C-17 C.4.1 Program Management Reviews C-17 C.4.2 Integrated Product Team Meetings C-18 C.4.3 System Requirements Reviews C-18 C.4.4 SYSTEM/Subsystem Design Reviews C-19 C.4.5 Preliminary Design Review C-19 C.4.6 Critical Design Review C-20 C.4.7 Test Readiness Review C-21 C.4.8 Functional Configuration Audit C-22 C.4.9 Physical Configuration Audit C-22 C.5 Working Groups C-23 C.5.1 System Safety Working Group C-23 C.5.2 Software System Safety Working Group C-23 C.5.3 Test Integration Working Group/Test Planning Working Group C-25 C.5.4 Computer Resources Working Group C-25 C.5.5 Interface Control Working Group C-25 C.6 Resource Allocation C-26 C.6.1 Safety Personnel C-26 C.6.2 Funding C-27 C.6.3 Safety Schedules and Milestones C-27 C.6.4 Safety Tools and Training C-28 C.6.5 Required Hardware and Software C-28 C.7 Program Plans C-29 C.7.1 Risk Management Plan C-29 C.7.2 Quality Assurance Plan C-30 C.7.3 Reliability Engineering Plan C-30 C.7.4 Software Development Plan C-31 C.7.5 Systems Engineering Management Plan C-32 C.7.6 Test and Evaluation Master Plan C-33 C.7.7 Software Test Plan C-34 C.7.8 Software Installation Plan C-34 C.7.9 Software Transition Plan C-35 C.8 Hardware and Human Interface Requirements C-35 C.8.1 Interface Requirements C-35 C.8.2 Operations and Support Requirements C-36 C.8.3 Safety/Warning Device Requirements C-36 C.8.4 Protective Equipment Requirements C-37 C.8.5 Procedures and Training Requirements C-37 C.9 Managing Change C-37 C.9.1 Software Configuration Control Board C-37 Software System Safety Handbook Table of Contents v D. COTS AND NDI SOFTWARE D.1 Introduction D-1 D.2 Related Issues D-2 D.2.1 Managing Change D-2 D.2.2 Configuration Management D-2 D.2.3 Reusable and Legacy Software D-3 D.3 Applications of Non-Developmental Items D-3 D.3.1 Commercial-Off-the-Shelf Software D-3 D.4 Reducing Risks D-5 D.4.1 Applications Software Design D-5 D.4.2 Middleware or Wrappers D-6 D.4.3 Message Protocol D-7 D.4.4 Designing Around It D-7 D.4.5 Analysis and Testing of NDI Software D-8 D.4.6 Eliminating Functionality D-8 D.4.7 Run-Time Versions D-9 D.4.8 Watchdog Timers D-9 D.4.9 Configuration Management D-9 D.4.10 Prototyping D-10 D.4.11 Testing D-10 D.5 Summary D-10 E. GENERIC REQUIREMENTS AND GUIDELINES E.1 Introduction E-1 E.1.1 Determination of Safety-Critical Computing System Functions E-1 E.2 Design And Development Process Requirements And Guidelines E-2 E.2.1 Configuration Control E-2 E.2.2 Software Quality Assurance Program E-3 E.2.3 Two Person Rule E-3 E.2.4 Program Patch Prohibition E-3 E.2.5 Software Design Verification and Validation E-3 E.3 System Design Requirements And Guidelines E-5 E.3.1 Designed Safe States E-5 E.3.2 Standalone Computer E-5 E.3.3 Ease of Maintenance E-5 E.3.4 Safe State Return E-6 E.3.5 Restoration of Interlocks E-6 E.3.6 Input/output Registers E-6 E.3.7 External Hardware Failures E-6 E.3.8 Safety Kernel Failure E-6 E.3.9 Circumvent Unsafe Conditions E-6 E.3.10 Fallback and Recovery E-6 E.3.11 Simulators E-6 E.3.12 System Errors Log E-7 E.3.13 Positive Feedback Mechanisms E-7 Software System Safety Handbook Table of Contents vi E.3.14 Peak Load Conditions E-7 E.3.15 Endurance Issues E-7 E.3.16 Error Handling E-8 E.3.17 Redundancy Management E-9 E.3.18 Safe Modes And Recovery E-10 E.3.19 Isolation And Modularity E-10 E.4 Power-Up System Initialization Requirements E-11 E.4.1 Power-Up Initialization E-11 E.4.2 Power Faults E-11 E.4.3 Primary Computer Failure E-12 E.4.4 Maintenance Interlocks E-12 E.4.5 System-Level Check E-12 E.4.6 Control Flow Defects E-12 E.5 Computing System Environment Requirements And Guidelines E-14 E.5.1 Hardware and Hardware/Software Interface Requirements E-14 E.5.2 CPU Selection E-15 E.5.3 Minimum Clock Cycles E-16 E.5.4 Read Only Memories E-16 E.6 Self-Check Design Requirements And Guidelines E-16 E.6.1 Watchdog Timers E-16 E.6.2 Memory Checks E-16 E.6.3 Fault Detection E-16 E.6.4 Operational Checks E-17 E.7 Safety-Critical Computing System Functions Protection Requirements And Guidelines E-17 E.7.1 Safety Degradation E-17 E.7.2 Unauthorized Interaction E-17 E.7.3 Unauthorized Access E-17 E.7.4 Safety Kernel ROM E-17 E.7.5 Safety Kernel Independence E-17 E.7.6 Inadvertent Jumps E-17 E.7.7 Load Data Integrity E-18 E.7.8 Operational Reconfiguration Integrity E-18 E.8 Interface Design Requirements E-18 E.8.1 Feedback Loops E-18 E.8.2 Interface Control E-18 E.8.3 Decision Statements E-18 E.8.4 Inter-CPU Communications E-18 E.8.5 Data Transfer Messages E-18 E.8.6 External Functions E-19 E.8.7 Input Reasonableness Checks E-19 E.8.8 Full Scale Representations E-19 E.9 Human Interface E-19 E.9.1 Operator/Computing System Interface E-19 E.9.2 Processing Cancellation E-20 Software System Safety Handbook Table of Contents vii E.9.3 Hazardous Function Initiation E-20 E.9.4 Safety-Critical Displays E-21 E.9.5 Operator Entry Errors E-21 E.9.6 Safety-Critical Alerts E-21 E.9.7 Unsafe Situation Alerts E-21 E.9.8 Unsafe State Alerts E-21 E.10 Critical Timing And Interrupt Functions E-21 E.10.1 Safety-Critical Timing E-21 E.10.2 Valid Interrupts E-22 E.10.3 Recursive Loops E-22 E.10.4 Time Dependency E-22 E.11 Software Design And Development Requirements And Guidelines E-22 E.11.1 Coding Requirements/Issues E-22 E.11.2 Modular Code E-24 E.11.3 Number of Modules E-24 E.11.4 Execution Path E-24 E.11.5 Halt Instructions E-25 E.11.6 Single Purpose Files E-25 E.11.7 Unnecessary Features E-25 E.11.8 Indirect Addressing Methods E-25 E.11.9 Uninterruptable Code E-25 E.11.10 Safety-Critical Files E-25 E.11.11 Unused Memory E-25 E.11.12 Overlays Of Safety-Critical Software Shall All Occupy The Same Amount Of Memory E-26 E.11.13 Operating System Functions E-26 E.11.14 Compilers E-26 E.11.15 Flags and Variables E-26 E.11.16 Loop Entry Point E-26 E.11.17 Software Maintenance Design E-26 E.11.18 Variable Declaration E-26 E.11.19 Unused Executable Code E-26 E.11.20 Unreferenced Variables E-26 E.11.21 Assignment Statements E-27 E.11.22 Conditional Statements E-27 E.11.23 Strong Data Typing E-27 E.11.24 Timer Values Annotated E-27 E.11.25 Critical Variable Identification E-27 E.11.26 Global Variables E-27 E.12 Software Maintenance Requirements And Guidelines E-27 E.12.1 Critical Function Changes E-28 E.12.2 Critical Firmware Changes E-28 E.12.3 Software Change Medium E-28 E.12.4 Modification Configuration Control E-28 E.12.5 Version Identification E-28 [...]... software- intensive aeronautical and space systems for many years To support the required planning of software safety activities on these research and operational procurements, NASA published NASA Safety Standard (NSS) 1740.13, Interim, Software Safety Standard, in June 1994 “The purpose of this standard is to provide requirements to implement a systematic approach to software safety as an integral part... of more than 140 FAA Orders, standards, and other references FAA Order 8000.70 “FAA SYSTEM SAFETY PROGRAM” requires that the FAA SSP be used, where applicable, to enhance the effectiveness of FAA safety efforts through the uniform approach of system safety management and engineering principles and practices.3 3 FAA System Safety Handbook, Draft, December 31, 1993 2–8 Software System Safety Handbook Introduction... this handbook 2–5 Software System Safety Handbook Introduction to the Handbook Paragraph 4, General Requirements, 4.1, System Safety Program: The contractor shall establish and maintain a SSP to support efficient and effective achievement of overall system safety objectives Paragraph 4.2, System Safety Objectives: The SSP shall define a systematic approach to make sure that: (b.) Hazards associated... document a Software/ User Interface Analysis and the development of software user procedures Task 307, Software Change Hazard Analysis: The purpose of Task 307 is to require the contractor to perform and document a Software Change Hazard Analysis The contractor shall analyze all changes, modifications, and patches made to the software for safety hazards 2.4.1.3.2 MIL-STD-882C MIL-STD-882C, System Safety. .. emphasis from management as well as a continuing engineering analysis throughout the development and operational life cycles of the system This SSSH is a joint effort The U.S Army, Navy, Air Force, and Coast Guard Safety Centers, with cooperation from the Federal Aviation Administration (FAA), National Aeronautics and Space Administration (NASA), defense industry contractors, and academia are the primary... and industry, PMs and other acquisition managers shall develop a contracting approach appropriate to the type of system being acquired 2.4.1.2 DOD 5000.2R DOD 5000.2R, Mandatory Procedures for MDAPs and MAIS Acquisition Programs, March 15, 1996, provides the guidance regarding system safety and health 4.3.7.3 System Safety and Health: The PM shall identify and evaluate system safety and health hazards,... information provided in Table 2-1 demonstrated that the lack of any standardized approach for the accomplishment of software safety tasks that were levied contractually It also appeared as if the safety engineer either tried to accomplish the required tasks using a standard system safety approach, or borrowed the most logical tool available from the software development group In either case, they remained... events) shall be identified and appropriate actions taken to incorporate them in the software (and related hardware) specifications.” Task 202 is included as a representative description of tasks integrating software safety The general description is also applicable to all the other tasks specified in MIL-STD-882C The point is that software safety must be an integral part of system safety and software development... 2-1 graphically portrays the managerial element for the integrated team Program Management Software Engineering Systems Engineering Safety Engineering HW/ SW/ HF Figure 2-1: Management Commitment to the Integrated Safety Process 2.6.4 Introduction to the “Systems” Approach System safety engineering has historically demonstrated the benefits of a “systems” approach to safety risk analysis and mitigation... operate, maintain, or support the system Each management decision to accept the risks associated with an identified hazard shall be formally documented The Component Acquisition Executive (CAE) shall be the final approval authority for acceptance of high-risk hazards All participants in joint programs shall approve acceptance of high-risk hazards Acceptance of serious risk hazards may be approved at . SOFTWARE SYSTEM SAFETY HANDBOOK Joint Software System Safety Committee A Technical & Managerial Team Approach December 1999 This Handbook was funded. Naval Space and Warfare Systems Command, San Diego, CA Benny White , National Aeronautics and Space Administration, Washington, DC Martin Sullivan, EG&G