More Related Content
What's hot (20)
Viewers also liked (20)
Similar to Revised IEEE 1633 Recommended Practices for Software Reliability (20)
Recently uploaded (20)
Revised IEEE 1633 Recommended Practices for Software Reliability
- 1. Advantages of IEEE 1633 Recommend Practices for Software Reliability Chair: Ann Marie Neufelder, SoftRel, LLC Vice Chair: Martha Wetherholt, NASA Secretary: Debra Haehn, Philips IEEE Standards Association Chair: Louis Gullo, Raytheon Missile Systems Division 1
- 2. Software reliability timeline 1960’s 1970’s 1980’s 1990’s 1962 First recorded system failure due to software Many software reliability growth estimation models developed. Limitation– can’t be used until late in testing. 1968 The term “software reliability” is invented. First predictive model developed by USAF Rome Air Development Center with SAIC and Research Triangle Park – Limitations– model only useful for aircraft and never updated after 1992. SoftRel, LLC models based on RL model. Can be used on any system. Updated every 4 years. 2000’s 2 Martin Trachtenberg notices the “bell curve” Larry Putnam/QSM quantifies the bell curve used for both scheduling and staffing
- 3. Introduction and motivation • Software reliability engineering • Has existed for over 50 years. • Fundamental prerequisite for virtually all modern systems • Rich body of software reliability research generated over last several decades, but… • Practical guidance on how to apply these models has lagged significantly • Diverse set of stakeholders requires pragmatic guidance and tools to apply software reliability models to assess real software or firmware projects during each stage of the software development lifecycle • Reliability engineers may lack software development experience • Software engineers may be unfamiliar with methods to predict software reliability • Both may have challenges acquiring data needed for the analyses 3
- 4. Abstract • Newly revised IEEE 1633 Recommended Practice for Software Reliability provides actionable step by step procedures for employing software reliability models and analyses • During any phase of software or firmware development • With any software lifecycle model for any industry or application type. • Includes • Easy to use models for predicting software reliability early in development and during test and operation. • Methods to analyze software failure modes and include software in a system fault tree analysis. • Ability to assess the reliability of COTS, FOSS, and contractor or subcontractor delivered software. • This presentation will cover the key features of the IEEE 1633 Recommended Practices for software reliability. • Current status of this document - Approved by IEEE Standards Association Ballot of May 24, 2016 4
- 5. Acknowledgement of IEEE 1633 Working Group members • Lance Fiondella • Peter Lakey • Robert Binder • Michael Siok • Ming Li • Ying Shi • Nematollah Bidokhti • Thierry Wandji • Michael Grottke • Andy Long • George Stark • Allen Nikora • Bakul Banerjee • Debra Greenhalgh Lubas • Mark Sims • Rajesh Murthy • Willie Fitzpatrick • Mark Ofori-kyei 5 • Sonya Davis • Burdette Joyner • Marty Shooman • Andrew Mack • Loren Garroway • Kevin Mattos • Kevin Frye • Claire Jones • Robert Raygan • Mary Ann DeCicco • Shane Smith • Franklin Marotta • David Bernreuther • Martin Wayne • Nathan Herbert • Richard E Gibbs III • Harry White • Jacob Axman • Ahlia T. Kitwana • Yuan Wei • Darwin Heiser • Brian McQuillan • Kishor Trivedi Chair: Ann Marie Neufelder, SoftRel, LLC Vice Chair: Martha Wetherholt, NASA Secretary: Debra Haehn, Philips IEEE Standards Association Chair: Louis Gullo, Raytheon Missile Systems Division
- 6. IEEE 1633 Working Group • Defense/aerospace contractors – 11 members • Commercial engineering – 9 members • US Army – 6 members • US Navy – 5 members • Academia – 4 members • DoD – 3 members • NASA – 3 members • Medical equipment – 2 members • Software Engineering Institute – 1 member • Nuclear Regulatory Commission – 1 member 6
- 7. Table of contents Section Contents 1,2,3 Overview, definitions and acronyms 4 Tailoring guidance 5 “Actionable” Procedures with Checklists and Examples 5.1 Planning for software reliability. 5.2 Develop a failure modes mode 5.3 Apply SRE during development 5.4 Apply SRE during testing 5.5 Support Release decision 5.6 Apply SRE in operation Annex A Supporting information on the software FMEA Annex B Detailed procedures on predicting size and supporting information for the predictive models Annex C Supporting information for the software reliability growth models Annex D Estimated cost of SRE Annex E SRE tools Annex F Examples 7
- 8. Section 4 SRE Tailoring 8 • The document is geared towards 4 different roles, any industry and any type of software. • Hence, section 4 provides guidance for tailoring the document. • By role – recommended sections if you are a reliability engineer, software QA, software manager or acquisitions. • By life cycle - How to apply the document if you have an incremental or agile life cycle model. • By criticality – Some SR tasks are essential, some are typical and some are project specific.
- 9. Typical tasks by role 9
- 10. Section 5.1 Planning for software reliability 10
- 11. Planning • An often overlooked but essential step in SRE 11 Topic Description Characterize the software system What are the Line Replaceable Units? (Applications, executables, DLLs, COTS, FOSS, firmware, glueware) Which are applicable for SRE? What is the operational profile? Define failures and criticality There is no one definition fits all. Failures need to be defined relative to the system under development. Perform a reliability risk assessment Determine a simple Red/Yellow/Green SRE risk. Use that to determine the degree of SRE. Assess the data collection system The available data and SRE tools will determine which tasks are feasible Review the available SRE tools Finalize the SRE plan The Software Reliability Program Plan can be part of the Software Development Plan or the Reliability Plan or a standalone document
- 12. Section 5.2 Develop Failure ModesAnalysis 12
- 13. Section 5.2 Develop Failure ModesAnalysis 13 • This section focuses on the 3 analyses that identify potential failure modes. • Understanding the failure modes is essential for development, testing, and decision making. Real examples are included in the document. • Perform Defect Root Cause Analysis (RCA) • Perform Software Failure Modes Effects Analysis (SFMEA) • Prepare the SFMEA • Analyze Failure Modes and Root Causes • Identify consequences • Mitigate • Generate a Critical Items List (CIL) • Understand the differences between a hardware FMEA and a software FMEA • Include Software in the System Fault Tree Analysis
- 14. SFMEA and SFTA Viewpoints These are complementary methods 14
- 15. Software defect root cause analysis • The RCA ensures that any SRE improvement efforts address the right types of defects. • Example, if most of the defects are introduced in the design phase, you don’t want to put all of your SRE effort into improving coding practices. • Software reliability assessment identified certain gaps and strengths which can lead to a certain “volume” of defects • But, a root cause analysis can confirm the “types” of defects • Faulty requirements? • Faulty design? • Faulty implementation? • Faulty interfaces? • Faulty changes or corrective actions? • Faulty source and version control? • These can and will be unique for each configuration item even if they have the same development processes Copyright © SoftRel, LLC 2011. This presentation may not be copied in part or in whole without written permission from Ann Marie Neufelder.
- 16. Example of a root cause analysis Defects are introduced because of either bad requirements, bad design, bad coding practices or bad change control. • Requirements defect – The “whats” are incorrect, ambiguous or incomplete. • Design defect – The “whats” are correct but the “hows” are not. Logic, state, timing, exception handling are all design related. • Coding defect- The “whats” and “hows” are correct but the software engineer did not implement one or more lines of code properly. 16Copyright © SoftRel, LLC 2011. This presentation may not be copied in part or in whole without written permission from Ann Marie Neufelder.
- 17. Section 5.3 Apply SRE during development 17
- 18. Section 5.3 Apply SRE during development Tasks Description 1. Determine/obtain system reliability objectives in terms of reliability, availability, MTBF Today’s system are software intensive. This makes it difficult to establish a reasonable system objective. This document provides 3 approaches for this. 2. Perform software reliability assessment and prediction See upcoming slides 3. Sanity check the early prediction One reason why SRE prediction models haven’t be used is that reliability engineers are unsure of the results. The document has typical reliability values based on the size of the software. 4. Merge the predictions into the over system prediction Once the predictions are done, the reliability engineer will want to integrate them into the overall system RBD or fault tree. The document has several methods for doing so. 5. Determine the total software reliability needed to reach the objective Since software engineering is often managed centrally, the software manager will want to know what the software components as an aggregate need to achieve. 18
- 19. Section 5.3 Apply SRE during development 6. Plan the reliability growth needed to reach the system objective Once the software objective is established, plans can and should be made to ensure that there is sufficient reliability growth in the schedule. Reliability growth can only happen if the software is operated in a real environment with no new feature drops. 7. Perform a sensitivity analysis Quite often there isn’t sufficient schedule for extended reliability growth so a sensitivity analysis is needed to determine how to cut the defects to reach the objective. 8. Allocate the required objective to each software LRU If the software components are managed by different organizations or vendors, the software level objective will need to be further allocated. 9. Employ software reliability metrics There are other metrics that can support decision making, testing and delivery that also support more reliable software. 19
- 20. Section 5.3.2 Perform software reliability assessment and prediction 20 • Since the 1970s most of the software reliability models are usable only during test or operation when it’s too late to do planning, tradeoffs, improvements. • The models presented in this section can be used for the code is even written. The predictions are then merged with the hardware reliability predictions and compared to the system reliability objective.
- 21. If you can predict this fault profile you can predict all of the other reliability figures of merit The predictive models predict the fault profile first and then then failure rate, MTBF, reliability and availability is predicted from that 21
- 22. Section 5.4 Apply SRE during testing 22
- 23. Section 5.3 Apply SRE during development Tasks Description 1. Develop a reliability test suite Software reliability growth models are useless unless the software is being exercised. The first step is to make sure that it is. 2. Measure test coverage The models can’t measure what they don’t know. The higher the test coverage, the higher the confidence in the models. 3. Increase test effectiveness via fault insertion Many software reliability issues are due to the software performing an unexpected function as opposed to it failing to perform a required function. This increases the confidence in the reliability. 4. Collect failure and defect data All of the models require the testing/operational hours and either the time of each failure observation or the total number of failures in a day. 5. Select and use reliability growth models Before you use any model, you need to plot the failure data and see which models are applicable. The document provides complete guidance on how to do this. 23
- 24. Section 5.3 Apply SRE during development Tasks Description 6. Apply SRE metrics Certain metrics provide information about the maturity of the software which are essential for decision making and planning of resources. 7. Determine accuracy of the models The failure trend can change at any time during testing. Hence, the best model can change with it. The best way to measure accuracy is to compare the estimations to the next time to failure. 8. Support release decision The release decision should not be made solely based on the SRG models. The decision is based on the test coverage and approach, degree of fault insertion, other SRE metrics which can indicate troubled releases as well as the SRG models. 24
- 25. Section 5.4Apply SRE during testing 25 • Software reliability growth models have existed since the 1970s • Many of them provide nearly identical results • SWRG models have been difficult to implement and understand due to poor guidance from academic community • Several models assume data which is not feasible to collect on non-academic large software systems This document provides • Models that are feasible for real software systems • Consolidation of models that provide similar results • Step by step instructions for how to select the best model(s) based on • The observed defect discovery trend (see next slide) • Inherent Defect Content • Effort required to use the model(s) • Availability of data required for the model(s) • How to apply them when you have an incremental life cycle • Test coverage methods which affect the accuracy of all SWRG models
- 26. Selecting the best SWRG model • Most important criteria is the current defect discovery trend. • A few models can be used when the discovery rate is increasing or peaking. Most can be used when decreasing or stabilizing. • If the defect discovery is plotted first, the user will know which models can be used 26 0 2 4 6 8 10 12 NonCumulativedefects discovered Normalized usage period Increasing Peaking Decreasing Stabilizing
- 27. Section 5.5 Support Release Decision 27
- 28. Section 5.5 Support Release Decision 28 Once the development and testing is complete the SRE analyses, models and metrics can be used to determine whether a decision should be accepted • Decision is based on • Requirements and Operational Profile coverage • Stress test coverage • Code coverage • Adequate defect removal • Confidence in reliability estimates • SRE Tasks performed prior to acceptance • Determine Release Stability – do the reliability estimates meet the objective? • Forecast additional test duration – If the objective hasn’t been met how many more test hours are required? • Forecast remaining defects and effort required to correct them – Will the forecasted defects pile up? Impact the next release? • Perform a Reliability Demonstration Test – Determine statistically whether the software meets the objective
- 29. Section 5.6Apply SRE in Operations 29 Once the software is deployed the reliability should be monitored to assess any changes needed to previous analyses, predictions and estimations
- 30. Section 5.6 Apply SRE in Operations Tasks Description 1. Employ SRE metrics to monitor software reliability The best way to improve the accuracy of the predictions and SWRG models is to measure the actual software reliability once in operation.2. Compare operational and predicted reliability 3. Assess changes to previous characterizations or analyses The operational failure modes may be different than what’s visible in testing. If so, the software failure modes analyses will need to focus on the operational failure modes to improve the reliability of the next release. 4. Archive operational data Operational data is valuable for future predictions, sanity checking, etc. 30
- 31. Summary • IEEE P1633 2016 puts forth recommended practices to apply qualitative software failure modes analyses and qualitative models • Improve product and ensure software or firmware delivered with required reliability • IEEE P1633 2016 includes improved guidance • Offers increased value more accessible to a broader audience • Reliability engineers • Software quality engineers • Software managers • Acquisitions 31