EXTENT-2016: Managing QA for Complex Systems in Agile Development Framework
Download as ppt, pdf2 likes1,514 views
Alexey Zverev, CEO of Exactpro, presented on testing complex systems within agile development frameworks. The presentation outlined fundamental principles of QA management, including continuously learning about the system and improving test design through iterations. It then discussed common "anti-patterns" such as over-relying on requirements traceability matrices or test plans. Finally, it summarized Exactpro's principles, such as developing team knowledge, versus these anti-patterns.
More Related Content
What's hot (20)
Viewers also liked (18)
Similar to EXTENT-2016: Managing QA for Complex Systems in Agile Development Framework (20)
More from Iosif Itkin (20)
Recently uploaded (20)
EXTENT-2016: Managing QA for Complex Systems in Agile Development Framework
- 1. Alexey Zverev CEO Exactpro Testing Complex Systems within Agile Development Framework
- 2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com2 QA management at Exactpro Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com Alexey Zverev, Co-CEO, Exactpro
- 3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com3 Introduction
- 4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com4 Agenda 1. Fundamental principles of QA Management 2. The anti-patterns 3. Summary of our principles vs. the anti-patterns 4. Questions & Answers
- 5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com5 Fundamental principles of QA Management QA is a continuous learning and development process 1. Learn how the system should operate in production 2. Learn about the actual behaviour 3. Make sure all discrepancies are either fixed or documented 4. Develop Test Design, Test Tools and an Automated Test Library through a series of small iterations This is a permanent process throughout the delivery cycle and after the actual Go-Live, whilst building the test tools adds more efficiency, and therefore, we find as many issues as possible.
- 6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com6 The anti-patterns 1. “The Requirement Traceability Matrix” 2. “Test Automation” 3. “The Test Library” 4. “The Test Plan” 5. “Non-Functional Testing” 6. “The Test Tool” 7. “The Scrum Team”
- 7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com7 “The Requirements Traceability Matrix” “We need to write Test Cases at the beginning of the project” “All Test Cases must contain direct links to Requirements” “Stakeholders must see test coverage and correspondence between tests cases and software requirements”
- 8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com8 “Test Automation” “We must automate 100% of our Test Library. Let’s start now” “Complete regression must happen overnight as part of the build process” “As a result of test automation we should reduce the QA team to a few people”
- 9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com9 “The Test Library” “We must document all our test scenarios and put them into Test Management Software” “Test Library consists of Test Suites which consist of Test Scenarios which consist of Test Cases” “Once we accomplish this we will start test automation” “Exploratory Testing? What is it? No we don’t do it” “All test cases contain detailed test steps and Testers must put Pass/Fail as they execute them”
- 10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com10 “The Test Plan” “Let’s estimate QA effort required to deliver this project” “Lets plan QA tasks and start implementing them in waterfall model” “Let’s monitor process closely and put our effort into achieving targets and milestones”
- 11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com11 “Non-Functional Testing” “We need to test that the system supports 10,000 orders per second” “We need to make sure that latency is within expected limits” “We will allocate production-like hardware to focus on capacity and latency measurements”
- 12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com12 “The Test Tool” “We have purchased a licence for a magical test automation tool” “We will use it for all of our test automation purposes” “It allows interacting with GUI / via FIX / via SWIFT and has the capacity to record/replay test scenarios” “It is expensive”
- 13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com13 “The Scrum Team” “We are using scrum and will test everything as a part of the sprint” “Testing is done and automated by Developers who can temporarily allocate their time within scrum” “We will put some QA engineers into Scrum team and let them write unit tests” “We are using DevOps we don’t need QA, QA is dead”
- 14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com14 Summary of our principles vs. the anti-patterns 1. Developing team knowledge vs. “The Requirement Traceability Matrix” 2. Building software to test software vs. “Test Automation” 3. Test Design vs. “The Test Library” 4. Agile iterative approach vs. “The Test Plan” 5. Testing at the confluence of Functional and Non-Functional Testing vs. “Non-Functional Testing” 6. Pragmatic in-house development of test tools vs. “The The Test Tool” 7. The QA team is a pacemaker vs. “The Scrum Team”
- 15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com15 Questions & Answers Thank you!
Editor's Notes
- #8: Formal approach to testing dramatically decreases the test coverage. Testing is limited to and is only as good as the requirements document. QA is dispensable and does not add value. Test scenarios must be based on deep knowledge of the system acquired by QA, not on some formal and outdated requirements document. Outcome: Test coverage is inadequate and insufficient.
- #9: Change is part of the software development process. Attempts to develop a large automated test library against volatile software is doomed, unless we invest in proper design and implement it through various iterations. The most efficient test automation will happen when the software under test is stable enough. Outcome: A lot of effort is wasted on test automation and the desired outcome is not achieved.
- #10: Adequate test coverage of a complex (Trading/Post Trade) system assumes tens of thousands test cases. Investing in documenting and maintaining them is inefficient. Outcome: The team maintains a small test library based on their documenting capability. Test coverage is limited. Test process limits the speed of delivery and quality.
- #11: There is no straightforward way to plan QA. We are dealing with unknown from the beginning to the very end. The test team must be agile and deal with software changes keeping strategic focus on final project delivery. Outcome: Inadequate management. Focus on tasks that do not lead to successful software delivery.
- #12: The separation of approaches to functional and non-functional testing leads to the fact that the vast area of test conditions is not covered. QA must examine carefully how the system behaves under load. This should not be limited to latency/performance measurements. Rather, the system behavior should be verified in detail. Outcome: Many issues “under the load with unspecified conditions” occur in production.
- #13: In our complex environment, GUI represents only a tiny part of the test universe. Record/Replay scenarios are unreliable and unsupportable. For vast majority of tools, our work on test automation helps overcome test tool limitations. Generic testing tools add little value to test automation. Outcome: Inefficient spend of money and resources. The test tool paradigm limits the test approach.
- #14: Financial technology at the current stage can’t afford going into production without QA. The cost of a software fault is too high. Independent and thorough QA is a requirement. Outcome: No integration testing. Insufficient testing by someone who understands business. High risk of critical faults in Production.