TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again

World®
’16
Tech Talk: Going Full Circle - Linking
Code to Tests to Requirements and
Back Around Again
James Walker – Principal Software Engineer – CA Technologies
DO5T05
DEVOPS

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
Dependency mapping links requirements, code and tests, and is central to good development
practice.
This session will show:
§ How CA Agile Requirements Designer can be used to create a three way bind between the
requirements, code and tests.
§ How formal modeling can be used to create a full dependency map of even the most complex
systems, while introducing traceability between the design, code and tests.
With full dependency mapping, when a change is made to a given component in the
requirements, its impact across the entire system can be identified, informing the developers of
exactly what needs changing and where. Tests can be automatically generated from the formal
model, and can be updated automatically to validate that this change has been successfully
implemented in the code.
When another change is made, round it goes again…
James
Walker
CA Technologies
Principal Software
Engineer

SOFTWARE DEVELOPMENT CHALLENGES
MODEL BASED TESTING (MBT)
FEEDBACK DRIVEN TESTING
CONCLUSION
1
2
3
4
Agenda

About Me – James Walker
§ BSc, MRes, PhD – Swansea
University, Wales
§ Research in Data Visualisation /
Big Data problems
§ Grid-Tools – Software Engineer
(2012 – 2015)
§ CA Technologies – Lead Engineer
on CA ARD (2015)

Maybe you won’t quite get to enjoy your weekend yet!

Typical Project
Requirements
Design
Development
QA
Release
Analyst & Customer
Architect
Developer
Tester
Release

Waterfall

Problematic...
§ Inflexible (linear)
§ Few business systems have stable requirements
§ Long release cycle
– Expensive
– Fail late

Waterfall Project
Requirements
Design
Development
QA
Release
Analyst & Customer
Architect
Developer
Tester
Release
Months / Years
Winston Royce – demonstrated the
waterfall as a criticism for how not to
develop software!

Continuous Delivery

Continuous Delivery Project
Requirements
Design
Development
QA
Release (MVP)
Months / Years
Requirements
Design
Development
QA
Release (MVP)
Requirements
Design
Development
QA
Release (MVP)
Weeks

In Theory This is Great…
§ Each time we test the deliverable
§ Highly adaptive to change
§ Deliver faster
§ Earlier feedback (fail fast)

In Reality…
Hero culture — Full throttle
to save the project

The Problem:
A lack of clarity and vision during each iteration
Business Analyst Programmer TesterUser
The User Knows what they want
The Analyst specifies what that is
The Programmer writes the code
The Tester tests the program
The further the visions diverge

Traditional testing methods are too slow, manual, and linear for
Continues delivery, compromising speed and quality
0% 10% 20% 30% 40% 50% 60% 70%
Time/resources in test data compliance (PII)
Defects stemming from ambiguous requirements
Testing innefficiencies leading to higher cost
Lack of test coverage creating defects/rework
Difficulty finding the right data for a particular test
Manual testing leading to project delays
What's the biggest problem you face?

1 – Poor Requirements
§ A plethora of techniques exist, most are
written in ambiguous natural language
§ The requirements are “static” - they offer
no way to derive tests directly from
them…
§ … no way to update tests when the
requirements change
– this has to be done manually
Bottlenecks

2 – Manual Test Case Design
§ Currently manual – a time consuming, error-prone process
§ Is unsystematic, ad hoc, and has no real notion of “coverage”
§ Over-testing and under-testing – 10-20% coverage with 4
times over-testing
§ Poor requirements lead to poor overall testing, with testers
having to fill in the gaps
§ No linkage to test data – process is manual, painstaking and
very time-consuming
§ No flexibility for change requests: a critical weakness in an
agile or continuous delivery environment. Changes take
longer than the original requirement.

3 – Poor Testing -> Automated testing
Automated testing: Manual script generation
§ Automated testing frameworks are heavily scripted
§ Script generation is usually done manually
§ As well as the maintenance of scripts
§ Alternative solutions use:
§ Record playback
§ Script-less automation frameworks (keyword)
But you’re still doing manual test case design!

The Vision
Clarity and Vision During Development
Business Analyst Programmer TesterUser
There are less bugs and the product is delivered faster
The closer the vision means the user gets a quality product

Model Based Testing

Model based testing let’s you define the behavior of a system
under test – in other words what is supposed to happen.
What is Model Based Testing?
Model
System
The model is then used to test the actual system to see if it does
what it is meant to.

Model Requirements as an “Active” Flowchart
§ A formal model that is accessible to the
business who already use VISIO, BPM,
etc.
§ Which is also a mathematically precise
model of a system, so that it eliminates
ambiguity and incompleteness
§ It can be used by testers and
developers – it brings the end-user,
business and IT into close alignment

New Delivery Paradigm: From Ideation to Design,
Development and Testing

Testing With the “Active” Flowchart
§ Testers can overlay the
flowchart with all the
functional logic and
data involved in a
system
§ Tests can therefore be
automatically derived
from it

§ Exhaustively test a model – extract all possible routes from start -> end
§ Each route / path becomes a test case
§ Number of routes grows exponentially with every added decision
32 nodes + 62 edges = 1,073,741,824 possible routes
Generating Test Cases
2145 Paths

§ Coverage, in most cases, is related to how much functionality is being
covered in a test case.
§ Traverse the model by satisfying coverage constraints:
Optimization / Coverage
19 PathsEdge Pairs
9 PathsIn Out Edges
5 PathsEdges
3 PathsNodes

Overlaying Automation
§ Test automaters overlay
code snippets onto
processes which correspond
to the code which performs
each action
§ Automation scripts can
therefore be automatically
derived from it

Provide Full Traceability With Requirements
§ Know what needs to be re-tested and when the integrity of a
system is at risk… “If I change this, what will I break?”
§ The impact of a change made to an individual component is
identified system wide
§ The impact on test
cases and user stories
up and down a system
can also be identified
automatically

DEMO
Visit our booth for more!

Feedback Loops – The Future

Run Results
Typical Build Process
Execution Engine
Build Engine
Development /
Code
Different visual
representations of
Pass / Fail results.
Traffic Lights Dashboarding Email (Jenkins)

Scenario
Testing – Gray area

Needle in a Haystack Without the Right Infrastructure

Testing Bermuda Triangle
Code
RequirementsTest Cases
The further apart these are the less quality
• Model represents the requirements
• Code is implemented using the model (can
add links to specific objects / methods).
• Test cases are generated from the model
Using these we obtain a close binding between
all three.
Full traceability enables solving the previous
problems efficiently

The Future is Feedback Loops (New World Order)
Execution Engine
Build Engine
Development /
Code
Model
Run results

Scenario Revisit – What does the pass/fail represent?
Test results bound directly to their routes through
the system.

Scenario Revisit – Exploratory testing
After a failure the model can be used to generate me more test
cases by pin point testing– test cases focused on discovering more
knowledge about the failure.

Scenario Revisit – Risk Analysis
Use meta-deta assigned to the model to perform risk analysis. Are
the effected routes through the system above a set threshold?
= 63% of User Activity
Can’t Release

Scenario Revisit – Getting it Fixed!
We’ve found the process(es) responsible – now we can trace back
to the responsible teams!
Development
Pinpointed Failure
Defect created

Conclusion

Why do I need the code, requirements, and tests
strongly bound?
§ A shifting IT landscape has transformed the role of testers –
testing is now cross-functional and iterative, with an emphasis
on feedback throughout
§ New test and development emerged, but the “hero culture”
remained…
§ …Along with the challenges of slow, manual testing,
ambiguous requirements and “tools hops”

The Vision: Shift Left, Shift Right – Overlapping Feedback Loops
§ Shift Left: work iteratively with users to gain upfront
knowledge of the requirements and use these to drive activity
throughout the development cycle
§ Shift Right: connect-up toolstacks and use (test) analytics to
harness the insights of testing, production and operations
§ Feed these insights back into the model, and the whole
process goes back round again so you can enjoy your
weekend
Why do I need the code, requirements, and tests
strongly bound?

Recommended Sessions
SESSION # TITLE DATE/TIME
DO5X27S What’s New in CA Agile Requirements Designer 11/17/2016 at 12:45 pm
DO5X29S
Model-Based Testing for Achieving Maximum CD Pipeline
Acceleration
11/17/2016 at 3:00 pm
DO5X40S Case Study: Why Rabobank Was Inspired 11/17/2016 at 3:45 pm

Stay connected at communities.ca.com
Thank you.

DevOps – Continuous Delivery
For more information on DevOps – Continuous Delivery, please
visit: http://cainc.to/PiTFpu

TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again (20)

More from CA Technologies (20)

Recently uploaded (20)

TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again