Evaluating and Improving Software Usability

Evaluating and
Improving
Software Usability
902 : Thursday, 9:30am - 10:45am

Philip Lew

www.xbosoft.com

Understand,
Evaluate
and

Improve

2

Agenda

•  Introduc7on

–  Importance
of
usability

–  What
is
usability?

–  What
is
UX?

•  Usability
Modeling
and
Measurements

•  Case
studies
on
measuring
and
improving

•  Summary

What
is
Quality?

Pertaining
to
soEware
J

Requirements…

4

Importance
of
Usability

Why
is
Usability
Important?

•  Usability
is
important
especially
for
web

applica7ons

–  And
now
especially
mobile
applica7ons

•  Saas,
mobile
are
in
alignment
with
and
soon

becoming
equivalent
to
cloud

•  Most
prevalent
implementa7on
method
in
the
next
5
years

•  Without
good
usability:

– Users
will
leave
the
applica7ons

– For
mobile,
if
they
can’t
learn
in
30
seconds,

they
won’t
come
back

Web
and
Mobile
have
Changed
the

Terrain

•  Business
models

have
changed

–  Instead
of
paying

upfront
and

‘owning’
the

soEware

–  Pay
as
you
go,
pay

by
subscrip7on

•  Behavior
and

expecta7ons
have

changed

What
is
Usability

In
rela7on
to
quality

In
rela7on
to
user
experience

When
you
think
Usability…

•  Naviga7on

•  Eﬃciency

•  Responsiveness-‐performance

•  Learnability

•  AZrac7veness
Anything

else
come
to

•  Understandability

mind?

Current
Research
–
Usability

User
Quality
Experience

Satisfaction Usability

ISO 9241-11
ISO
9126-1

Usability
Design ISO
Experts Product 25010
Owners

Usability
-‐
Key
Characteris7c
of

Product
Quality

Source: ISO 25010

Quality
in
Use

ISO 25010

Notes
on
Sa7sfac7on
and
Usability

don’t have
what I want I’m unsatisfied •  Sa7sfac7on
is
a

subjec7ve
feeling

Highly usable dependent
on
many

software things
other
than

usability:

My password
doesn’t work –  A
user
can
be
highly

sa7sﬁed
but
the

applica7on
with
low

Nice weather I’m usability.

today satisfied! –  An
applica7on
can
be

! highly
usable
(high

low usability
software usability)
but
the
user
is

not
sa7sﬁed!

Finished
my work today

Usability-‐Major
Component
of
Quality

Quality
and
Usability
in
the
so1ware
development
lifecycle

Usability

Usability

•  Can
be
measured
from
the
design
point
of

view
or
of
the
product

•  Can
be
measured
‘in-‐use’
with
real
users

Evolu7on
of
SoEware
quality

ISO 9000 ISO 9126 ISO 9241 ISO 25010

Software Software Software
Type of Process
Quality Quality Quality
quality quality
(internal) (external) (in use)

What is
measured
Software
processes
Code Product ?
CMMI
How
measured?
assessment
model
white box
testing
black box
testing ?
CMMI
Who Assessment Programmer Tester End User
measures?
Company

What
is
Usability-‐Eﬀect
of
the

SoEware
Product

Degree to which specified
users can achieve specified
goals with effectiveness,
efficiency and satisfaction in a
specified context of use.

Source: ISO 25010

Usability
in
Actual
Usage

•  User
role
specified
users

•  Objec7ve

specified
goals

•  Task

•  Environment

specified
context
of
use

•  Domain

•  …
What
else
can
you
think
of?

Usability and
User
Experience
UX, Usability and Quality in Use
•  Difficult to understand their relationships
•  Lack of consensus in meaning in literature
•  Not totally clear how related to quality
•  Standard definition for user experience is still not available

Some
UX
Deﬁni7ons

UX DESCRIPTION
D1 a continuous process of user engagement with the product [10]
D2 entire set of affects that results in user-product interaction [11]
D3 the evolution of usability [12]
D4 elaboration of the satisfaction component of usability [13]
D5 a categorization of “do-goals” (pragmatics) and “be-goals”(hedonics) [3], [14]

D6 infinite small experiences relating to people, products and contexts [6]
D7 consequence of user’s and product’s characteristics when interacted in a
specific environment [1]
D8 degree to which specified users can achieve actual usability, safety, and
satisfaction in use in a specified context of use [15]
D9 A person's perceptions and responses that result from the use or anticipated use
of a product, system or service [16]

D10 the degree of positive or negative emotions that can be experienced by a
specific user in a specific context during and after product use and that
motivates for further usage [17]

User Experience (UX)
•  Pragmatic (do-goals) : refers to the product's
perceived ability to support the task achievement
and focuses on the product’s utility and usability in
completing tasks, the ‘do-goals’ of the user.
•  Hedonic (be-goals) : product's ability to support
the user’s achievement of ‘be-goals’, such as
being satisfied, pleasure.
•  Real goal of the end user is fulfilling be-goals such
as being autonomous, competent, related to
others, stimulated, etc.”
•  Be-goals are the driver of UX
Hassenzahl

Rela7ng
Usability
and
User
Experience

Quality
508
User
Interface

Func7onality
Learnability

characteris7c
compliancy
Usability

Product
Text/Sound
Appropriate

Easy
to
use
Easy
to
learn

aZributes
aZributes
Func7ons

UX
Pragma7c

Effec7ve
and
Efficient

Do
Goals

Usability

Effec7veness
in
use
Efficiency
in
use
Learnability
in
use

Measurement

UX
Hedonic

Sa7sfac7on
(pleasure,
sense
of
community…)

Be
Goals

Sa7sfac7on
Sense
of

Pleasure
Comfort
Trust

Measurement
Community

Context

USER
EXPERIENCE

21

Now
that
we
understand
it
ALL…

We
Need
to
Deﬁne
Usability
For
Our

Own
Purposes

22

Let’s
Deﬁne
Usability/UX

From
the
Product
Viewpoint

Usability

Characteris7c
1

Naviga7on
Understandable

Characteris7c
2 Characteris7c
n

Subcharacteris7c
1

Control
Stability
Subcharacteris7c
2

Previous-‐Next
Subcharacteris7c

n

AZribute
1

Home
Loca7on
AZribute
2

Save
Loca7on
Ease
of
ﬁnding
AZribute
n

Defining
Usability
from
an

Effect-‐Real
usage
Point
of
View

Usability
Quality

Effec7veness
Efficiency
Sa7sfac7on
Characteris7c
n

Accuracy
Completeness
Subcharacteris7c

n

Errors
AZribute
2
AZribute
3
AZribute
1

Some
Poten7al
AZributes

Measurements
for
‘in
use’
Usability

•  Eﬀec9veness

–  Comple7on
rates
Can
you
think

–  Error
rate
of
any
others,

–  Help
usage
par9cular
to

•  Eﬃciency
your

applica9on?

–  Task
7me

–  Backtracking

•  Learnability

–  Learning
rate

–  Task
7me
devia7on

Adding
in
UX

Sa9sfac9on
in
use

•  Degree
to
which
users
are
sa7sﬁed
in
a
speciﬁed

context
of
use.
Sa7sfac7on
is
the
user’s
response
to

interac7on
with
the
product,
including
amtudes

towards
use
of
the
product.
Subdivided
into
sub-‐
characteris7cs:

–  Likability
(cogni7ve
sa7sfac7on)
UX Component
–  Pleasure
(emo7onal
sa7sfac7on)

–  Comfort
(physical
sa7sfac7on)

–  Trust

26

Example
of
Product
Quality
Model

with
AZributes

(Operability in ISO 25010)

External
Quality
Requirements
(for

Shopping
Cart
Entity)

1 Usability

1.1 Understandability

1.1.1 Icon/label
ease
to
be
recognized

1.1.2 Information
grouping
cohesiveness

1.2 Learnability

1.2.1 ………………………………………………………..

1.3 Ease
of
Use

1.3.1 Control
permanence

1.3.2 Helpfulness

Learnability

•  Degree
to
which
the
soEware
product
enables
users
to

learn
its
applica7on
-‐
7
principles
(ISO
9241-‐10)

–  Suitability
for
the
task
-‐
should
be
suitable
for
the
user’s

task
and
skill
level

–  Self-‐descrip7veness-‐
should
be
clear
what
the
user

should
do
next

–  Controllability

-‐
the
user
should
be
able
to
control
the

pace
and
sequence
of
the
interac7on

–  Conformity
with
user
expecta7ons
-‐
should
be

consistent

–  Error
tolerant
-‐
forgiving

–  Suitability
for
individualiza7on
-‐
should
be
able
to
be

customized
to
suit
the
user

–  Suitability
for
learning
-‐
should
support
learning

28

Ease
of
use

•  Degree
to
which
the
soEware
product

makes
it
easy
for
users
to
operate
and

control
it.

– controllability

– error
tolerance
(by
operator)

– conformity
with
user
expecta7ons

29

Ease
of
Use
-‐
Helpfulness

•  Degree
to
which
the
soEware
product

provides
help
when
users
need

assistance
including
help
that
is:

– easy
to
ﬁnd

– comprehensive

– eﬀec7ve

30

Example:
Usability
Measurement

ACribute
Scale How Calcula9on Goal Current

Help
Percent

of
Compare
% 90% 40%
completeness Menu
items
menus
and

with

help
items
help
Ease
of
Keystrokes
Sample
50
average 3.5 12
access to
items
ﬁnd/use
a

feature/
func7on/
informa7on
Consistency Number
Examine
integer 1 5
loca7ons
for
menus
and

same
buZon doc.

What types of usability measures are these?
31

Using
a
measurement
model

Evalua7on

Usability

composed of Indicator

Metrics

Evalua7on

Evalua7on

Characteris7c

Measurement

composed of Func7on

Evalua7on

Evalua7on
Measurements

Sub-‐characteris7cs

32

Usability-‐Review

usability
usability
in
use

UX

•  Can
be
measured
from
the
design
point
of

view
or
of
the
product

•  Can
be
measured
‘in-‐use’
with
real
users

Measuring
Usability

Methods
and
models

Now
that
we
know
what
it
is,
and

what
to
measure,
then
HOW?

34

Measurable
AZributes

•  Usability
characteris7c

•  Descrip7on
and
purpose
Once you have a
model (what you
•  How
to
measure
are going to
•  What
is
measured
measure), then
•  Measurement/Calcula7on
you start doing IT!
•  Range
(min,
max)

•  Metric

•  Objec7ve
(goal)
versus
Current
(indicator)

UX/Usability

Measurement
Methods

Focus

groups

Ques9on
naires

Observa9on

Heuris9c
Labs

Evalua9on

Logging

Let’s
get
started

Measurement
methods

Expert
Evalua9on
(Heuris9c
evalua9on)

•  Deﬁne
a
set
of
rules
or
criteria
with

measurements
and
evaluate
against
them

Web-‐based
logs

•  Collect
user
ac:vity
data

– Mistakes
and
errors

– How
long
it
takes
to
complete
tasks

– Comple:on
rates

Ques9onnaires

•  Quan7ta7ve
subjec7ve
measurement
of
UX

characteris7cs
37

Heuris7c
Evalua7on

Using
a
checklist

But
a
liZle
more
complicated
than

yes/no

38

Heuristic Evaluation of a Pharmacy
application
Each usability attribute has a quantifying metric.
•  For
example,
for
Predic9ve
textual
aCribute,
users
should
be
able
to

understand
a
buCon’s
results
prior
to
pressing
it.

•  Direct
metrics
need
to
be
designed,
i.e.:

–  (0)
No
support
at
all

–  (1)
Par9al

–  (2)
Complete

•  Need
a
mapping
from
0,
1,
2
to
something
more
understandable,
i.e.
2
=

100,
and
1
=
60

•  Need
an
indicator
to
interpret
the
level
of
sa9sfac9on
met
with
decision

criteria
with
acceptability
ranges
in
a
percentage
scale:

–  0-‐40
(unsa9sfactory
–red)
means
changes
must
take
place
with
high

priority;

–  40-‐70
(marginal
–yellow)
indicates
a
need
for
improvement
ac9ons;

–  70-‐100
indicates
a
sa9sfactory
level
–green-‐
for
the
analyzed

aCribute.

39

2.2.2 Error Recovery Support
" In the current state, users filling a new prescription are
supported well in error recovery (2.2.2) in automatic cursor
positioning

40

Heuristic Evaluation – Pharmacy Software
The purpose of the evaluation
•  Understand the external quality level of the learnability sub-characteristic for
filling a new prescription

Table 2 - excerpt of the whole current evaluation 41

Heuris7c
Evalua7on
of
Usability-‐
Shopping
Cart

External Quality Requirements Measure EI value P/GI value
Global Quality Indicator 61.97%
1 Usability 60.88%
1.1 Understandability 83%
1.1.1 Icon/label ease to be recognized 100%
1.1.2 Information grouping cohesiveness 66%
1.2 Learnability 51.97%
1.2.1 ……………………………………………… …
1.3 Operability 49.50%
1.3.1 Control permanence 100%
1.3.2 Expected behaviour 50%
2 Content Quality 63.05%
2.1 Content Suitability 63.05%
2.1.1 Basic Information Coverage 50%
2.1.1.1 Line item information completeness 2 50%
2.1.1.2 Product description appropriateness 50%
2.1.2 Coverage of other Contextual Information 76.89%
2.1.2.1 ……………………………………………….. …
2.1.2.2 Return policy information completeness 33%

JIRA Heuristic Usability Evaluation

Usability
Logging

Collec7ng
user
ac7vity
and
behavior

Tradi7onally
used
for
other
things

like…

44

Usability
Logging

•  Iden7fy
users
by
using
session

ID
to
iden7fy
a
unique
user.

•  Itera7vely
insert
code
into
the

applica7on

•  Collect
data

•  Analyze
the
data
for
each

aZribute
in
diﬀerent

dimensions
and
aggrega7ons

•  Determine
the
need
for
further

calcula7ons
and
what
aZributes

to
measure
further

•  Revise
the
data
we
are

collec7ng,
adding
or
decreasing

granularity

Some
easy
and
some
hard

46

Jira Evaluation Summary

47

Using
Both

Product
Perspec7ve
and
the
User
Perspec7ve

•  If
we
ﬁnd
some
aZribute
with
low

performance
from
the
user
perspec7ve

•  Then
we
can
go
back
to
the
product

perspec7ve,
ﬁgure
out
why,
and
improve

48

JIRA Evaluation by Attribute
Excerpt

49

For
example:Login
BuZon
Visibility

(product
usability
aZribute)

50

Using
Ques7onnaires

The
process:

1. Use
your
model
of
what
you
want
to
measure

and
improve

2. Design
the
ques7onnaire
according
to
the

model

3. Execute
the
ques7onnaire

4. Analysis

Ques7onnaires
Used

SUMI
(So1ware
Usability
Measurement
Inventory
)

QUIS
(Ques:onnaire
for
User
Interface
Sa:sfac:on
)

PSSUQ
(Post-‐Study
System
Usability
Ques:onnaire
)

ASQ
(A1er-‐Scenario
Ques:onnaire
)

PUTQ
(Purdue
Usability
Tes:ng
Ques:onnaire
)

PHUE
(Prac:cal
Heuris:cs
for
Usability
Evalua:on
)

SUS
(System
Usability
Scale
)

IUI
(Isometrics
Usability
Inventory)

What
do
you
do
with
all
this?

56

Itera7ve
Improvement

•  Measure
external
quality
–
usability
with

heuris7cs

•  Measure
in-‐use
usability
with
logging
or

ques7onnaires

Goal
is
to
Understand,
Evaluate
and
Improve

58

Next
Steps

•  Produce
an
ac7on
plan

–  What
usability
aZributes
are
important
to
your

organiza7on?

•  Develop
a
model

–  What
data
can
you
collect/Which
technique
can

you
use

•  Maybe
some
elements
of
the
model
drop
out-‐can’t
be

measured
that
easily

•  Start
collec7ng
and
developing
benchmarks

Conclusion

•  Usability
and
UX
are
both
abstract
concepts

–  Cri7cal
component
of
quality

•  Deﬁning
is
diﬀerent
for
each
organiza7on

•  Need
a
model
for
your
organiza7on

•  The
model
is
the
founda7on
of
what
to

measure

•  Once
you
can
measure,
then
you
can
evaluate

and
improve

Resources

These are all journal publications that can
be accessed via these links… 61

Thanks

Ques7ons
and
Answers

Please
ﬁll
out
an
evalua:on
form
and
drop
it
in
the

collec:on
basket
located
at
the
back
of
the
room.

Philip
Lew

@xbosoE

philip.lew@xbosoE.com

phone:
408-‐350-‐0508

www.xbosoE.com

Evaluating and Improving Software Usability

Recommended

More Related Content

Viewers also liked (20)

Similar to Evaluating and Improving Software Usability (20)

More from XBOSoft (20)

Recently uploaded (20)

Evaluating and Improving Software Usability