GUI Performance Regression Testing with Eggplant Functional

GUI Performance Regression Testing
with Eggplant Functional
Comparing GUI response times across versions of an
application
October 9, 2019

GUI Performance Regression Testing with
Eggplant
John Crouch
Sr. Software
Development
Engineer in Test
Advanced Analytics
Division
SAS Institute Inc.

GUI Performance Regression Testing with Eggplant
Topics to be discussed
• How we used Eggplant to automate GUI performance
measurements of SAS® Enterprise Miner
• How we used Base SAS, SAS® Graph, and Excel to analyze the
performance measurements and detect regressions
• How you can use Eggplant and SAS to measure and analyze any
GUI performance regressions in your applications-under-test

Overview: Comparing the GUI response times of EM 6.2 and EM 7.1
Dell M600
SAS®
Enterprise
Miner Server
6.2
Production
release
perf_Dave_71
operates EM
6.1 and 7.1
Eggplant
Functional
Dell M600
SAS®
Enterprise
Miner Server
7.1
Future
release
Eggplant
Identical
hardware & OS
Same GUI,
different mid-tier
Base SAS®
Read
SAS reads
transaction logs,
compares
response times

How we used Eggplant to automate GUI
performance measurements of SAS®
Enterprise Miner

Mid-tier architecture changed for EM 7.1
The mid-tier architecture (AP) of EM 6 was replaced with a
new architecture (WIP) in the EM 7.1 release (2011).
AP replaced by WIP
The EM 7.1 GUI maintained consistency with the EM
6.2 GUI, so Eggplant script didn’t need updating.
GUI did not change
GUI performance was impacted for some actions,
unaffected for other actions.
GUI response times

Mid-tier architecture changed, GUI performance was impacted for some user
actions
SAS® Enterprise
Miner Server 6.2
Production
release
SAS® Enterprise
Miner Server 7.1
Future
release
Identical hardware
Identical GUIs
Test Engineer
?
7.1 is
slower
than 6.2?
AP mid-tier WIP mid-tier

Automatically measuring GUI response times
with Eggplant is more precise and repeatable
Production
release
Eggplant script
for GUI
response time
measurement
Eggplant
Functional
Future
release
Identical hardware
Identical GUIs
SAS® Enterprise
Miner Server 6.2
AP mid-tier
SAS® Enterprise
Miner Server 7.1
WIP mid-tier
Response
times for
both
releases

Example: Measuring response times for EM 6.2 and EM 7.1 with Eggplant

Example: Measuring response times for EM 6.2
and EM 7.1 with Eggplant
Production
release
perf_Dave_71
script operates
EM 6.1 or EM 7.1
Eggplant
Functional
Future
release
Identical
hardware & OS
Same GUI,
different mid-tier
EM 6.2 release
response times
used as
benchmark
SAS® Enterprise
Miner Server 7.1
WIP mid-tier
SAS® Enterprise
Miner Server 6.2
AP mid-tier

Step 1
Production
release
perf_Dave_71
script records
EM 6.2 response
times
Eggplant
Functional
Future
release
Eggplant
1
Run script five times
back-to-back, discard
first “priming” run.
EM 6.2
Timed
transactions
SAS® Enterprise
Miner Server 6.2
AP mid-tier
SAS® Enterprise
Miner Server 7.1
WIP mid-tier

Step 2
Production
release
perf_Dave_71
script records
EM 7.1 response
times
Eggplant
Functional
Future
release
Eggplant
2
Run script five times
back-to-back, discard
first “priming” run.
EM 7.1 and EM
6.2 timed
transactions
SAS® Enterprise
Miner Server 6.2
AP mid-tier
SAS® Enterprise
Miner Server 7.1
WIP mid-tier

Step 3
Production
release
Response
times,
EM 6.2 and 7.1
Eggplant
Functional
Future
release
Eggplant
3 SAS reads
transactions,
averages
elapsed
times
Base SAS®
Read
SAS® Enterprise
Miner Server 6.2
AP mid-tier
SAS® Enterprise
Miner Server 7.1
WIP mid-tier

Step 4
Eggplant
4
SAS compares
the average
elapsed times
and produces
reports
Base SAS® and
SAS ® Graph
transaction 6.2 Baseline 7.1 April 13 port 7.1 May 01 port 7.1 May 11 port* Increase from 6.2 %Increase from 6.2
(13) Create new diagram 1.16712 6.19601 4.5954 4.81995 3.65283 312.978
(12) Open vars list on existing data
source GV20K
1.03403 2.98677 10.5974 3.62339 2.58936 250.415
(17) Open 10-node diagram 1.20288 3.94484 3.0964 2.74017 1.53729 127.801
(1) Logon to EM 3.02938 5.65935 6.604 5.62787 2.59848 85.776
(11) Show list of vars (basic advisor) for
Gv20k
3.19564 2.92113 13.9113 5.56068 2.36504 74.008
(2) Get list of projects 1.32258 2.22838 2.0087 2.27316 0.95059 71.874
(15) Open vars list on data source
GV20K in PFD
5.49412 6.21249 14.5413 7.91207 2.41795 44.01
(14) Drop datasource GV20K on diagram 0.98933 1.88833 1.3084 1.3104 0.32107 32.454
(10) Browse metadata for Gv20k 1.19314 0.90545 0.9806 1.55011 0.35697 29.919
(9) Browse SAS libraries 0.29294 0.38905 0.3415 0.34864 0.05571 19.017
(8) Finish project wizard 3.99 3.676 3.9656 3.99184 0.00184 0.046
(7) Open new project wizard 1.56957 1.52354 1.5579 1.52143 -0.04814 -3.067
(4) Open existing project 1.84596 1.60142 1.6077 1.54665 -0.29931 -16.214
(6) Expand list of 10 diagrams 1.46395 1.53491 1.5827 1.21466 -0.24929 -17.029
(5) Expand list of 10 data sources 1.48421 1.12748 1.5438 1.15796 -0.32625 -21.981
(3) Open empty project 2.97574 2.04638 2.2042 2.29152 -0.68422 -22.993
(16) Open (2) existing project 2.56115 1.3623 1.2774 1.2515 -1.30965 -51.135
Comparison of EM 6.2 and EM 7.1 GUI Response Times
Report prepared 17May2011
Green = 7.1 response time within 1.0 sec of 6.2 response time, or faster
* 11may port + EM7.1 catalogs and ear file updates
transaction ResponseTime131 ResponseTime132 Change Pct_Change
(5) Run IGN Node 17.664 17.4406 -0.2234 -1.2647
(6) Launch Interactive Groupingwindow 5.798 5.6752 -0.1228 -2.118
(4) ImportdiagramIGN Diagram 6.252 6.0528 -0.1992 -3.1862
(3) Finish projectwizard 3.828 3.4632 -0.3648 -9.5298
(1) Logon to EM 8.9086 7.535 -1.3736 -15.4188
(11) Open SplitBin 0.246 0.2058 -0.0402 -16.3415
(17) Open Results window 6.438 4.9954 -1.4426 -22.4076
(2) Open new projectwizard 1.32 0.967 -0.353 -26.7424
Green =13.2is fasterthan 13.1
Red italics =13.2is slowerthan 13.1
Gray =13.2is within 0.5seconds of 13.1
Comparing Mean EM GUI Response Times (sec), IGN 13.1 vs. IGN 13.2
WX6single machine, June 11weekly port
0 2 4 6 8 10 12 14
(5)Run IGNNode
(6)Launch Interactive Groupingwindow
(4)ImportdiagramIGNDiagram
(3)Finish project wizard
(1)Logon to EM
(11)Open SplitBin
(17)Open Resultswindow
(2)Open new projectwizard
Mean EM Response Times, IGN 13.1 vs. IGN 13.2
ResponseTime132 ResponseTime131

Sample summary table output by SAS, viewed
in Excel
source GV20K
1.03403 2.98677 10.5974 3.62339 2.58936 250.415
(1) Logon to EM 3.02938 5.65935 6.604 5.62787 2.59848 85.776
Gv20k
3.19564 2.92113 13.9113 5.56068 2.36504 74.008
GV20K in PFD
5.49412 6.21249 14.5413 7.91207 2.41795 44.01

What is a “timed
transaction”? Elapsed time measured while
a User action completes, plus
a unique label for the action
Timing begins immediately after user
action has been taken, and ends
immediately after expected visual state
is detected
Descriptive label and the elapsed time are written
to a log file as a “timed transaction”.

What does a
timed transaction
look like? Timed GUI transaction log is C:UsersPublicEMEM132tranlogsrun_perf_Dave_132JWS_715929.log
TRANSACTION END: (1) Logon to EM
Seconds elapsed: 8.19
(start time: 09:30:08 end time: 09:30:16)
TRANSACTION END: (2) Get list of projects
TRANSACTION END: (3) Open project Empty Project
(…)
Timed
transactions

How is a
transaction timed
in Sensetalk?
Eggplant script measures the
time elapsed between an
action and detection of the
resulting state.
Detection of the resulting visual state is
typically achieved when a waitfor
command succeeds.
Elapsed time for the transaction includes the time it
takes Eggplant to detect that the desired visual
state has displayed. Use searches that are fast and
reliable!
(* ============ Get list of projects -============*)
// Open Project list w/keyboard shortcut (Ctl-O)
typetext ControlKey, "o“
// *** START TIMER ***
put the long time into startTime
// Wait up to <maxwait> secs for Open Project dialog to display
waitfor maxwait, openprojTB_props
// *** STOP TIMER ***
put the long time into endTime
put endTime - startTime into secondsElapsed

Adding Code: Creating a “Get list of
projects” timed transaction for EM

Step 1: Document the action->
visual state pair to adapt for
timed transaction generation
Note: Scenario is run three times, once for each data source: sampsio.dmagecr, sasdata.gv10k, and
sasdata.gv20k.
(1) Logon to EM
(Browse to JWS launch URL with IE version 8; not timed)
(Click Launch link to display EM logon dialog; not timed)
(Type in userid and password; not timed)
Click Logon button - start timer
EM Welcome window displays - stop timer
(2) Get list of projects
Type Control-o in Welcome window - start timer
Open Project dialog displays - stop timer
(3) Open empty project
(Open Projects dialog is already displayed - not timed)
(Browse to/click on empty project named "Empty Project"; not timed)
Click OK button- start timer
Project window opens with "Empty Project" in the window title bar - stop timer
(Type Escape, click Yes to close project window, return to Welcome screen; not timed)
(4) Open existing project
(Click "Recent Projects..." link; not timed)
(Recent Projects dialog displays; not timed)
(Browse to/click on project named "Existing Project"; not timed)
Click Open button - start timer
Project window opens with "Existing Project" in the window title bar - stop timer
o (5) Expand list of 10 data sources (…)
“Get list of projects”
transaction documented

Step 2: Locate the code for the
action->visual state transition
pair to be timed.
Action: type command to open a project
Visual state transition: Open Project dialog
displayed
Container holding text property list, or image name

Step 3: Add a line of code to
record transaction start time

Step 4: Add a line of code to
record transaction end time
Container with elapsed time
for “Get list of projects”
transaction

Step 5: Add a line of code
to calculate elapsed time
Container with elapsed time
for “Get list of projects”
transaction

Step 6: Add a call to
write_transaction helper
script to write log file entry
(* ============ TRANSACTION BEGIN: Get list of projects -============*)
write_transaction "Get list of projects“
(* ============ TRANSACTION END: Get list of projects -============*)
“Get list of projects” identifies
the transaction in the log file

How we used Base SAS, SAS® Graph, and
Excel to analyze the performance
measurements and detect regressions

Comparing GUI response times of EM 6.2 and
EM 7.1
Dell M600
SAS® Enterprise
Miner Server
6.2
Production
release
perf_Dave_71
operates EM
6.1 and 7.1
Eggplant
Functional
Dell M600
SAS® Enterprise
Miner Server
7.1
Future
release
Identical
hardware & OS
Same GUI,
different mid-
tier
Base SAS®
Read
SAS reads
transaction logs,
compares
response times

Comparing GUI response times of EM 6.2 and
EM 7.1
SAS analyzes
timed
transactions
and produces
reports
Base SAS® and
SAS ® Graph
source GV20K
1.03403 2.98677 10.5974 3.62339 2.58936 250.415
(1) Logon to EM 3.02938 5.65935 6.604 5.62787 2.59848 85.776
Gv20k
3.19564 2.92113 13.9113 5.56068 2.36504 74.008
GV20K in PFD
5.49412 6.21249 14.5413 7.91207 2.41795 44.01
transaction ResponseTime131 ResponseTime132 Change Pct_Change
(5) Run IGN Node 17.664 17.4406 -0.2234 -1.2647
(6) Launch Interactive Groupingwindow 5.798 5.6752 -0.1228 -2.118
(4) ImportdiagramIGN Diagram 6.252 6.0528 -0.1992 -3.1862
(3) Finish projectwizard 3.828 3.4632 -0.3648 -9.5298
(1) Logon to EM 8.9086 7.535 -1.3736 -15.4188
(11) Open SplitBin 0.246 0.2058 -0.0402 -16.3415
(17) Open Results window 6.438 4.9954 -1.4426 -22.4076
(2) Open new projectwizard 1.32 0.967 -0.353 -26.7424
Green =13.2is fasterthan 13.1
Red italics =13.2is slowerthan 13.1
Gray =13.2is within 0.5seconds of 13.1
Comparing Mean EM GUI Response Times (sec), IGN 13.1 vs. IGN 13.2
WX6single machine, June 11weekly port
0 2 4 6 8 10 12 14
(5)Run IGNNode
(6)Launch Interactive Groupingwindow
(4)ImportdiagramIGNDiagram
(3)Finish project wizard
(1)Logon to EM
(11)Open SplitBin
(17)Open Resultswindow
(2)Open new projectwizard
Mean EM Response Times, IGN 13.1 vs. IGN 13.2
ResponseTime132 ResponseTime131

Comparing GUI response times of EM 6.2
and EM 7.1
Report samples

First results: Comparing
response times of EM 6.2 and
latest EM 7.1 (January 2011)
transaction AvgSecs_62 AvgSecs_71 delta_time pct_change
2.16933 6.435 4.2657 196.63
Open 10-node diagram 1.57252 4.3193 2.7467 174.67
Browse SAS libraries 0.3609 0.7247 0.3638 100.81
Browse metadata for Gv20k 0.27637 0.2921 0.0157 5.68
Create new diagram 0.83825 6.4997 5.6615 675.39
Drop datasource GV20K on diagram 0.96025 1.7009 0.7407 77.13
Expand list of 10 data sources 1.35761 2.7702 1.4126 104.05
Expand list of 10 diagrams 1.31047 1.6497 0.3392 25.89
Finish project wizard 4.14054 5.2697 1.1292 27.27
Get list of projects 1.3384 1.7077 0.3693 27.6
Open (2) existing project 2.33873 4.5881 2.2494 96.18
Open empty project 3.5416 5.5488 2.0072 56.67
Open existing project 2.44553 3.8007 1.3552 55.42
Open new project wizard 1.64075 1.7911 0.1504 9.17
Open vars list on data source GV20K in PFD 5.18312 17.9217 12.7386 245.77
Open vars list on existing data source GV20K 1.73096 28.4409 26.7099 1543.07
Show list of vars (basic advisor) for Gv20k 4.10379 14.9194 10.8156 263.55
Logon to EM 3.73888 7.4502 3.7113 99.26
Transactions
color-coded red
have a duration
increase > 1
second

Comparing response times of
EM 6.2 and latest EM 7.1
(February 2011)
transaction AvgSecs_62 AvgSecs_71jan AvgSecs_71feb AvgSecs_71feb16 Delta_Time_Feb16 Pct_Change_Feb16
source GV20K
1.73096 28.4409 21.6774 14.1214 12.3904 715.815
Gv20k
4.10379 14.9194 10.9226 12.9908 8.887 216.556
GV20K in PFD
5.18312 17.9217 12.1577 14.5302 9.347 180.336
2.16933 6.435 5.4048 4.9666 2.7972 128.945
(16) Open (2) existing project 2.33873 4.5881 4.226 4.3746 2.0358 87.049
(1) Logon to EM 3.73888 7.4502 7.2873 6.3392 2.6003 69.547
(4) Open existing project 2.44553 3.8007 4.0909 3.9476 1.5021 61.422
(3) Open empty project 3.5416 5.5488 6.0189 4.8271 1.2855 36.298
(2) Get list of projects 1.3384 1.7077 1.5648 1.2839 -0.0545 -4.071
(10) Browse metadata for Gv20k 0.27637 0.2921 0.2948 0.2381 -0.0383 -13.847
Transaction
s color-
coded
green are
faster in 7.1

Eggplant
Comparing response times
of EM 6.2 and latest EM 7.1
(March 2011)
transaction AvgSecs_62 AvgSecs_71feb16b AvgSecs_71mar Delta_Time_Feb16B Pct_Change_Feb16B Delta_Time_Mar Pct_Change_Mar
(13) Create new diagram 1.16712 5.8765 5.68811 4.7094 403.507 4.52099 387.363
(12) Open vars list on existing data source GV20K 1.03403 11.2664 3.75914 10.2323 989.562 2.72512 263.544
(17) Open 10-node diagram 1.20288 3.3529 3.77933 2.15 178.742 2.57646 214.191
(14) Drop datasource GV20K on diagram 0.98933 1.9407 1.99974 0.9514 96.168 1.01041 102.131
(1) Logon to EM 3.02938 5.6343 6.10537 2.6049 85.988 3.07599 101.538
(2) Get list of projects 1.32258 1.4098 2.18333 0.0872 6.595 0.86076 65.082
(4) Open existing project 1.84596 3.603 3.03086 1.7571 95.185 1.1849 64.189
(3) Open empty project 2.97574 5.4774 3.97601 2.5016 84.067 1.00028 33.614
(16) Open (2) existing project 2.56115 3.6289 2.99641 1.0678 41.692 0.43526 16.995
(11) Show list of vars (basic advisor) for Gv20k 3.19564 9.6949 3.59428 6.4992 203.378 0.39864 12.474
(8) Finish project wizard 3.99 5.5273 4.21278 1.5373 38.529 0.22278 5.583
(9) Browse SAS libraries 0.29294 0.2867 0.30597 -0.0062 -2.124 0.01303 4.449
(15) Open vars list on data source GV20K in PFD 5.49412 11.9352 5.37287 6.4411 117.236 -0.12125 -2.207
(10) Browse metadata for Gv20k 1.19314 0.544 1.15352 -0.6492 -54.409 -0.03962 -3.321
(7) Open new project wizard 1.56957 1.5795 1.4088 0.0099 0.63 -0.16077 -10.243
(6) Expand list of 10 diagrams 1.46395 1.2665 1.14976 -0.1974 -13.487 -0.31419 -21.462
(5) Expand list of 10 data sources 1.48421 1.3112 1.08627 -0.173 -11.655 -0.39794 -26.811

(April 2011)
transaction AvgSecs_62 AvgSecs_71 AvgSecs_71fe
b
AvgSecs_71feb1
6
AvgSecs_71feb16
b
AvgSecs_71mar AvgSecs_71mar28 AvgSecs_71apr22 Delta_Time_Ap
r
Pct_Change_Apr
(13) Create new diagram 1.16712 6.4997 6.9669 6.8649 5.8765 5.68811 6.19172 6.19601 5.02889 430.881
(17) Open 10-node diagram 1.20288 4.3193 4.2113 3.3389 3.3529 3.77933 4.06042 3.94484 2.74196 227.95
source GV20K
1.03403 28.4409 21.6774 14.1214 11.2664 3.75914 2.72971 2.98677 1.95274 188.848
(14) Drop datasource GV20K on diagram 0.98933 1.7009 2.1326 1.9564 1.9407 1.99974 1.79106 1.88833 0.89901 90.871
(1) Logon to EM 3.02938 7.4502 7.2873 6.3392 5.6343 6.10537 7.92037 5.65935 2.62997 86.815
(2) Get list of projects 1.32258 1.7077 1.5648 1.2839 1.4098 2.18333 2.59777 2.22838 0.90581 68.488
(9) Browse SAS libraries 0.29294 0.7247 0.676 0.667 0.2867 0.30597 0.39295 0.38905 0.09612 32.811
GV20K in PFD
5.49412 17.9217 12.1577 14.5302 11.9352 5.37287 5.35508 6.21249 0.71838 13.075
(6) Expand list of 10 diagrams 1.46395 1.6497 1.5069 1.2585 1.2665 1.14976 2.3259 1.53491 0.07096 4.847
(7) Open new project wizard 1.56957 1.7911 1.7614 1.2146 1.5795 1.4088 1.53749 1.52354 -0.04604 -2.933
(8) Finish project wizard 3.99 5.2697 5.2588 5.1282 5.5273 4.21278 4.48749 3.676 -0.314 -7.87
Gv20k
3.19564 14.9194 10.9226 12.9908 9.6949 3.59428 3.82022 2.92113 -0.27451 -8.59
(4) Open existing project 1.84596 3.8007 4.0909 3.9476 3.603 3.03086 1.46727 1.60142 -0.24455 -13.248
(5) Expand list of 10 data sources 1.48421 2.7702 1.1273 1.3504 1.3112 1.08627 1.96531 1.12748 -0.35673 -24.035
(10) Browse metadata for Gv20k 1.19314 0.2921 0.2948 0.2381 0.544 1.15352 0.87731 0.90545 -0.28769 -24.112
(3) Open empty project 2.97574 5.5488 6.0189 4.8271 5.4774 3.97601 6.26567 2.04638 -0.92936 -31.231
(16) Open (2) existing project 2.56115 4.5881 4.226 4.3746 3.6289 2.99641 1.31758 1.3623 -1.19886 -46.809

(May 2011)
transaction AvgSecs_62 AvgSecs_71apr22 ResponseTime ResponseTime17may ResponseTime26may Change_May26 Pct_Change_May26
(13) Create new diagram 1.16712 6.19601 4.5954 4.81995 3.52644 2.35932 202.149
source GV20K
1.03403 2.98677 10.5974 3.62339 2.83246 1.79843 173.925
(17) Open 10-node diagram 1.20288 3.94484 3.0964 2.74017 2.4967 1.29382 107.561
(2) Get list of projects 1.32258 2.22838 2.0087 2.27316 2.61403 1.29146 97.647
(1) Logon to EM 3.02938 5.65935 6.604 5.62787 5.67484 2.64546 87.327
(14) Drop datasource GV20K on diagram 0.98933 1.88833 1.3084 1.3104 1.24863 0.25931 26.21
(9) Browse SAS libraries 0.29294 0.38905 0.3415 0.34864 0.36269 0.06975 23.81
Gv20k
3.19564 2.92113 13.9113 5.56068 3.44163 0.24599 7.698
(7) Open new project wizard 1.56957 1.52354 1.5579 1.52143 1.53624 -0.03333 -2.123
GV20K in PFD
5.49412 6.21249 14.5413 7.91207 5.36437 -0.12974 -2.362
(6) Expand list of 10 diagrams 1.46395 1.53491 1.5827 1.21466 1.39719 -0.06676 -4.56
(4) Open existing project 1.84596 1.60142 1.6077 1.54665 1.59813 -0.24783 -13.425
(10) Browse metadata for Gv20k 1.19314 0.90545 0.9806 1.55011 1.00388 -0.18926 -15.862
(3) Open empty project 2.97574 2.04638 2.2042 2.29152 2.47698 -0.49876 -16.761
(8) Finish project wizard 3.99 3.676 3.9656 3.99184 3.20736 -0.78264 -19.615
(5) Expand list of 10 data sources 1.48421 1.12748 1.5438 1.15796 1.11667 -0.36754 -24.763
(16) Open (2) existing project 2.56115 1.3623 1.2774 1.2515 1.38933 -1.17183 -45.754
Comparison of GUI Response Times, EM 6.2 versus EM 7.1
Latest Response Times Measured 26May

EM 7.1 was followed by
seven more releases in four
years
The mid-tier architecture (WIP) first used in EM 7.1 is used
in the EM releases that followed over the next 4 years:
12.1m0, 12.1m1, 12.3, 13.1, 13.2, 14.1, 14.3
WIP mid-tier in
use
The EM GUI continued to maintain consistency with
EM 7.1, and Eggplant script didn’t need updating.
GUI still did not
change!
The same core Eggplant scripts ran successfully
against 7 new releases of EM that followed EM 7.1.
Eggplant consistency,
reliability

2012: Comparing response times of EM 7.1 to
response times of EM 12.1m0
SAS®
Enterprise
Miner Server
7.1
Production
Elapsed time
measurement
script and
transaction logs
Eggplant
Functional
SAS®
Enterprise
Miner Server
12.1m0
Development
Parse logs
and
compare
response
times
Base SAS®
Production
release response
times used as
benchmark

2013: Comparing response times of EM 12.1m0
to response times of EM 12.1m1
SAS®
Enterprise
Miner Server
12.1m0
Production
Elapsed time
measurement
script and
transaction logs
Eggplant
Functional
SAS®
Enterprise
Miner Server
12.1m1
Development
Parse logs
and
compare
response
times
Base SAS®
Production
release response
times used as
benchmark

2013: Comparing response times of production EM
12.1m1 to response times of EM 12.3
SAS®
Enterprise
Miner Server
12.1m1
Production
Elapsed time
measurement
script and
transaction logs
Eggplant
Functional
SAS®
Enterprise
Miner Server
12.3
Development
Parse logs
and
compare
response
times
Base SAS®
Production
release response
times used as
benchmark

Eggplant
2014: Comparing response times of production EM
13.1 to response times of EM 13.2
SAS®
Enterprise
Miner Server
13.1
Production
Elapsed time
measurement
script and
transaction logs
Eggplant
Functional
SAS®
Enterprise
Miner Server
13.2
Development
Parse logs
and
compare
response
times
Base SAS®
Production
release response
times used as
benchmark

Setup using one SUT’s browser to access two
different EM JWS versions or web app versions
Enterprise
Miner Server
14.3 (Java Web
Start)
Production
release
Eggplant
Functional
Future
release
Enterprise
Miner Server
14.1 (Java Web
Start)
Chrome
VNC Server
Windows 10
desktop
1
perf_Dave_71JWS
script records
EM 14.1
response times

Setup using one SUT’s browser to access two
different EM JWS versions or web app versions
Enterprise Miner
Server 14.3 (Java
Web Start)
VMware
Production
release
Eggplant
Functional
Future
release
Enterprise Miner
Server 14.1 (Java
Web Start) Chrome
VNC Server
Windows 10
desktop
2
perf_Dave_71JWS
script records
EM 14.3
response times
Parse logs
and
compare
response
times
Base SAS®
Random web app Random web app

How you can use Eggplant and SAS to
measure and analyze any GUI performance
regressions in your applications-under-test

Modifying your script for
measuring response times

Modifying your scripts for
measuring response times
• Follow along with the steps shown above in Code snippet: Creating a “Get list of projects”
timed transaction for EM.
• For each user action to be timed in your script, insert the code for starting a timer,
stopping the timer, and calculating the elapsed time as shown in the example.
• Insert a call to the write_transaction helper script to create the timed transaction entry in
the log file “times”.
• Helper scripts will be available in a package of sample code for conference attendees.

Generic setup using two SUTs
Device A
Application
version 1
Production
release
Your script for
GUI response
time
measurement
Eggplant
Functional
Device B
Application
version 2
Future
release
Eggplant
Identical
hardware
Identical GUIs

Sensetalk Code Samples:
Helper scripts

Optional: pause_or_prompt allows
manual control of action initiation
(* ============ TRANSACTION BEGIN: Get list of projects -============*)
// prompt for pause?
pause_or_prompt parmlist
// Open Project (Ctl-O)
typetext ControlKey, "o"
// Wait for Open Project dialog to display
write_transaction "Get list of projects“
(* ============ TRANSACTION END: Get list of projects -============*)

pause_or_prompt helper script

write_transaction.script
(* write_transaction
Author: John Crouch
Copyright 2011-2019 SAS Institute Inc. *)
params transname
global times, transnum, secondsElapsed, startTime, endTime
add 1 to transnum
write "TRANSACTION END: (" & transnum & ") " & transname & cr to file times
log "TRANSACTION END: (" & transnum & ") " & transname
write "Seconds elapsed: " & secondsElapsed & cr to file times
log "Seconds elapsed: " & secondsElapsed
write "(start time:" && startTime & " end time:" && endTime & ")" & cr to file times
log "(start time:" && startTime & " end time:" && endTime & ")"
write cr to file times
Timed transaction
log file name stored
in container times

Sample output from
write_transaction helper
scriptTimed GUI transaction log is C:UsersPublicEMEM132tranlogsrun_perf_Dave_132JWS_715929.log
TRANSACTION END: (1) Logon to EM
TRANSACTION END: (3) Open project Empty Project
(…)
“Get list of projects”
transaction in the log file

SAS Code Samples: Analyze timed
transactions with Base SAS® software

run_EM132_compare.
sas
%include 'C:UserssasjycDocumentsMy SAS Files9.4parseEM131_eggplantLog.sas';
%include 'C:UserssasjycDocumentsMy SAS Files9.4parseEM132_eggplantLog.sas';
%include 'C:UserssasjycDocumentsMy SAS Files9.4mean_EM131.sas';
%include 'C:UserssasjycDocumentsMy SAS Files9.4mean_EM132.sas';
%include 'C:UserssasjycDocumentsMy SAS Files9.4build_compareEM132.sas';

parseEM131_eggplantLo
g.sas
/* Read/parse EM transactions and timings from eggPlant-produced log files. */
/* The *_b designates aam600b.na.sas.com was the SUT for EM 13.1 on this iteration.*/
data sasuser.EM131_b (drop=rec);
length rec $80 transaction $62 ;
infile 'C:UsersPublicEMEM131tranlogs*.log' TERMSTR=CR truncover;
input rec $char80. ;
if (index(rec, 'TRANSACTION END:') > 0) then
do;
transaction = left( substr(rec, index(rec, ':') + 1) );
*read next record, has elapsed time for current transaction;
input rec $char80.;
ResponseTime = input( scan(rec, 2, ':'), 8.6 );
output;
end;
run;

parseEM132_eggplantL
og.sas
/* Read/parse EM transactions and timings from eggPlant-produced log files
The *_a designates aam600a.na.sas.com was the SUT for EM 13.2 on this iteration */
data sasuser.EM132_a (drop=rec);
length rec $80 transaction $62 ;
infile 'C:UsersPublicEMEM132tranlogs*.log' TERMSTR=CR truncover;
input rec $char80. ;
if (index(rec, 'TRANSACTION END:') > 0) then
do;
transaction = left( substr(rec, index(rec, ':') + 1) );
*read next record, has elapsed time for current transaction;
input rec $char80.;
ResponseTime = input( scan(rec, 2, ':'), 8.6 );
output;
end;
run;

mean_EM131.sas
/* The *_b designates aam600b.na.sas.com was the SUT for EM 13.1
on this iteration.
*/
%let inputds = sasuser.EM131_b;
%let sorted_means = sasuser.mean_EM131_b;
title1 "Mean GUI Response Times, EM 13.1";
title2 "SUT = aam600b.na.sas.com";
proc means data=&inputds /*noprint*/ ; class transaction; ways 1; var ResponseTime;
output out=temp_means131(keep=transaction ResponseTime131) mean=ResponseTime131;
proc sort out=&sorted_means; by transaction;
run;

mean_EM132.sas
/* Four required changes in memname and varname, be careful! */
/* The *_a designates aam600a.na.sas.com was the SUT for EM 13.2 on this iteration.
*/
%let inputds = sasuser.EM132_a;
%let sorted_means = sasuser.mean_EM132_a;
title1 "Mean GUI Response Times, EM 13.2";
title2 "SUT = aam600a.na.sas.com";
proc means data=&inputds /*noprint*/ ; class transaction; ways 1; var ResponseTime;
output out=temp_means132(keep=transaction ResponseTime132) mean=ResponseTime132;
proc sort out=&sorted_means; by transaction;
run;

build_compareEM_13
2.sas
/* The *_a designates aam600a.na.sas.com was the SUT for EM 13.2,
*_b designates aam600b.na.sas.com was the SUT for EM 13.1
*/
%let ds_old = sasuser.mean_em131_b;
%let ds_new = sasuser.mean_em132_a;
%let dsmerged = sasuser.compare_EM_132_a;
data &dsmerged;
merge &ds_old(in=a) &ds_new (in=b) ;
by transaction;
/* Calculate response time change as: (new release response time) - (older release response time)*/
Change = ResponseTime132 - ResponseTime131;
/* Calculate % change as: (response times difference) / (older release response time) */
Pct_Change = (Change / ResponseTime131) * 100;
run;
proc sort; by descending Pct_Change;
run;

Output table from
build_compareEM_132
.sas
The table is sorted with the largest
performance regressions, by
percentage, at the top.
The table can be viewed in Excel, where it can
be enhanced with color-coding, bar charts, etc.

Any platform.
Any browser.
Any software.

GUI Performance Regression Testing with Eggplant Functional

Recommended

More Related Content

Similar to GUI Performance Regression Testing with Eggplant Functional (20)

More from Eggplant (20)

Recently uploaded (20)

GUI Performance Regression Testing with Eggplant Functional