Download Towards Regression Testing for Database Applications

Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Towards Regression Testing for
Database Applications
Gregory M. Kapfhammer†
Department of Computer Science
Allegheny College, Pennsylvania, USA
http://cs.allegheny.edu/~gkapfham/
ASTReNet and SOSoRNet, King’s College London, 2007
†
In Conjunction with Mary Lou Soffa (UVa/CS), Panos Chrysanthis (Pitt/CS), Bruce Childers (Pitt/CS)
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Testing Database Applications
Research Contribution
A regression testing framework for traditional database
applications. Future research includes service-oriented
applications that use Grid-enabled databases.
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
An Interesting Defect Report
Database Server Crashes
When you run a complex query against Microsoft SQL Server
2000, the SQL Server scheduler may stop responding.
Additionally, you receive an error message that resembles the
following: Date Time server Error: 17883 Severity: 1, State:
0 Date Time server Process 52:0 (94c) ...
An Input-Dependent Defect
This problem occurs when one or more of the following
conditions are true: The query contains a U N I O N clause or a
U N I O N A L L clause that affects many columns. The query
contains several J O I N statements. The query has a large
estimated cost. BUG 473858 (SQL Server 8.0)
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
An Interesting Defect Report
Database Server Crashes
When you run a complex query against Microsoft SQL Server
2000, the SQL Server scheduler may stop responding.
Additionally, you receive an error message that resembles the
following: Date Time server Error: 17883 Severity: 1, State:
0 Date Time server Process 52:0 (94c) ...
An Input-Dependent Defect
This problem occurs when one or more of the following
conditions are true: The query contains a U N I O N clause or a
U N I O N A L L clause that affects many columns. The query
contains several J O I N statements. The query has a large
estimated cost. BUG 473858 (SQL Server 8.0)
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Real World Example
A Severe Defect
The Risks Digest, Volume 22, Issue 64, 2003
Jeppesen reports airspace boundary problems
About 350 airspace boundaries contained in Jeppesen NavData
are incorrect, the FAA has warned. The error occurred at
Jeppesen after a software upgrade when information was pulled
from a database containing 20,000 airspace boundaries worldwide
for the March NavData update, which takes effect March 20.
An Important Point
Practically all use of databases occurs from within application
programs [Silberschatz et al., 2006, pg. 311]
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Real World Example
A Severe Defect
The Risks Digest, Volume 22, Issue 64, 2003
Jeppesen reports airspace boundary problems
About 350 airspace boundaries contained in Jeppesen NavData
are incorrect, the FAA has warned. The error occurred at
Jeppesen after a software upgrade when information was pulled
from a database containing 20,000 airspace boundaries worldwide
for the March NavData update, which takes effect March 20.
An Important Point
Practically all use of databases occurs from within application
programs [Silberschatz et al., 2006, pg. 311]
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Program and Database Interactions
Basic Operation
P
Program P creates SQL
statements in order to view
and/or modify the state of the
relational database
m
insert
select
update
D1
delete
De
Gregory M. Kapfhammer
SQL Construction
Static analysis does not reveal
the exact SQL command since
the program constructs the full
SQL statement at run-time
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Database Interaction Granularity
P
R1
A
B
E
F
I
J
mi
Dk
C
D
R2
G
H
R3
mj
K
L
Dl
Database Interactions
Program P interacts with two relational databases D k and Dl at
different levels of granularity (relation, record, attribute, ...)
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Overview of the Coverage Monitoring Process
Test Suite
Calculating Coverage
Adequacy
Criterion
Program
Use instrumentation probes
to capture and analyze a
program’s interaction with the
databases
Instrumented
Program
Instrumentation
Instrumented
Test Suite
Database
Test Coverage
Monitoring
Coverage
Results
Test
Requirements
Adequacy
Calculation
Adequacy
Measurements
Gregory M. Kapfhammer
Regression Testing
The adequacy
measurements can be used
to support both test suite
reduction and prioritization
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Database-Aware Coverage Trees
Test Case
Method Invocation
Database Interaction Point
Instrumentation Probes
Use static and dynamic
(load-time) instrumentation
techniques to insert coverage
monitoring probes
Database
Coverage Trees
Relation
Attribute
Record
Attribute Value
Store the coverage results in
a tree in order to support the
calculation of many types of
coverage (e.g., data flow or
call tree)
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Comparing the Coverage Trees
Tree Characteristics
Tree
CCT
DCT
DI-CCT
DI-DCT
DB?
×
×
X
X
Context
Partial
Full
Partial
Full
Probe Time
Low - Moderate
Low
Moderate
Moderate
Tree Space
Low
Moderate - High
Moderate
High
Table Legend
Database? ∈ {×, X}
Context ∈ {Partial, Full}
Probe Time Overhead ∈ {Low, Moderate, High}
Tree Space Overhead ∈ {Low, Moderate, High}
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Database-Aware Regression Testing
Original
Test Suite
GRT Repeat
Begin
Coverage Report
Reduction
or Prioritization
Modified
Test Suite
Test Suite
Execution
Testing Results
End
Program and
Database
VSRT Repeat
Regression Testing Overview
Reduction aims to find a smaller test suite that covers the same
requirements as the original suite. Prioritization re-orders the
tests so that they cover the requirements more effectively.
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
R3
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
R3
T11
R4
Finding the Overlap in Coverage
R4
R4
R4
T5
R4
R4
T10
R
R5
R2
R1
R3
R4
T1
T3
T11
R6
R7
R5
R6
T10
R7
R7
T8
T2
T4
T12
T7
T9
T10
T5
T6
Test Suite Reduction
Rj → Ti means that requirement Rj is covered by test Ti
T = hT2 , T3 , T6 , T9 i cover all of the test requirements
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Measuring Coverage Effectiveness
Cover R(T1 ) Cover
i=1
R(Ti )
Cover R(T )
Covered Test Reqs C(T ,t)
acements
Sn−1
Tn Done
...
Area
Testing Time
T1 Done
R t(n)
0
C(T , t)
(t)
Tn−1 Done
Prioritize to increase the CE of a test suite CE = Actual
Ideal
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Configuring the Regression Testing Framework
Configuration of the Regression Tester
Technique
Reduction
Greedy
Overlap-Aware
Prioritization
HGS
Test Cost
Type
Delayed Greedy
Unit
Reverse
Not Overlap-Aware
Cost
Coverage
Requirements
Actual
Type
Random
Traditional
Data Flow
Unique
Ratio
Super Path
Database-Aware
Coverage Tree Path
Dominant
Type of Tree
Containing Path
DCT
CCT
Regression tester uses several algorithms and test requirements
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Characterizing the Case Study Applications
Test Suites
Application
RM
FF
PI
ST
TM
GB
# Tests
13
16
15
25
27
51
Gregory M. Kapfhammer
Test NCSS / Total NCSS
227/548 = 50.5%
330/558 = 59.1%
203/579 = 35.1%
365/620 = 58.9%
355/748 = 47.5%
769/1455 = 52.8%
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Details About the Database Interactions
Static Interaction Counts
Application
RM
FF
PI
ST
TM
GB
executeUpdate
3
3
3
4
36
11
executeQuery
4
4
2
3
9
23
Total
7
7
5
7
45
34
Dynamic Interaction Counts
Database interactions that occur in iterative or recursive
computations are executed more frequently
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Reducing the Size of the Test Suite
(Size of Reduced Test Suite, Reduction Factor)
App
RM (13)
FF (16)
PI (15)
ST (25)
TM (27)
GB (51)
All (24.5)
Rel
(7, .462)
(7, .563)
(6, .600)
(5, .800)
(14, .481)
(33, .352)
(12, .510)
Attr
(7, .462)
(7, .563)
(6, .600)
(5, .760)
(14, .481)
(33, .352)
(12.17, .503)
Rec
(10, .300)
(11, .313)
(8, .700)
(11, .560)
(15, .449)
(33, .352)
(14.667, .401)
Attr Value
(9, .308)
(11, .313)
(7, .533)
(10, .600)
(14, .481)
(32, .373)
(13.83, .435)
Reduction factor for test suite size varies from .352 to .8
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Reducing the Testing Time
GRO
GRC
GRV
GRR
RVR
RAR
Rc
0.94
Time
1
0.94
0.81
Factor
0.78
0.8
0.6
Reduction
Av
0.81
0.79
0.78
0.79
0.39
0.36
0.4
0.2
0.06
0.06
Rc
Database
Interaction
HGB L
Av
GRO reduces test execution time even though it removes few tests
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Preserving Requirement Coverage
GRO
GRC
GRV
GRR
RVR
RAR
Rc
1.
Av
1.
0.98
0.99
0.98
0.96
1
0.94
0.8
0.72
0.71
Factor
0.6
Preservation
-
Coverage
0.91
0.4
0.3
0.2
0.09
Rc
Database
Interaction
HGB L
Av
GRO guarantees coverage preservation while the others do not
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Improving Coverage Effectiveness
GPO
GPC
GPV
GPR
RVP
ORP
RAP
Rc
1
0.94
Av
0.94
0.93
0.88
0.93
0.9
0.87
0.87
0.86
Coverage
Effectiveness
0.84
0.8
0.56
0.55
0.6
0.4
0.22
0.22
0.2
Rc
Database
Interaction
HGB L
Av
GRO is the best choice and the original ordering is poor
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Future Work: Avoid Database Restarts
T2
T16
T12
T3
T1
T8
T0
T13
T7
T5
T4
T10
T14
T11
T15
T6
T9
Use prioritization to reduce testing time by avoiding database restarts
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications
Introduction to Database Applications
Database-Aware Test Coverage Monitoring
Regression Testing
Experimental Study
Future Work and Conclusions
Conclusions and Future Work
Concluding Remarks
A new perspective on software testing and an efficient and
effective method for database-aware regression testing
Future Work
Challenges associated with grid-enabled databases
Conduct experiments with larger database applications
Resources
http://cs.allegheny.edu/~gkapfham/research/diatoms/
Gregory M. Kapfhammer
Towards Regression Testing for Database Applications