Q-patterns

Q-Patterns for Reusable Test Cases
October 8, 2007
ASTA Conference 2007

PureTesting offers end-to-end testing solutions. Our offerings includeTest Consulting, Testing Services, Test Trainings, and SDLC Products. PureTesting is recognized as a Testing Thought LeaderOur people work on Innovative concepts, write white papers, and present concepts at international forums.
PureTesting\'s unique value proposition of offering better, faster, less expensive testing solutions is compelling to our clients in India, Europe, New Zealand, and US who regard us as a long-term testing partner. The domains in which PureTesting has expertise are: Banking & Financial Services, Datacom and Telecom, Pharmaceuticals, Embedded Systems, eLearning, EAI, ISVs.
PureTesting was incorporated in 2005 and is headquartered in NOIDA (near Delhi), India with offices in Texas, USA; London, UK; and Auckland, New Zealand.
The author, Vipul Kochar is a co-founder of PureTesting. Hel has over 12 years of testing experience and is also the founder of www.whatistesting.com, a popular testing portal.
Prior to PureTesting Vipul worked at Adobe Systems India Pvt. Ltd. where he managed the testing of Adobe Reader (formerly known as Adobe Acrobat Reader) on Linux, Solaris, AIX and HP-UX and Handheld platforms. Vipul has won several awards including Logica CMG Triple Star Award for most original contribution at EuroStar 2005, Denmark. Vipul is a regular speaker at QAI International Conference, CONQUEST, World Conference on Quality etc. He is also the President of Indian Testing Board, member of ISTQB. Vipul holds a MS in Computer Engineering from IIT, Delhi.
Contact Information:
A-108B, Sector 58, NOIDA (UP) 201 301
India
Tel: +91 (120) 4324426, 4324427, 4311262
Fax: +91 (120) 4316115
Email: info@puretesting.com
sales@puretesting.com

What Are We Going to Talk About?

Reusable Test Cases Using Questioning Patterns Roadmap

Origin of Q-Patterns
Advantages and issues with test case reuse
Introduction to Q-Patterns
Applying Q-Patterns in organizations
Using Q-Patterns with exploratory testing
Caveats
Resources

Need for reuse in testing

Deja Vu

Have you ever looked at a new system to be tested and thought I have done this before?
What are the things that you had done before?

Learning and communicating it

How do you utilize your learning in new situation?
How do you communicate your learning?
How do you learn from other's learning?

If Wishes Were Horses

Have you ever wished for
More domain specific testing information
More platform specific testing information
More technology specific testing information
More use-cases for the given type of solutions
Pre-written test cases that you could use for testing or
for learning
Have you looked at the test cases for the famous triangle problem?
One answer to all the wishes-Reuse

Test Case Reuse: What is it?

Write once, use multiple times
For similar applications
For different applications with similar
features
For the same or different organization

Does not imply 100% reuse
Customization of tests may be necessary

Test Case Reuse: Issues
Specs reuse = test case reuse?
Changes of varying granularity required for reuse?
What format for writing test cases?
What sources for writing the tests?
Information on the net is too little, too scattered
IP issues?
Organizations not too keen to share tests.
Q-Patterns solve some of these issues
Origin of Q-Patterns

Frustration with reinvention of wheel
Write test cases for similar features
Need to educate, train and mentor testers
How do I impart my EXPERIENCE to others?
Need to learn testing for new domains, technologies and platforms
How is testing web-based applications different from testing embedded
systems? .NET/J2EE?
Need to find more bugs from test cases instead of ad-hoc testing
Q-Patterns: A Silver Bullet?

Q-Patterns provide:
Means to communicate experience
Aid in writing test cases
Provide a mechanism for test case reuse
Also aid in reviewing specifications and design
Q-Pattern: An Example

Consider this requirements fragment for login functionality of a system
Present a screen for login. The user id field is a combo
box and password field is a text field.
Successful login takes you to the next' screen
Unsuccessful login results in message to the user about
login failure
Some Questions

User ID/Password constraints/properties
What is the min/max length of user id/password?
Are they mandatory to be alphanumeric/numeric only?
Is user id/password case sensitive?
UI
Is the field wide enough to display user id?
Login failure
What is the number of allowed retries for login failure?
What happens after all retries are exhausted?
Where is cursor placed after login failure and what fields are blanked, if any?
More Questions

Does the password expire?
In how many days? (Say X)
Can the user set the value of X?
Can administrator set Password does not expire for a user?

Password change
Display in clear text or as some other character as * etc.? Confirm password feature?
When shown as *, how many characters are displayed?
When the password is changed after expiry can it be reset to the same password again? Or
last N number of password can not be used?
Can it be reset to the same password again?
... (And many more questions)

Was that a Q-Pattern?

We saw unstructured questions
We saw incomplete questions
Need to put more thoughts into a well defined structure to create a Q-
Pattern
Why is a structure required?
What structure can exist for Q-Patterns of
1. Different granularity
2. Different domains
3. Different concepts etc.
Introducing Q-Patterns

Examples:
Password based logon management
Combo box
Error messages
What are Q-Patterns
A set of interrelated QUESTIONS grouped together
relate to some aspect of user or software requirements
provide various alternatives to arrive at a solution.
You can also think of them as
Set of pre-cooked test cases
A tool for requirement elicitation, defining your specs etc.
Structure of a Q-Pattern

Name of the Q-Pattern
Intent/Explanation/definition
Classification
Metadata
Template Version
Q-Pattern Version
Author
Author Contact information
Keywords
Structure of a Q-Pattern

Questions on
Usage
User Interface
Administration
Performance
Security
Internationalization
Localization
Examples
Associated Q-Patterns
Specialization
Examples
Let us take a look at some Q-Patterns examples
Combo-boxes
Error Message
Password based login system

Combo Box Q-Pattern

Name: Combo Box
Intent: Behavior of combo boxes
Administration: NA
Usage:
Data Source
Is the combo-box populated with values by default?
What is the default value?
Where does the combo-box get its values from? Local file, hard coded, external
data source?
What if the data source fails? (local file does not exist/connection to external
source can't be established)
Combo Box Q-Pattern

Usage
Data addition/deletion/modification
How are new values added? Deleted?
Can values be modified? How?
Where are newly added values displayed? Beginning, end or
sorting based position?
Is it single or multi select? How is multiple selection done?
What font is used? Does it change with system settings?
Are text adornments such as bold, italics or underline allowed?
Are there any actions associated with tab movement, mouse
clicks?
Is copy-paste allowed?
Combo Box Q-Pattern

Usage
Sorting
Are values sorted? On what basis?
What is the default sort?
Can sort order be changed


Combo Box Q-Pattern

User Interface
Is box wide enough to display the widest element? If not, is scrolling possible?
How is selection displayed? (Bounding by dashed line or highlighting or checkboxes etc.)
If highlight is used, is the contrast of font and highlight color easy to read?
Does the box drop down to reveal all entries or some? How do you scroll vertically?
Can the box drop down and selections made using keyboard? Using Mouse?
What are the Background/Foreground colors? Can they be changed by the user?
How is text aligned?
Is there type-ahead feature?
Is the selection retained?
Internationalization
Does the box accept and display extended ASCII characters?
Does the box accept and display multi-byte characters?
Does the box accept and display right-to-left (Arabic, Hebrew) scripts? Top to bottom scripts?

Combo Box Q-Pattern
Security: NA
Performance
Time taken to populate the combo box
Time taken for the drop down action
Time taken for selecting items
Time taken for adding/deleting items
Examples: File Open dialog (File name)
Associated patterns: None
Specialization: Depending on toolkit used
Error Message Q-Pattern
Name: Error Handling
Intent: Handling and Display of errors.
Administration
Can a user/administrator decide not to display errors?
How can the errors be displayed again?
Can the errors be also logged in some files?
Can error messages be modified or customized?
Error Message Q-Pattern
Usage
Are errors categorized? For example, fatal, serious, minor etc.?
Is the error message text useful? Does it explain the problem?
Does the error message suggest possible solution or sources for further help?
Does the error message give you a facility to send the error report to its source?
Is user consent sought before any error information is sent anywhere?
Does the feature include collection of all relevant information from various logs and configuration files?
Error Message Q-Pattern

User Interface
Are error messages displayed as dialogs, part of a screen say,
status bar or logged into text logs?
Are error messages differentiated from warnings and information
messages?
How is this differentiation done?
When action is required on errors what kind of options are provided
to the user?
When error dialogs are dismissed, where does the control go?
Error Message Q-Pattern

Internationalization
Do error messages support extended ASCII and multi-
byte character set?
Localization
Are error messages localized?
Can the language of the error messages be changed?
Error Message Q-Pattern

Security: NA
Performance: NA
Accessibility: Are the errors accessible to challenged people?
Examples: Far too many .
Associated patterns: NA
Specialization: Depending on the application requirements

Q-Patterns: an Example

NAME
Password based Login Management

INTENT
Enumerating various strategies for password based authentication of a system or a user

CLASSIFICATION
Function oriented
Usage
User ID/Password constraints/properties
What is the min/max length of user id/password?
Are they mandatory to be alphanumeric/numeric only?
Is user id/password case sensitive?
Are Blank password allowed?
Can characters like #,$,or accented characters such as , be used?
Successful Login
Is the next screen presented to the user based on user role?

Is there a Save Password' feature?
Can this be turned off?
Can saved passwords be unsaved
What is the limit on number of saved used id-password combinations?
Where is this information saved? (Also see section on Security)

Usage

Login failure
What is the number of allowed retries for login failure?
What happens after all retries are exhausted?
Does the password expire?
In how many days? (Say X)
Can the user set the value of X?
Can administrator set Password does not expire for a user?
Password change
How is password change affected? (original password required?
Confirm password used?)
Display in clear text or as some other character as * ?
When shown as *, how many characters are displayed?
Does the password expire?
Can user request a password reset, if he has forgotten the password?
User Interface

Login screen
When the application is invoked, is the cursor placed at the right place say, user id field
Is the field wide enough to display user id?
Is password shown as stars while entering/changing/resetting the password
Is user id field a text field or a combo box? Can user be selected instead of being entered?
Is the tab order as desired?
Is enter the default button?
Are shortcuts and hotkeys present

User Interface

What is the strategy for conveying login failure?
Exact error cause (user does not exist, password is incorrect) shown or a generic message about any one of them being incorrect shown?
Where is cursor placed after login failure message is dismissed?
User id/password or both blanked out after incorrect login?
What message is conveyed if all retries are exhausted? Can a legitimate user get help from that message?
What is the strategy for conveying login failure?
Exact error cause (user does not exist, password is incorrect) shown or a generic message about any one of them being incorrect shown?
Where is cursor placed after login failure message is dismissed?
User id/password or both blanked out after incorrect login?
What message is conveyed if all retries are exhausted? Can a legitimate user get help from that message?
Is the feature accessible?

Administration

Does the system have a distinction of user and administrator?
What is the default password for administrator?
Are multiple administrators allowed?
Who creates administrative users?
Is there any super admin-admin hierarchy?
What is the difference of rights between users and administrators?

User Creation
Who can create users?
What is the default value for password of new users?
Can users be created in bulk by importing from lists, files etc.

Password resetting
Can an administrator read the password of a user?
Can administrator reset the password for other users?
Can administrator\'s password be reset?

Security

Is password shown as stars while entering or changing/resetting the password
How many stars are shown
When password is being entered?
When it is to be changed?
Where are the passwords stored?
Are they encrypted before storing? If yes what is the encryption algorithm used?
Encryption before transmission?
Number of failed login allowed?
Who can reset the password? How?

Security

Expiry time for the password?
Display of password in debug logs?
Expiry of session after some interval of inactivity?
Where does save password feature store the password? Is the password encrypted? What is the algorithm used?

Performance

Response time performance
What is the time taken for authentication? (It may depend upon)
Repository for password storage
Encryption algorithm
Connectivity
What is the time taken for displaying next screen?
What is the time taken for notification of login failure?
What is the response time for above scenarios when there are multiple simultaneous logins happening?
Storage performance
Is the storage independent of user id and password length?
Does the storage adjust itself when users are deleted?

Examples

OS login
Web based mail systems
Console based login (Telnet)

Associated Q-Patterns

Combo box
Error Handling

Specialization

Prune the questions to suite your needs
Add more questions to the specialized list OR enhance the parent Q-Pattern.
Example: Password based Login Management System for Web based software. (Specialize for say, use of GET/POST methods, Security in transmission etc.)

Practical Application: List Handling/Management

A very common construct composed of elements'
denotes a record set
Has Single or Multiple columns
Some examples
User list, File list, access rights list etc.
A form or a table displaying database records etc.
Combo/List box

Some examples of Lists


Exercise

Let us create a Q-Pattern for List Handling
Administration

Who creates the list?