What are Manual Testing Methodologies

It is a project level document. This document defines required testing approach for corresponding project testing. project manager like people are developing test methodology depending on company level test strategy. Due to this reason, test methodology is also known as a refinement form of test strategy.

What are Manual Testing Methodologies 

To develop test methodology, project manager/quality analyst follows below approach. Before start every project testing.
Step 1:- collect test strategy.
Step 2:- identify current project type.

Project Type Information Gathering &
Analysis Design Coding System Testing Maintenance
Traditional
Outsourcing
Maintenance

Note: - depending on project type, project manager delete some of the columns from TRM (test responsibility matrix) for this project testing.

Step 3:- study project requirements.
Note: – depending on requirements in the project, PM delete unwanted factors (rows) from TRM for this project testing.

Step 4: – determine the scope of project requirements.
Note: – depending on expected future enhancements, PM is adding some of previously deleted factors to TRM for this project testing.

Step 5: – identify tactical risks.
Note: – depending on analyzed risks, PM is deleting some of the factors from selected TRM for this project testing.
Step 6:- finalized TRM for current project testing.
Step 7:- prepare system test plan.
Step 8:- prepare module test plan.

Manual TESTING PROCESS:

PET PROCESS
This process developed by HCL Chennai it is also a refinement form of v-model. This process defines mapping between development process and testing process. From this process model, organizations are maintaining separate testing team for functional and system testing. The remaining stages of testing are done by developers.

PET stands for process experts tools and technology.

Test Plan

After completion of test initiation and testing process finalization, test lead category people are concentrating on test plan document preparation with “what to test?”, “how to test?” , “when to test?” , “who to test?” . in this test plan document preparation test lead is following below approach.
 Testing team formation: in general test planning process starts with testing team formation. In this stage test lead is depending on below factors.

Availability of test engineers
Test duration the 3 factors are dependent
Availability of test environment resources

Case study:
Test Duration: C/S, Web, ERP è 3 to 5 months of system testing
System s/w è 7 to 9 months of system testing
Machine critical è 12 to 15 months of system testing

Team Size: Developers: Testers = 3: 1

Identify Tactical Risks: After formation of testing team, test lead is analyzing selected team level risks. This risk analysis is also known as Root Cause Analysis.

Ex: Risk 1: lack of knowledge of testing team on that domain.
Risk 2: lack of budget (time).
Risk 3: lack of resources (testing tools not available)
Risk 4: lack of test data (improper documents)
Risk 5: delays in delivery
Risk 6: lack of development process rigor
Risk 7: lack of communication (in b/w testing team to development team)

Prepare test plan: after completion of testing team formation and risks analysis, test lead concentrate on test plan document preparation in IEEE format (Institute of Electrical and Electronics Engineers).

Format:
Test plan ID: unique number / name.
Introduction: about project
Test items: names of all modules in that project ex: website.
Features to be tested: new module names for test design. (What to test)
Features not to be tested: which ones and why not? (Copy test cases from server)

Approach: selected list of testing techniques by project manager to be applied on above modules,(finalized TRM)
Testing tasks: necessary operations to do before start every module testing.
Suspension criteria: possible raised problems during above modules testing.
Ex: exception handling.

Feature pass/fail criteria: when a module is pass and when a module is fail.
Test environment: required hardware’s and software’s to conduct testing on above modulus. Ex: WinRunner
Test deliverables: names of testing documents to be prepared during above modulus

EX: test cases, test procedures, test scripts, test log, defect reports for every modules.
Staff and training needs: names of selected test engineers for this project testing
Responsibilities: mapping between names of test engineers and names of modules.
(Work allocation)
Schedule: dates and times.
Risks and mitigations: raised problems during testing and solutions to overcome.
Approvers: signatures of project manager and test lead.

Review test plan

After completion of first copy of test plan document development, test lead conducts a review on that document for completeness and correctness. In this review meeting test lead concentrate on coverage analysis.

Coverage analysis:
Business requirement based coverage (what to test?)
TRM based coverage (how to test?)
 Risks based coverage (when & who to test?)

After finalizations of test plan, test lead is providing some training sessions to selected testing team on project requirements.

Test Design

After finalization of test plan and after completion of training sessions, test engineers are concentrating on test cases development for responsible modules.

There are three methods to prepare test cases such as:
Business logic based test case design (depending on srs)
Input domain based test case design (design documents)
User interface based test case design.

Business logic based test case design
In general, test engineers are preparing maximum test cases depending on use cases in SRS. Every use case is describing functionality in terms of inputs, process and outputs.

From the above model test engineers are preparing test cases depending on that use case. Every use case is also known as functional specification. Every test case describes a testable condition to be applied on build.

To study use cases, test engineers are following below approach.

Step1: collect required use cases for responsible modules.
Step2: selecting a use case and their dependencies from above collected list of

Use case.
Step2.1: identify entry condition (base state)
Step2.2: identify input required (test data)
Step2.3: identify output and outcome (expected)
Step2.4: study normal flow (navigation)
Step2.5: study end condition (end state)
Step2.6: study alternative flow and exceptions
Step3: prepare test cases depending on above study of use cases
Step4: review the test cases for completeness and correctness
Step5: go to step2 until all use cases study completion

Test case format

During test design test engineers are preparing test cases in IEEE format. Through these formats test engineers are documenting every test case.

Format:
Test case id: unique number/name
Test case name: the name of test conditions.
Feature to be tested: corresponding module or function name.
Test suit id: the corresponding batch id, in that batch this case is also member.
Priority: the importance of test case in terms of functionality.
EX: P0—basic functionality (requirements)
P1—general functionality (recovery, Compatability, inter systems, load—)
P2—cosmetic functionality (user inter face)

Test environment: required hard wares and soft wares including testing tool to execute this test case.

Test efforts: (person/hour) time to execute this test case .
EX: 20 min average time

Test duration: Date and time

Test setup: necessary tasks to do before start this case execution

Test procedure: this step-by-step procedure from base state to end state

Test case pass/fail criteria: when this case is pass/ when this case is fail
NOTE: in general, test engineers are not maintaining complete format for every test case. They can try to maintain test procedure as monetary for every test case.

Input domain based test case design

In general, test engineers are preparing test cases depending on use cases or functional specifications in SRS. Sometimes they can go to depending on design documents also. Because, use cases are not providing complete information about size and type of input objects. Due to this reason, test engineers are studying data models in design documents.


EX: ER-diagrams (entity relationship diagrams)
In this study, test engineers are following below approach

Step1: collect data models of responsible modules from design documents
Ex: ER-diagrams
Step2: study every input attribute in terms of size and type with constraints
Step3: prepare BVA and ECP for every input attribute in below format

I/P Attribute ECP BVA (Size/Range)
Valid Invalid Min Max


This table is called DATA MATRIX. This table is providing information about every object
Step4: identify critical and non-critical inputs in above list
Ex:
Critical inputs are involving in internal manipulations. Non-critical inputs used for printing purpose.
NOTE: If our test case is covering an operation, then test engineers are preparing step-by-step procedure from base state to end state. If our test case is covering an object, then test engineers are preparing data matrix.

User interface based test case design

To conduct usability testing, test engineers are preparing test cases depending on global user inter face convection, our organization rules and interest of customer site people.

Example test cases:

Test case1: spelling check
Test case2: graphics check (alignment, font, style, color and other micro soft six rules)
Test case3: meaning full error messages
Test case4: accuracy of data display
Test case5: accuracy of data in the database as a result of user input
Test case6: accuracy of data in the database as a result of external factors
Ex: file attachment, export files, import files etc
Test case7: meaning full help messages
NOTE: test case1 to test case6 are indicating user inter face testing and test case7 is indicating manuals support testing.

Test design review

Before receiving build from development team to start test execution, test lead is analyzing the completeness and correctness of prepared test cases by test engineers through a review meeting. In this review test lead is depending on coverage analysis.

Business requirement based coverage
Use cases based coverage
Data model based coverage
User inter face based coverage
Test responsibility matrix based coverage
At the end of this review, test lead is preparing requirements trace ability matrix (RTM). This matrix defines mapping between customer requirements and prepare test cases. This matrix is also known as requirements validation matrix (RVM).

Test Execution

After completion of test design and their reviews, testing team is receiving initial build from development team to start test execution.

Levels of test execution:
Level –0 testing on initial build
Level –1 testing on stable build
Level –2 testing on modified stable build
Level –3 testing on master build (ready to release)

Levels of Test Execution Vs Test Cases:
Level –0 Initial build è All Po test cases (Basic functionality).
Level –1 Stable build è All Po, P1, and p2 Test cases as test batches.
Level –2 Modified build è Selected Po, P1, and P2 test cases w.r.t modifications.
Level –3 Master build è Selected Po, P1, and P2 test cases w.r.t bug density.

Build version control

In general, testing team is receiving build from development team. With the help of existing network protocols. From the above model, testing team is receiving build from development team through FTP. To access soft base in network server. Soft base in server consists of old builds and modified builds; development people are assigning unique version number to that builds. This version numbering system is understood able to testing team. For this build version controlling, development people are using version control tools.
Ex: vss (visual source safe)
Test harness: test harness means that readiness to start test execution.

Level-0 (sanity testing): after receiving initial build from development team, testing team is concentrating on sanity testing to estimate stability of that build to be applied complete testing. In this preliminary testing, testing team concentrates on basic functionality of that build. In this functionality coverage, testing team concentrate on below factors.

Understand ability
Opera ability
Absorbability
Controllability Testability
Consistency
Simplicity
Maintainability
Automat ability

From the above 8 factors, sanity testing is also known as testability testing or octangle testing, tester acceptance testing and build verification testing.

Test automation if possible: After receiving stable build from development team, test engineers are creating automated test scripts with required checkpoints; If possible all test cases are not automat able. Due to this reason test engineers are making automation test scripts for repeatable and complex test cases.

Case study: In general, testing teams are following selective automation only. In this selective automation test engineers are creating test scripts using testing tools for functionality or requirements test cases and load/stress test cases.

Test Execution Type Testing Techniques Testing Tools Comments
Manual UI testing manuals support testing — Easy to conduct
Manual / automation Functionality testing WR, QTP, Robot, Silk Test Basic functionality testing is repeatable
Manual Recovery, Compatability, Configuration, Inter systems, Installation, Sanitation and Parallel Testing — No tools in market
Manual / with Automation Load and Stress Testing Load Runner, SQA load Test, Silk Performer, J meter Expensive manually and complex to conduct
Manual Storage and data volume testing security testing — No tools in market for this type of testing.

Level-1 (comprehensive testing): After receiving stable build and after completion of all possible automation, testing team arrange test cases as batches. Every test batch consists of the set of dependent test cases. These test batches are also known as test suit or test set. During these test batches execution, test engineers are preparing test log documents. This test log document consists of three types of entries.
–Passed, all expected values are equal to actual values
–Failed, anyone expected value vary with actual
 –Blocked, postponed due to in correct parent functionality

Level-2 (regression testing): During level-1/comprehensive testing, test engineers are reporting mismatches to development team. After receiving modified build from development team, test engineers are concentrating on regression testing, test engineers are following below approach with respect to seriousness of that mismatches.

Case1: If the development team resolved bug severity is high, then test engineers are re-executing all p0, all p1 and carefully selected p2 test cases on that modified build with respect to modifications.

Case2: If the development team resolved bug severity is medium, then test engineers are re-executing all p0, carefully selected p1 and some of p2 test cases with respect to modifications.

Case3: If the development team resolved bug severity is low then test engineers are re-executing some of p0, p1 and p2 test cases with respect to modifications.

Case4: If the development team released modified build due to sudden changes in customer requirements then test engineers are re-executing all p0, all p1 and carefully selected p2 test cases with respect to that change in the requirements.

Error, Defect, Bug: A mistake in coding is called error.
Coding errors found by testing team during testing called defect or issues.
Testing team reported issues accepted by development team to be solved, called bug.

Test Reporting Or Defect Tracking
During level-1 and level-2 test execution, test engineers are reporting mismatches to development team in IEEE format.

Format:

Defect id: unique number and name.

Description: summary of defect

Build version id: version number of build, in which test engineers found this defect

Feature: the corresponding module name, in which test engineer found this defect

Test case name: the name of test condition, during this case execution, test engineer found this defect

Reproducible: yes, means defect appears every time in test execution No, means defect appears rarely in test execution

If yes, attach test procedure

If no, attach snap shot and strong reasons

Found by: the name of test engineer

Detected on: date of submission

Assigned to: the responsible person at development side to receive this defect
Status: New – Re-reporting defect or Reopen – Reporting defect first time

Severity: The seriousness of defect in terms of functionality
High – Not able to continue test execution without resolving that defect
Medium – Able to continue remaining testing but compulsory to solve
Low – Able to continue remaining testing but optional to resolve (may/may not)

Priority: the importance of this defect in terms of customer

Suggested fix (optional): expected possibilities to resolve this defect by developers

Fixed by: project manager/project lead

Resolved by: programmer name

Resolved on: date of resolving

Resolution type:

Approved by: signature of project manager
NOTE: in above format development people try to change priority of defect with respect to customer importance

Defect age: The time gap between defects reported date and defect resolved date is called defect age

Defect submission: Large-scale organizations,Small & medium scale organizations

Defect status:
Bug life cycle:

Defect resolution type: During test execution, test engineers are reporting mismatches to development team as defects. After receiving defect reporting from testing team, development people are conducting bug-fixing review and they will send resolution type report to corresponding testing team. There are 12 types of resolutions to report to testing team.
Duplicate: Rejected due to this defect equal to previously reported defect.
Enhancement: rejected due to this defect related to future requirement of the customer
Hard ware limitation: rejected due to this defect related to limitations of hard ware devices
Software limitation: rejected due to this defect related to limitations of soft ware technologies
Not applicable: rejected due to improper meaning of defect
Functions as designed: rejected due to coding is correct with respect to design logic
Need more information: not accepted and not rejected but developer’s required extra information to understand the defect
Not reproducible: not accepted and not rejected but developer’s required correct producer to reproduce that defect
No plan to fix it: not accepted and not rejected but developer’s required extra time to fix
Fixed: accepted and ready to resolve
Fixed indirectly (deferred): accepted but postponed to future version
User misunderstanding: extra negotiation between developers and tester

TYPES OF BUGS: During test execution either in manual or in automation, test engineers are finding below types of bugs.
Users inter face bugs: (low severity)
Ex1: spelling mistake (high priority)
Ex2: improper right alignment (low priority)

Error handling bugs: (medium severity)
Ex1: does not return error message (high priority)
Ex2: complex meaning in error message (low priority)

Input domain bugs: (medium severity)
Ex1: allows in valid inputs (high priority)
Ex2: allows in valid type also (low priority)

Calculations bugs: (high severity)
Ex1: dependent out puts are wrong (application show stopper) (high priority)
Ex2: find output is wrong (module show stopper) (low priority)

Race condition bugs: (high severity)
Ex1: deadlock or hang (application show stopper) (high priority)
Ex2: improper order of functionalities (low priority)

Load condition bugs: (high severity)
Ex1: does not allows multiple users (high priority)
Ex2: does not allows customer expected load (low priority)

Hard ware bugs: (high severity)
Ex1: not able to establish connection to hard ware device (high priority)
Ex2: wrong output from device (low priority)
Version control bugs: (medium severity)
Ex1: mismatches in between two consecutive build versions

ID-control bugs: (medium severity)
Ex: wrong logo, logo missing, copy right window missing, wrong version number, soft ware title mistake, team members names missing——etc.

Source bugs: (medium severity)
Ex: mistakes in help documents

TEST CLOSER: After completion of all possible test execution and bugs resolving, test lead concentrate on test closer to stop testing process. In this review test lead is depending on below factors.

Coverage analysis:

Business requirements based coverage
Use cases based coverage
Data model based coverage
In put domain based coverage
User inter face based coverage
Test responsibilities matrix based coverage

Bug density:
Ex: A – 20%
B – 20%
C – 40% ç Need for Regression
D – 20%
 100%
1. Analysis of deferred bug: whether the deferred bugs are postponable or not?
At the end of this review meeting, test lead can go to select high bug density module in our application build for final regression (level-3)

Above approach is also known as level-3 testing (or) final regression testing (or) pre-acceptance testing (or) post mortem testing
After completion of this final regression, testing team concentrate on user acceptance with the help of real customers (or) model customers.

USER ACCEPTANCE TESTING: 

After completion of test closer test management is concentrating on user acceptance testing to collect feedback from customer site people. There are two approaches to conduct user acceptance testing such as

SIGN OFF: After completion of user acceptance testing and their modifications, test lead is preparing final test summary report. It is a part in “soft ware release note”(SRN). This final test summary report consists of below documents as members.
• Test methodology
• Test plan
• Requirements trace ability matrix
• Automated test scripts
• Final bugs summary report

Bug description Found by Feature Severity Status (closed/deferred) Comments

Final test summary report (FSTR)
Case study: (five months of testing process)

Deliverable Responsibility Completion time
Test Cases Preparation
Test Cases Review
Requirements Traceability Matrix
Test Automation
Test Execution (level-1 & Level-2)
Defect Reporting
Communication and Status
Reporting
Test Closer and Final Regression (Level-3)
User Acceptance Test
Sign Off Test Engineers
Test Lead & Test Engineers
Test Lead
Test Engineers
Test Engineers
Test Engineer, Test Lead
Test Lead
Test Lead & Test Engineer
Customer site people including Testing Team
Test Lead 15-20 days
4-5 days
1-2 days
10-15 days
40-60 days
On going
Weekly twice
4-5 days
4-5 days
2-3 days

Auditing: During testing and maintains of soft ware’s project and test management people are using three types of measurements and metric such as
(1) Quality assessment measurement (QAM)
(2) Test management measurement (TMM)
(3) Process capability measurement (PCM)

Quality assessment measurement: (QAM) during soft ware testing process, quality analyst or project manager is using these measurements to estimate quality assurance level in that testing process. (Monthly once)
è Stability:
Duration No of Bugs
20%
80% 80%
20%

Sufficiency:
• Requirements coverage (modules)
• Type-trigger analysis (what type of test completed)
Defect severity distribution: organization trend limit check.

Test management measurements: During a project testing process, test lead category people are using these measurements to estimate testing process coverage. (Weekly twice)
Test status:
• Completed cases execution
• In execution
• Yet to execute
Delays in delivery:
• Bug arrival rate
• Bug resolution rate
• Bug ageing

Test efficiency: • Cost to find a bug (Ex: 5 bugs/person-day)


Process capability measurements: During soft ware maintenance in customer place, QA and PM are using these measurements to improve testing team capability. (Yearly once)

Defect escapes: (missed bugs by testing team)

a. Type- phase testing
b. Type- triggers analysis
c. Defect resolution rate (or) defect removal efficiency i.e. DRE =A/ (A+B)
1. Bugs found by testing team
2. Bugs found by customer during maintenance