1 Reading Materials • Ron Patton Software Testing, second edition, Pearson Education, 2006 Software Testing and Quality Assurance • Roger S Pressman Software Engineering - A practitioner’s approach, sixth edition, McGraw-Hill Companies, 2005 • Shari Vance Pfleeger Software Engineering - Theory and Practice - second edition, Pearson Education, 2001 • Kshirasagar Naik and Priyadarshi Tripathy Software Testing and Quality Assurance, Theory and Practice A John Wiley & Sons, Inc., Publication, 2008 Dang Van Hung College of Technology • Sommerville, Software Engineering, 8th edition, Pearson Education Limitted, 2007 Chapters 22 and 23 Vietnam National University • Jeff Tian Software Quality Engineering Testing, Quality Assurance, and Quantifiable Improvement, Chapters 11, 12, the IEEE Computer Society, 2005 Part 1: Introduction - A global View Software Testing Background Topics Topics: • Software Testing Background • How software bugs impact our lives • The software Development Process • What bugs are and why they occur • The Realities of Software Testing • Who software tester are and what they What is a Bug? • Terms for software failures: Defect, Variance, Fault, Failure, Problem, Inconsistency, Error, Feature, Incident, Bug, Anomaly Infamous Software Error Case Studies • Generic terms: Problem, error, bugs (those cause software failure) • Disney’s Lion King, 1994-1995 • Formal definition of Software Bug: The software doesn’t something that the product specification says it should • Intel Pentium Floating-Point Division Bug, 1994 • NASA Mars Polar Lander, 1999 The software does something that the product specification says it shouldn’t • Patriot Missile Defense System, 1991 The software does something that the product specification doesn’t mention • The Y2K Bug, circa 1974 • Dangerous Viewing Ahead, 2004 The software doesn’t something that the product specification doesn’t mention, but it should The software is difficult to understand, hard to use, slow, or–in the software tester’s eyes–will be viewed by the end-user as just plain not right Why Bugs Occur? Bugs are caused by numerous reasons, but the main cause can be traced to the specification The Cost of Bugs 10 What make a Good Software Tester? • They are explorers What Does a Software Tester Do? • They are trouble shooter • The goal of a software tester is to find bugs • They are relentless (stronger and stronger) • The goal of a software tester is to find bugs and find them as soon as possible • They are creative • The goal of a software tester is to find bugs and find them as soon as possible and make sure they get fixed! • They are perfectionists • They exercise good judgment • They are tactful and diplomatic • They are persuasive 11 12 Example: A Square Root Program Example: SQRT Program with Specification Program SquareRoot Var r,s,x,y: integer; input y; r:=o; s:=-1; x:= 0; while r 0} r:=o; s:=-1; x:= 0; while r y} ; output x end 13 14 chan inpt = [0] of {short}; Automatic Testing (with Spin) active proctype sqrt() { short r = 0,s = -1, x = 0; short y; end: :: inpt? y -> r = 0; s = -1; x = 0; :: r s = s + 2; r = r + s; x = x + 1; :: else -> break od; x = x-1; assert((! (x*x > y)) && (y < (x+1)*(x+1))); printf ("if y = %d then x = %d\n",y, x) Test Case Generator: chan inpt = [0] of {short}; active proctype inputgen() { short i = 1; :: i < 1000 -> inpt!i; i=i+1 :: else -> break od } od } 15 16 Testing Results Full statespace search for: never-claim - (none specified) assertion violations + acceptance cycles - (not selected) invalid endstates + State-vector 36 byte, depth reached 4641, errors: 4387 states, stored states, matched 4387 transitions (= stored+matched) atomic steps hash conflicts: (resolved) (max size 2ˆ18 states) 1.596 memory usage (Mbyte) The Software Development Process Topics: • What major components go into a software product • How software progress from an idea to a final product 17 18 What Effort Goes into a Software Product The Software Development Lifecycle Models • Customer Requirements • Big-Bang • Specification • Code-and-Fix • Schedule • Software design documents (architecture, data flow diagram, state transition diagram, flowchart, commented code) • Waterfall • Spiral • Test documents (test plan, test cases, bug reports, test tools and automation) 19 Big-Bang Model 20 Code and Fix Model • The Approach: P eople + M oney + Energy = P erf ect Sof tware or N othing! • Advantages: Simple, needs less planning, less formal development process • Disadvantages: No formal testing activities, can’t monitor, high risk • The Approach Informal specification → loop of code and fix till enough-is-enough decision → Release • Advantages Little planning and documentation work • Disadvantages Hard to manage the testing 21 22 Spiral Model Waterfall Model • The Approach Round1 (Waterfall → Prototype) Evaluate Round1 Round2 (Waterfall → Prototype) Loop • The Approach: Discrete phases, a review after each phase will decide to move next or stay at this phase till it’s ready • Advantages: Easy to test (clear requirements) • Advantages: Easy to test, the customer is tightly involved • Disadvantages: High bugs fixing cost (bugs creep), customer is far from this till the end • Disadvantages: Need risk-assessments expertise 23 The Realities of Software Testing 24 Testing Axioms • It’s impossible to test a program completely Topics: • Why software can never be perfect – large input space – large output space – large state space • Why software testing isn’t just a technical problem • The terms commonly used by software testers – large number of possible execution paths – subjectivity of specifications 25 26 Testing Axioms Testing Axioms Large number of possible execution paths Number of paths = 2n f or (int i = 0; i < n; + + i){ if (a.get(i) == b.get(i)) x[i] = x[i] + 100; else x[i] = x[i]/2; } Large Input/State Space int exF unction(int x, int y){ } • Exhausted testing ⇒ try all possible combination of x and y! N Number of paths 10 1025 20 1048577 60 1.15 E+018 27 Testing Axioms Upper limit to total number of tests (Binder) 2n × (L1 × × LX ) × (V1 × ì VY ) ã n the number of decisions 28 Testing Axioms • Software testing is a risk based exercise – need to balance cost and risk of missing bugs • Testing can’t prove the absence of bugs • Li number of times a decision can loop • X the number of decisions that cause loops • The more bugs you find, the more bugs there are • Vi the number of possible values each input variable could have • The pesticide paradox • Y the number of input variables – a system tends to build resistance to a particular testing technique 29 30 Software Testing Terms and Definitions Testing Axioms • Precision and accuracy • Not all bugs found are fixed • Difficult to say when a bug is a bug – only when observed (latent otherwise) • Verification and validation (V&V) • Product specifications are never final • Quality and reliability • Software testers aren’t the most popular members of a project team • Testing and quality assurance • Software testing is a disciplined technical profession that requires training 31 32 What Is The V&V • Verification: Are we building the product right? - Confirming that something (software) meets its specification Testing and Quality Assurance • Validation: Are we building the right product? - Confirming that it meets the user’s requirements • The goal of a software tester is to find bugs and find them as soon as possible and make sure they get fixed! • The two major V&V activities are reviews, and testing So, testing is a part of V& V • A software quality assurance person’s main responsibility is to create and enforce standards and methods to improve the development process and to prevent bugs from ever occurring! • Validation testing – alpha testing: test conducted at the developer’s site by end-user – beta testing: test conducted at the end-user site 33 34 Examining The Specification Part 2: Testing Fundamentals Topics: Topics: • What is black-box and white-box testing • Examining the specification • How static and dynamic testing differ • Testing the Software with Blinders On • What high-level techniques can be used for reviewing a product specification • Examining the code • What specific problem you should look for when reviewing product specification in detail • Testing the Software with X-Ray Classes 35 Black-box and White-Box Testing 36 Black-box and White-Box Testing • Black-box (functional, behavioral, data-driven) – The software under test considered as a black-box and no knowledge of the internal structure or how the software actually works is utilized in testing – Testing based on inputs and respective outputs – The size of the black box can vary from one class or component to a whole system • White-box (structural, logic-driven) – Testing based on knowing the inner structure of the system and the program logic 37 38 Performing a High-level Review of The Specification Dynamic and Static Testing • Pretend to be the customer • Dynamic methods - testing - executes code • Research existing standards and guidelines – Software testing in the traditional sense: running and using software – Corporate terminology and conventions – Dynamic analysis methods – Industry requirements • Static methods not execute code – Government standards – Reviews, inspections, static analysis; testing the specification is black-box static testing – Graphical user interface – Security standards – Some authors name these static testing • Review and test similar software (things to look for include scale, complexity, testability, quality/reliability, and security) 39 40 Low-level Specification Test Techniques • Specification attributes check list – Complete – Accurate • Specification Terminology Check lists – Always, every, all, none, never (think of cases that violate them!) – certainly, Therefore, Clearly, Obviously, Evidently (Don’t fall to the trap) – Precise, – Some, sometimes, often, usually, ordinarily, customarily, most, mostly (too vague to be tested) – Unambiguous, – Etc, and so forth, so on, such as (aren’t testable) – Clear – Good, fast, cheap, efficient, small, stable (unquantifiable, need to explain) – Consistent – Relevant – Code-free – Handled, processed, rejected, skipped, eliminated (these hide some functionalities that need to be specified) – Testable – If then (no else) (incomplete) – Feasible Testing The Software with Blinders On Topics: • What is dynamic black box testing • How to reduce the number of test cases by equivalence partitioning • How to identify troublesome boundary condition • Good data values to use to induce bugs • How to test software states and state transitions • How to use repetition, stress, and high loads to locate bugs • A few secret places where bugs hide 109 Subdomain 110 Testing for Domain Errors Program P Domain D1 D3 D2 D4 D5 • Sources of Domains: how program predicates behave as an input classifier Comp for D1 • Types of Domain Errors: how minor modifications to program predicates, which can be interpreted as programming defects, can lead to domain errors Program output Concept input classifier D6 • Selecting Test Data to Reveal Domain Errors: A test selection criterion to pick input values to reveal the specific kinds of domain errors Comp for D6 111 112 Sources of Domains Example: A function Domains can be identified from both specifications and programs To identify domains from source code: int codedomain(int x, int y){ int c, d, k c = x + y; if (c > 5) d = c - x/2; else d = c + x/2; if (d >= c + 2) k = x + d/2; else k = y + d/4; return(k); } • Draw a control flow graph from the given source code • Find all possible interpretations of the predicates (a predicate in a program may have multiple interpretations, because control may arrive at a predicate node via different paths) • Analyze the interpreted predicates to identify domains 113 114 and The Domains for It and Its CFG P1: False P2: x>=4 Initialize: x, y P1: True P2: x=c+2 TF c=x+y TT k=y+d/4 k=x+d/2 P1 Domains obtained w.r.t P1: x+y>5 True P1: False P2P1 c>5 FF d=c+x/2 return(k) d=c−x/2 P2 FT 115 116 Types of Domain Errors • A program path will have a domain error if there is incorrect formulation of a path predicate After an interpretation of an incorrect path predicate, the path predicate expression causes a boundary segment to Test Selection Criterion – be shifted from its correct position or – have an incorrect relational operator • Data points on or near a boundary are most sensitive to domain errors They are: ON and OFF points to a boundary • A domain error can be caused by – an incorrectly specified predicate or – an incorrect assignment which affects a variable used in the predicate • Types of errors: Closure Error (A closure error occurs if a boundary is open when the intention is to have a closed boundary, or vice versa), Shifted-Boundary Error, and Tilted-Boundary Error • Test Selection Criterion: For each domain and for each boundary, select three points A, C , and B in an ON-OFF-ON sequence 117 118 Test Automation (1) Test Automation (2) A software testing workbench is an integrated set of tools to support the testing process Test data generator Test manager manages the running of program tests Specification Test generator generates test date for the program to be tested Source code Test manager Dynamic analyser Program being tested Execution report Test data Oracle File comparator compares the results of the program tests with the previous test results and reports the differences between them Test predictions Test results Simulator Oracle generates predictions of expected test results Dynamic analyzer adds code to a program to count the number of times each statement has been executed File comprator Simulator simulates the machine on which the program is to execute Report generator Test results report 119 120 An Overview of Configuration Testing Configuration Testing • Configuration Testing is the process of checking the operation of the software with various types of hardware • Different configuration possibilities for a standard Windows based PC: Topics: – The PC, Components • Why configuration testing is necessary – Peripherals (external devices), Their Interfaces • Why configuration testing can be a huge job – Options and Memories, Device Drivers • A basic approach to configuration testing • How to find the hardware you need to test with • What to if you’re not testing software for a desktop computer • Isolating Configuration Bugs – your software may have a bug that appears under broad class of configurations as well as being specific to one particular configuration – The hardware device or its drivers may have a bug that only your software reveals, or can be seen with lots of other software • Sizing Up the Job: Configuration Testing can be a huge undertaking 121 A PC Connected to Peripherals 122 Approaching The Task The general process used when planning your configuration testing: • Decide the types of hardware you’ll need • Decide what hardware brands, models, and device drivers are available • Decide which hardware features, modes, and options are possible • Pare down the identified configurations to a manageable set • Identify your software’s unique features that work with the hardware configurations • Design the test cases to run on each configuration • Execute the tests on each configuration • Rerun the tests until the results satisfy your team 123 124 Compatibility Testing Other Issues Topics • Obtaining the Hardware • What it means for software to be compatible • Identifying Hardware Standards • How standards define compatibility • Configuration Testing Other Hardware • What platforms are and what they mean for compatibility • Why being able to transfer data among software applications is the key to compatibility 125 126 Compatibility Testing Overview • Software compatibility testing means checking that your software interacts with and shares information correctly with other hardware • Compatible software example: cutting text from a web page and pasting it into a document in a word processor, upgrading to a new database program and having all your existing databases load in and work Questions for Compatibility Testing • What other platforms and other application software is your software designed to be compatible with? • What compatibility standard and guidelines for the interaction between the software and those supposed to be compatible? • What types of data will your software use to interact and share information with other? 127 Platform and Application Versions • Backward and Forward Compatibility: If something is backward compatible, it will work with the previous version of the software; If something is forward compatible, it will work with the future version of the software • The Impact of Testing Multiple Versions: The criteria to decide what programs to choose for testing the compatibility: Popularity, Age, Type, manufacture 128 Standards and Guidelines How to test the compatibility? • High-level Standards and Guidelines: Depending on platforms, • Low-level Standards and Guidelines: should be treated as an extension of the software’s specification 129 130 Data Sharing Compatibility Examples for data sharing Foreign Language Testing • File save and file load • File export and file import Topics • Cut, copy and paste • DDE (Dynamic data exchange), COM (Component object model, OLE (Object linking and embedding) • Why just translating is not enough • How words and text are affected • Why footballs and telephones are important • The configuration and compatibility issues • How large of a job testing another language is 131 132 Translation Issues Making The Words and Pictures Make Sense • Have you ever read a user’s manual for an appliance that was poorly converted word for word from other language? It is bad if little effort is put into building the software for foreign languages • Good translation is not enough - what needs to be counted for is the region or locale - the user’s country or geographic area • The process of adapting software to a specific locale, taking into account its language, dialect, local convention, and culture, is called localization or internationalization How to test something that’s in another language? • Text Expansion • ASCII, DBCS, and Unicode • Hot Keys and Shortcuts • Extended Characters • Computations on Characters • Reading Left to Right and Right to Left • Text in Graphics • Keep the Text out of the Code 133 134 Localization Issues • Content: like left drive - right drive, strange samples in American English encyclopedia (in the Figure below) Configuration and Compatibility Issues • Data Formats: different locales use different formats for data • Foreign Platform Configurations • Data Compatibility 135 How Much Should You Test? Depending on 136 Usability Testing Topics: • Was the project intended to be localized from every thing from the beginning? • What usability testing involves • Was programming code changed to make the localized version? • What to look for when testing a user interface • What special usability features are needed by the disabled 137 138 What Makes a Good UI? • Follows Standards and Guidelines User Interface Testing • Intuitive: clean, unobtrusive, not busy, organized, laid out well, there is no excessive functionality, does the help system really help? • User interface (UI) is the way to use to interact with a software program • UI may be very different, but technically they all provide the same interaction with the computer - the means to give input and receive output • Consistent: short key and menu selection, terminology and naming, audience, placement for buttons such as OK and cancel • Flexible: state jumping, state termination and skipping, data input/output • Comfortable: appropriate, error handling, performance • Correct • Useful 139 140 Testing for Software Security Testing for The Disabled: Accessibility Testing Topics: • Why someone would want to break into a computer • Legal Requirements • What types of break-ins are common • Accessibility Features in Software • How to work with your design team to identify security issues • Why software security problems are nothing more than software bugs • What you, as a software tester, can to find security vulnerabilities • How the new field of computer forensics is related to software security testing 141 142 Some Terms on Security Understanding The Motivation • A secure product: that protects the confidentially, and availability of customer’s information, and the integrity and availability of processing resources, under control of the system’s owner and administrator – a security vulnerability is a flow in a product that make it infeasible - even when using the product properly - to prevent an attacker from usurping privileges an the user’s system, regulating its operation, compromising data on it, or assuming ungranted trust – hacker: one who is proficient at using or programming a computer, a computer fluff One who uses programming skills to gain illegal access to a computer network or file The five motives that a hacker might have to gain access to a system are • Challenge/Prestige • Curiosity • Use/Leverage • Vandalize • Steal 143 Threat Modeling • Threat Modeling is not the software tester’s responsibility • Steps for modeling – Assemble the threat modeling team – Identify the assets – Create an architecture overview – Decompose the application – Identify the threats – Document the threats – Rank the threats (damage potential, reproducibility, exploitability, affected users, discoverability) 144 Comprehensive Threat Modeling for Defining Security Vulnerabilities 145 Is Software Security a Feature? Is Security Vulnerability a Bug? 146 Other Issues • Software security can be viewed as another feature of a software system • Understanding the Buffer Overrun • If the software failed and allowed a hacker to see the bank information of users, the software is considered to be buggy • Using Safe String Functions • Computer Forensics • Software tester may be responsible for testing the software’s overall security 147 Buffer Overrun 1: 2: 3; 4: 5: 6: void myBufferCopy(char * pSourceStr){ char pDestStr[100]; int nLocalVar1=123; int nLocalVar2=456; strcpy(pDestStr,pSourceStr) } void foo(){ 148 Website Testing Topics: • What fundamental parts of a web page need testing • What basic white-box and black-box techniques apply to web page testing • How configuration and compatibility testing apply • Why usability testing is the primary concern of web pages } What happen if pSourceStr is a string longer than 100? • How to use tools to help test your website 149 150 Black-Box Testing Web Page Fundamentals • Unlike data that is stored solely on a CD-ROM, web pages aren’t constrained to a single PC • Web page authoring isn’t limited to programmer using expensive and technical tools • Many possible web features, some features that make the website much more complex (customizable lay out, customizable content, e-commerce Treating the web page or the entire website as a black box is the easiest way for testing Some ideas of what to look for • Text: is a document, tested as to check audience level, terminology, contents, accuracy, etc • Hyperlinks: test to make sure it jumps to the correct destination • Graphics: all graphics load and display properly • Forms How to test? What are clues? • Objects and Other Simple Miscellaneous Functionality 151 152 White-Box Testing Gray-Box Testing • Gray-Box Testing is a mixture of black-box and white-box testing • Gray-Box Testing is suitable to test website, most webpages are built with HTML (HyperText Markup Language), and are easy for investigating as a supplement to testing several features be more effectively tested with a white-box approach: • Dynamic content • Database-driven web pages • Programmatically created web pages • Server performance and loading • Security 153 154 Usability Testing Configuration and Compatibility Testing • Usability Testing is a difficult process to define • Top ten mistakes in web design The possible hardware and software configurations: • Gratuitous us of Bleeding-Edge Technology • Hardware platform • Scrolling text, marquees, and constantly running animations • Browser software and version • Long Scrolling pages • Browser plug-in • Non-standard link color • Browser options • Out dated information • Video resolution and color depth • Overly long download times • Text size • Lack of navigation support • Orphan pages • Modem speeds • Complex website address (URLs) • Using frames 155 156 Part 3: Software Quality Assurance Topics: • What it costs to create quality software • How software testing varies from software quality assurance • What different ways a software testing or quality group can fit into a project team • How the software Capability Maturity Model is used • The ISO 9000 standard What Is ’Software Quality Assurance’? Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with It is oriented to ’prevention’ 157 158 What Are The Main Quality Attributes? (1) Availability: Is it available when and where I need to use it? What Is The Quality Software? Efficiency: How few system resources does it consume? Flexibility: How easy is it to add new features? Quality software is fairly bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable However, quality is obviously a subjective term Installability: How easy is it to correctly install the product Interoperability: How easily does it interconnect with other systems? Maintainability: How easy it is to correct defects or make changes? Portability: Can it be made to work on other platforms? 159 160 What Are The Main Quality Attributes? (2) Reliability: How long does it run before causing a failure? Reusability: How easily can we use components in other systems? Quality Standards and Auditing Organizations Testability: Can I verify that it was implemented correctly? • ISO (International Organization for Standardization) Usability: How easy is it for people to learn or to use? • CMMI (Capability Maturity Model Integration) Performance: the response time, utilization, and throughput behavior of the system • IEEE (Institute of Electrical and Electronics Engineers) Security: system’s ability to resist unauthorized attempts at usage or behavior modification, while still providing service to authorized users 161 162 Why Do Companies Want ISO 9001:2000? ISO 9001:2000 • ISO 9001 is a series of documents that define requirements for the Quality Management System Standard ISO 9001 is one of the documents in this set; it contains the actual requirements an organization must be in compliance with to become ISO 9001 Registered Many organizations decide to Implement ISO 9001 and obtain registration because it assures customers that the company has a good Quality Management System (QMS) in place An organization with an effective QMS will typically meet customer expectations better than an organization that does not have an effective QMS Many organizations require their suppliers to have ISO 9001 Registration 163 164 What Are The Levels of CMMI? What Is CMMI ? Capability Maturity Model Integration (CMMI) is a process improvement approach that provides organizations with the essential elements of effective processes It can be used to guide process improvement across a project, a division, or an entire organization CMMI provide guidance for quality processes, and provide a point of reference for appraising current processes 165 166 What Are The Levels of CMMI? (Cont’d) What Are The Levels of CMMI? (Cont’d) Level One - Initial Company has no standard process for software development Nor does it have a project-tracking system that enables developers to predict costs or finish dates with any accuracy Level Two - Managed Company has installed basic software management processes and controls But there is no consistency or coordination among different groups Level Three - Defined Company has pulled together a standard set of processes and controls for the entire organization so that developers can move between projects more easily and customers can begin to get consistency from different groups Level Four - Quantitatively Managed In addition to implementing standard processes, company has installed systems to measure the quality of those processes across all projects Level Five - Optimized Company has accomplished all of the above and can now begin to see patterns in performance over time, so it can tweak its processes in order to improve productivity and reduce defects in software development across the entire organization 167 168 Part Further Readings What Is IEEE? The “Institute of Electrical and Electronics Engineers” (IEEE) is a non-profit organization that develops, defines, and reviews electronics and computer science standards • Automated Testing and Tools • Testing Process Management – Planing the test effort – Writing and tracking test cases – Reporting what you find – Measuring your success ... 3: Software Quality Assurance Topics: • What it costs to create quality software • How software testing varies from software quality assurance • What different ways a software testing or quality. .. Dynamic methods - testing - executes code • Research existing standards and guidelines – Software testing in the traditional sense: running and using software – Corporate terminology and conventions... Intergration Testing Coding • The problem of NASA Mars Polar Lander, 1999 caused by poor integration testing 71 Unit and Integration Testing 72 Unit Testing • Unit testing (or component testing)