1 Test Logs - Introduction Test Problem is a condition that exists within the software system that needs to be addressed. Carefully and completely documenting a test problem is the first step in correcting the problem. The following four attributes should be developed for all the test problems: Statement of condition. –Tells what it is. Criteria – Tells what should be. These two attributes are the basis for a finding. If a comparison between the two gives little or no practical consequence, no finding exists. Effect: Tells why the difference between what is and what should be is significant Cause: Tells the reasons for the deviation. Identification of the cause is the necessary as a basis for corrective action. A well developed problem statement will include each of these atttributes.When one or more these attributes is missing, questions almost arise, such as Criteria: Why is the current state inadequate? Effect: How significant is it? Cause: What could have cause of the problem? 1 Factors defining the Test Log Generation Document Deviation: Problem statements begin to emerge by process of comparision.Essentially the user compares” what is” with “what should be”. When a deviation is identified between what is found to actually exist and what the user thinks is correct or proper , the first essential step toward development of a problem statement has occurred. It is difficult to visualize any type of problem that is not in some way characterized by this deviation. The ‘What is”: can be called the statement of condition. The “What should be” shall be called the “Criteria”. These concepts are the first two and the most basic , attributes of a problem statement. The documenting of the deviation is describing the conditions, as they currently exist, and the criteria, which represents what the user desires. The actual deviation will be the difference or gap between “what –is” and “ what is desired”. The statement of condition is uncovering and documenting the facts, as they exist. What is a fact? The statement of condition will of course depend on the nature and extent of the evidence or support that is examined and noted. For those facts, making up the statement of condition, the I/S professional will need to ensure that the information is accurate, well supported, and worded as clearly and precisely as possible. The statement of condition should document as many of the following attributes as appropriate of the problem. Activities Involved:- The specific business or administered activities that are being performed during Test Log generation are as follows: Procedures used to perform work. – The specific step-by –step activities that are utilized in producing the output from the identical activities. Outputs /Deliverables – The products that are produced from the activity. Inputs - The triggers,events,or documents that cause this activity to be executed. Users/Customers served –The organization ,individuvals,or class users/customers serviced by this activity. Deficiencies noted – The status of the results of executing this activity and any appropriate interpretation of those facts. The Criterion is the user’s statement of what is desired. It can be stated in the either negative or positive terms. For example , it could indicate the need to reduce the complaints or delays as well as desired processing turn around time. Work Paper to describe the problem, and document the statement of condition and the statement of criteria. For example the following Work paper provides the information for Test Log Documentation: Field Requirements: Field Instructions for Entering Data Name of Software Tested : Put the name of the S/W or subsystem tested. Problem Description: Write a brief narrative description of the variance uncovered from expectations Statement of Conditions: Put the results of actual processing that occurred here. Statement of Criteria: Put what testers believe was the expected result from processing Effect of Deviation: If this can be estimated , testers should indicate what they believe the impact or effect of the problem will be on computer processing Cause of Problem: The testers should indicate what they believe is the cause of the problem, if known. If the testers re unable to do this , the work paper will be given to the development team and they should indicate the cause of the problem. Location of the Problem: The Tests should document where problem occurred as closely as possible. Recommended Action: The testers should indicate any recommended action they believe would be helpful to the project team. If not approved, the alternate action should be listed or the reason for not following the recommended action should be documented. Name of the S/W tested: Problem Description Statement of Condition Statement of Criteria Effect of Deviation Cause of a Problem Location of the Problem Recommended Action 2 Collecting Status Data Four categories of data will be collected during testing. These are explained in the following paragraphs. Test Results Data This data will include, Test factors -The factors incorporated in the plan, the validation of which becomes the Test Objective. Business objective –The validation that specific business objectives have been met. Interface Objectives-Validation that data/Objects can be correctly passed among Software components. Functions/Sub functions-Identifiable Software components normally associated with the requirements of the software. Units- The smallest identifiable software components Platform- The hardware and Software environment in which the software system will operate. Test Transactions, Test Suites, and Test Events These are the test products produced by the test team to perform testing. Test transactions/events: The type of tests that will be conducted during the execution of tests, which will be based on software requirements. Inspections – A verification of process deliverables against deliverable specifications. Reviews: Verification that the process deliverables / phases are meeting the user’s true needs. Defect This category includes a Description of the individual defects uncovered during the testing process. This description includes but not limited to : Data the defect uncovered Name of the Defect Location of the Defect Severity of the Defect Type of Defect How the defect was uncovered(Test Data/Test Script) The Test Logs should add to this information in the form of where the defect originated , when it was corrected, and when it was entered for retest. Storing Data Collected during Testing It is recommended that a database be established in which to store the results collected during testing. It is also suggested that the database be put in online through client/server systems so that with a vested interest in the status of the project can be readily accessed for the status update. As described the most common test Report is a simple Spread sheet , which indicates the project component for which the status is requested, the test that will be performed to determine the status of that component, and the results of testing at any point of time. Developing Test Status Reports Report Software Status Establish a Measurement Team Inventory Existing Project Measures Develop a Consistent Set of Project metrics Define Process Requirements Develop and Implement the Process Monitor the Process The Test process should produce a continuous series of reports that describe the status of testing. The test reports are for use of testers, test managers, and the software development team. The frequency of the test reports should be based on the discretion of the team and extensiveness of the test process. Use of Function/Test matrix: This shows which tests must be performed in order to validate the functions and also used to determine the status of testing. Many organizations use spreadsheet package to maintain test results. The intersection can be coded with a number or symbol to indicate the following: 1=Test is needed, but not performed 2=Test currently being performed 3=MINOR DEFECT NOTED 4=Major defect noted 5=Test complete and function is defect free for the criteria included in this test TEST FUNCTION 1 2 3 4 5 6 7 8 9 A B C D E Function Test Matrix 1 Methods of Test Reporting Reporting Tools - Use of word processing, database, defect tracking, and graphic tools to prepare test reports. Some Database test tools like Data Vision is a database reporting tool similar to Crystal Reports. Reports can be viewed and printed from the application or output as HTML, LaTeX2e, XML, DocBook, or tab- or comma-separated text files. From the LaTeX2e and DocBook output files you can in turn produce PDF, text, HTML, PostScript, and more. Some query tools available for Linux-based databases include: QMySQL dbMetrix PgAccess Cognos Powerhouse This is not yet available for Linux; Cognos is looking into what interest people have in the product to assess what their strategy should be with respect to the Linux ``market.'' GRG - GNU Report Generator The GRG program reads record and field information from a dBase3+ file, delimited ASCII text file or a SQL query to a RDBMS and produces a report listing. The program was loosely designed to produce TeX/LaTeX formatted output, but plain ASCII text, troff, PostScript, HTML or any other kind of ASCII based output format can be produced just as easily. Word –Processing: One way of increasing the utility of computers and word processors for the teaching of writing may be to use software that will guide the processes of generating, organizing, composing and revising text. This allows each person to use the normal functions of the computer keyboard that are common to all word processors, email editors, order entry systems, and data base management products. From the Report Manager, however, you can quickly scan through any number of these reports and see how each person's history compares. A one-page summary report may be printed with either the Report Manager program or from the individual keyboard or keypad software at any time. Individual Reports include all of the following information. Status Report Word Processing Tests or Keypad Tests Basic Skills Tests or Data Entry Tests Progress Graph Game Scores Test Report for each test Test Director:  Facilitates consistent and repetitive testing process  Central repository for all testing assets facilitates the adoption of a more consistent testing process, which can be repeated throughout the application life cycle  Provides Analysis and Decision Support  Graphs and reports help analyze application readiness at any point in the testing process  Requirements coverage, run schedules, test execution progress, defect statistics can be used for production planning  Provides Anytime, Anywhere access to Test Assets  Using Test Director’s web interface, tester, developers, business analysts and Client can participate and contribute to the testing process  Traceability throughout the testing process  Test Cases can be mapped to requirements providing adequate visibility over the test coverage of requirements  Test Director links requirements to test cases and test cases to defects  Manages Both Manual and Automated Testing  Test Director can manage both manual and automated tests (Win Runner)  Scheduling of automated tests can be effectively done using Test Director Test Report Standards - Defining the components that should be included in a test report. Statistical Analysis - Ability to draw statistically valid conclusions from quantitative test results. Testing Data used for metrics Testers are typically responsible for reporting their test status at regular intervals. The following measurements generated during testing are applicable: Total number of tests Number of Tests executed to date Number of tests executed successfully to date Data concerning software defects include Total number of defects corrected in each activity Total number of defects entered in each activity. Average duration between defect detection and defect correction Average effort to correct a defect Total number of defects remaining at delivery Software performance data us usually generated during system testing, once the software has been integrated and functional testing is complete. Average CPU utilization Average memory Utilization Measured I/O transaction rate Test Reporting A final test report should be prepared at the conclusion of each test activity. This includes the following  Individual Project Test Report  Integration Test Report  System Test Report  Acceptance test Report These test reports are designed to document the results of testing as defined in the testplan.The test report can be a combination of electronic data and hard copy. For example, if the function matrix is maintained electronically, there is no reason to print that, as paper report will summarize the data, draws appropriate conclusions and present recommendations.9 - Purpose of a Test Report: The test report has one immediate and three long term purposes. The immediate purpose is to provide information to customers of the software system so that they can determine whether the system is ready for production , and if so, to assess the potential consequences and initiate appropriate actions to minimize those consequences. The first of the three long term uses is for the project to trace problems in the event the application malfunctions in production. Knowing which functions have been correctly tested and which ones still contain defects can assist in taking corrective actions. The second long term purpose is to use the data to analyze the rework process for making changes to prevent the defects from occurring in the future. These defect prone components identify tasks/steps that if improved, could eliminate or minimize the occurrence of high frequency defects. The Third long term purpose is to show what was accomplished in case of an Y2K lawsuit.  Individual Project Test Report These reports focus on the Individual projects(software system),when different testers should test individual projects, they should prepare a report on their results.  Integration Test Report Integration testing tests the interfaces between individual projects. A good test plan will identify the interfaces and institute test conditions that will validate interfaces. Given is the Individual Project test report except that conditions tested are interfaces. 1.Scope of Test – This section indicates which functions were and were not tested 2.Test Results – This section indicates the results of testing, including any variance between what is and what should be 3.What works/What does not work - This section defines the functions that work and do not work and the interfaces that work and do not work 4. Recommendations – This section recommends actions that should be taken to Fix functions /Interfaces that do not work. Make additional improvements  System Test Reports A System Test plan standard that identified the objective of testing , what was to be tested, how was it to be tested, and when tests should occur. The system test Report should present the results of executing the test plan. If these details are maintained Electronically , then it need only be referenced , not included in the report.  Acceptance Test Report There are two primary objectives of Acceptance testing Report .The first is to ensure that the system as implemented meets the real operating needs of the user/customer. If the defined requirements are those true needs, testing should have accomplished this objective. The second objective is to ensure that software system can operate in the real world user environment, which includes people skills and attitudes, time pressures, changing business conditions, and so forth. The Acceptance Test Report should encompass these criteria’s for the User acceptance respectively. 2 Conclusion The Test Logs obtained from the execution of the test results and finally the test reports should be designed to accomplish the following objectives:  Provide Information to the customer whether the system should be placed into production, if so the potential consequences and appropriate actions to minimize these consequences.  One Long term objective is for the Project and the other is for the information technology function.  The project can use the test report to trace problems in the event the application malfunction in production. Knowing which functions have been correctly tested and which ones still contain defects can assist in taking corrective actions.  The data can also be used to analyze the developmental process to make changes to prevent defects from occurring in the future.  These defect prone components identify tasks/steps that if improved, could eliminate or minimize the occurrence of high frequency defects in future. . Report Word Processing Tests or Keypad Tests Basic Skills Tests or Data Entry Tests Progress Graph Game Scores Test Report for each test Test Director:  Facilitates. Integration Test Report  System Test Report  Acceptance test Report These test reports are designed to document the results of testing as defined in the testplan.The