TesTing for ConTinuous Delivery wiTh visual sTuDio 2012 TesTing for ConTinuous Delivery wiTh visual sTuDio 2012 For more information explore: msdn.microsoft.com/practices Software Architecture and Software Development patterns & practices Proven practices for predictable results Save time and reduce risk on your software development projects by incorporating patterns & practices, Microsoft’s applied engineering guidance that includes both production quality source code and documentation. The guidance is designed to help software development teams: Make critical design and technology selection decisions by highlighting the appropriate solution architectures, technologies, and Microsoft products for common scenarios Understand the most important concepts needed for success by explaining the relevant patterns and prescribing the important practices Get started with a proven code base by providing thoroughly tested software and source that embodies Microsoft’s recommendations The patterns & practices team consists of experienced architects, developers, writers, and testers. We work openly with the developer community and industry experts, on every project, to ensure that some of the best minds in the industry have contributed to and reviewed the guidance as it is being developed. We also love our role as the bridge between the real world needs of our customers and the wide range of products and technologies that Microsoft provides. As more software projects adopt a continuous delivery cycle, testing threatens to be the bottleneck in the process. Agile development frequently revisits each part of the source code, but every change requires a re-test of the product. While the skills of the manual tester are vital, purely manual testing can’t keep up. Visual Studio 2012 provides many features that remove roadblocks in the testing and debugging process and also help speed up and automate re- testing. This guide shows you how to: • Recordandplaybackmanualteststoreproducebugsandverifythexes. • Transformmanualtestsintocodetospeedupre-testing. • Monitoryourprojectintermsoftestspassed. • Createanduseeffectiveunittests,load,andperformancetests. • Runbuild-deploy-testworkowsonvirtuallabenvironments. • Evolveyourtestingprocesstosatisfythedemandsofagileandcontinuous delivery. You’ll learn how to set up all the tools you need for testing in Visual Studio 2012 and 2010, including Team Foundation Server, the build system, test controllers andagents,SCVMMandHyper-V.Eachchapterisstructuredsothatyoucan move gradually from entry-level to advanced usage. T e s T i n g f o r C o n T i n u o u s D e l i v e r y w i T h v i s u a l s T u D i o ® 2 0 1 2 Larry Brader Howie Hilliker Alan Cameron Wills Sprint Working software Stakeholder feedback Requirements Develop Operate Bugs and feedback Product backlog Monitor Ops backlog Testing for Continuous Delivery with Visual Studio 2012 Larry Brader Howie Hilliker Alan Cameron Wills 978-1-62114-019-1 This document is provided "as-is." Information and views expressed in this document, including URL and other Internet website references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. © 2012 Microsoft. All rights reserved. Microsoft, Hyper-V, IntelliSense, IntelliTrace, Internet Explorer, Microsoft Press, MSDN, SharePoint, Visual Basic, Visual C#, Visual Studio, Windows, Windows Server, and Windows Vista are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners. Contents Foreword xiii Preface xv The team who brought you this guide xvii Where to go for more information xviii 1 The Old Way and the New Way 1 Application lifecycle management with Visual Studio 2 Contoso and Fabrikam; or something old, something new 3 From Contoso to Fabrikam 8 Application lifecycle management tools 9 Moving to the new way: adopting Visual Studio for ALM 10 Chapter 2: Unit Testing: Testing the Inside 13 Chapter 3: Lab Environments 13 Chapter 4: Manual System Tests 14 Chapter 5: Automated System Tests 14 Chapter 6: A Testing Toolbox 15 Chapter 7: Testing in the Software Lifecycle 15 Appendix: Setting up the Infrastructure 15 The development process 16 Testers vs. developers? 16 Agile development 17 Summary 19 Where to go for more information 20 2 Unit Testing: Testing the Inside 21 In this chapter 21 Prerequisites 23 Unit tests in Visual Studio 23 Running unit tests 24 Debugging unit tests 25 v vi Test-first development 25 Limitations of test-first development? 28 Baseline tests 28 Tests verify facts about the application, not exact results 28 How to use unit tests 29 Testing within a development task 30 Code coverage 30 Check-in policies 32 How to write good unit tests 32 Arrange – Act – Assert 33 Test one thing with each test 33 Use test classes to verify different behavioral areas of the code under test 33 Test exception handling 34 Don’t only test one input value or state 34 Separate test data generation from verification 35 Pex generates test data 36 Isolation testing: fake implementations 36 Microsoft Fakes 38 Mocks 40 Shims 40 Shims in Visual Studio 2010 42 Testing and debugging 42 IntelliTrace 43 Coded UI tests 44 How to create and use coded UI tests 44 Create a coded UI test 44 Running coded UI tests 45 Edit and add assertions 45 Extend the basic procedure to use multiple values 45 Isolate 46 Test first? 47 Coded UI tests: are they unit or system tests? 47 Designing for coded UI tests 49 Maintaining coded UI tests 49 Continuous integration with build verification tests 49 How to set up a build in Team Foundation Server 51 Code coverage 53 Third-party build frameworks 53 Generate an installer from the build 53 Test the installer 54 vii Monitoring the build 55 How to set up build failure or completion emails 55 On the Team Foundation Server machine 55 On your desktop machine 56 To set up build alerts in Visual Studio 2010 57 Responding to build failure alarms 58 The builds check-in policy 58 The build report 59 Spreading the load of many unit tests 59 Summary 59 Differences between Visual Studio 2010 and Visual Studio 2012 60 Where to go for more information 61 3 Lab Environments 63 What is a lab environment? 65 Stored SCVMM environments 67 Lab management with third-party virtualization frameworks 68 How to use lab environments 68 Prerequisites 68 Lab center 69 Connecting to a lab environment 70 Using a deployed (running) environment 70 Deploying an environment 71 Creating a new virtual environment 73 Stored and running machines 75 Creating and importing virtual machines 75 Composed environments 77 Standard environments 78 Summary 79 Differences between Visual Studio 2010 and Visual Studio 2012 79 Where to go for more information 80 4 Manual System Tests 81 Microsoft Test Manager supports manual system tests 82 Exploratory testing 82 Creating bugs 86 Creating test cases 86 No more “no repro” 87 viii Testing with test cases 89 Creating test cases before coding the requirements 90 Test cases have steps 91 Creating a test case after the code is written 92 Running a test case 92 Replaying actions on later runs 96 Benefits of test cases in Microsoft Test Manager 97 How test cases are organized 98 Test plans 98 Test suites 99 Shared steps 99 Parameters 100 Configurations 101 Build 102 Testing in a lab environment 103 Client outside the lab: testing web servers 103 Create the lab environment 103 Tell the test plan about the environment 104 Get and install the latest build 104 Start testing 105 On finding a bug, save the virtual environment 105 Client in the lab: testing desktop and thick-client apps 105 Test impact analysis 106 Enabling test impact analysis 107 Using test impact analysis 107 Devising exploratory tests and scripted test cases 108 Mostly positive tactics 108 Storyboards 108 Create, read, update, and delete (CRUD) 109 States 111 Using models in testing 111 Mostly negative tactics 112 Exploratory and scripted testing in the lifecycle 113 Smoke tests 113 Monitoring test progress 114 Tracking test plan progress 114 Tracking test suite progress in Microsoft Test Manager 114 Tracking test plan results in Microsoft Test Manager 115 Leveraging test reports to track testing progress 116 We’re not done till the tests all pass 117 Benefits of system testing with Visual Studio 117 The old way 117 The new way 117 Summary 120 Differences between Visual Studio 2010 and Visual Studio 2012 120 Where to go for more information 120 ix 5 Automating System Tests 121 Who creates automated tests, and when? 124 How to automate a system test 124 Code the test method 125 Prerequisites 125 Generate the coded UI test 125 Coding integration tests by hand 133 Link the test method to the test case 134 Create an environment for automated tests 134 Network isolated environments 135 Set a test plan to perform automated tests 135 Automated deployment to a lab environment 136 Automating deployment with Windows Installer 137 Identifying a server build definition 141 Creating a lab build-deploy-test definition 142 Automating deployment of ClickOnce applications 144 Viewing the test results and logging bugs 146 Driving tests with multiple clients 146 Summary 147 Differences between Visual Studio 2010 and Visual Studio 2012 147 Where to go for more information 147 Where do my team members find their bugs? 149 6 A Testing Toolbox 149 What other Microsoft tools can I use to find bugs? 150 Performance and stress tests in Visual Studio 150 Adding diagnostic data adapters to test settings 150 Load testing 156 Web performance tests in load tests 157 Unit tests in load tests 157 Coded UI test in load tests 157 Creating load tests 157 Running and analyzing load tests 159 IntelliTrace 161 Configuring the IntelliTrace diagnostic data adapter 161 Fixing non-reproducible bugs with IntelliTrace 163 Feedback tool 164 Create a feedback request 164 Provide feedback 165 Remote debugging 167 Summary 168 Differences between Visual Studio 2010 and Visual Studio 2012 168 Where to go for more information 169 x 7 Testing in the Software Lifecycle 171 Process wars 172 Who creates system tests? 176 DevOps 176 Testing the business process 178 Operational monitoring 178 Shortening the devOps cycle 178 Updating existing code 180 Testing in the development cycle 180 Tests are executable requirements 180 Inception 181 Each sprint 182 Using plans and suites 183 Reports 185 Test plan progress report 185 User story test status 186 Test case readiness 186 Process improvements 187 What do we get by testing this way? 189 What to do next 191 Where to go for more information 192 The testing infrastructure 193 What software do I need? 193 Appendix: Setting up the Infrastructure 193 Configuration choices: spreading the software on the hardware 194 The components of the infrastructure 195 Example configurations 198 The non-starter setup 198 The economy setup 199 Separate server computers for high traffic 200 Separate file server for the VM library 201 More computers 202 Server components on virtual machines 203 Licensing 204 Service accounts 205 Installing the software 207 Installation overview 208 Hyper-V 208 Virtual machines for test infrastructure components 209 Install System Center Virtual Machine Manager (SCVMM) 212 Enable Microsoft .NET 3.5 and Internet Information Services (IIS) 214 Install SQL Server 2012 or 2008 R2 for Team Foundation Server 214 [...]... critical issue for 24x7 operations It is no longer acceptable to leave system testing to the end of the project; testing needs to happen continuously We’ll show you how to use the methodology in this book to make this work • We describe how to set up and configure the machinery for testing If your job is to administer Visual Studio, lifecycle management, and its testing tools, the Appendix is for you pr... applications, organizations cannot afford to implement long painful testing processes every time a feature needs tweaking or a line of code must change That’s why we wrote this book Fortunately, the continuous testing required today need not be as cumbersome as it was in the past If you perform application lifecycle management using the Microsoft tools based on Visual Studio, the testing piece of your development... Different processes are appropriate for different products and different teams Continuous delivery might be appropriate for a social networking website, but not less so for medical support systems Whether your process is rapid-cycle or very formal, you can lower the risks and costs of software development by adopting some of the principles of agile development, including rigorous testing and incremental development... controller Setting up physical and virtual machines for testing Creating a virtual machine by using SCVMM Where to go for more information 215 215 217 217 217 220 220 220 221 221 222 223 Index 225 Foreword This book tells the story of two companies, Contoso and Fabrikam Over my thirty years in the software industry, I’ve seen lots of companies that work like Contoso Fortunately, over the last decade, I’ve also... lifecycle management process In these pages, we’ll show you how to perform testing with Visual Studio Team Foundation Server 2012 Though we will focus on that version, you can also use Visual Studio 2010 (We note the differences where necessary.) With the setup guidance we provide here, your team should be able to test complex distributed systems within a few days The Old Way a nd the New Way Of course, you... prompting you with the steps you have to perform It lets you record the results and make notes, and will record the actions you take to help diagnose any bugs that you find You can log a bug with one click, complete with a screenshot, a snapshot of the machine states, and a log of the actions you took leading up to the failure.  Chapter 5: Automated System Tests System testing starts with exploration—just... automate testing, helps you track and record the process and results of testing, and helps you easily repeat that testing whenever you need to xv xvi The key to this improved testing process is integration Integrated tools mean that your test plan is a living and executable document that links directly to your tests It means that you can take snapshots of your testing environment configuration for storage... the development team Unfortunately, we can’t make software testing sound any more glamorous outside the industry Bear this in mind when introducing yourself at social occasions And if you have any good tips for that scenario, we’d love to hear from you The team who brought you this guide Larry Brader conceived, motivated, and led the creation of this book “I have been testing for a long time.” Chris... updates Before we dig into lifecycle management with Visual Studio, let’s take a look at the two fictitious companies that will introduce us to two rather different approaches to testing and deployment: Contoso, the traditional organization, and Fabrikam, the more modern company They will help define the ALM problem with greater clarity Contoso and Fabrikam; or something old, something new At Contoso, testing. .. known for the quality of its software products for several decades Recently however, testing has begun to seem like a dead weight that is holding the company back When customers report bugs to Contoso, product managers will typically remark “Well, the fix is easy to code, but we’d have to re-test the whole product We can’t do that for just one bug.” So they have to politely thank the customer for her . TesTing for ConTinuous Delivery wiTh visual sTuDio 2012 TesTing for ConTinuous Delivery wiTh visual sTuDio 2012 For more information explore: msdn.microsoft.com/practices Software. feedback Product backlog Monitor Ops backlog Testing for Continuous Delivery with Visual Studio 2012 Larry Brader Howie Hilliker Alan Cameron Wills 978-1-62114-019-1 This document is provided "as-is." Information. Runbuild-deploy-testworkowsonvirtuallabenvironments. • Evolveyour testing processtosatisfythedemandsofagileand continuous delivery. You’ll learn how to set up all the tools you need for testing in Visual Studio 2012 and 2010, including