Solving Enterprise Applications Performance Puzzles www.it-ebooks.info IEEE Press 445 Hoes Lane Piscataway, NJ 08854 IEEE Press Editorial Board Lajos Hanzo, Editor in Chief R. Abhari M. El - Hawary O. P. Malik J. Anderson B - M. Haemmerli S. Nahavandi G. W. Arnold M. Lanzerotti T. Samad F. Canavero D. Jacobson G. Zobrist Kenneth Moore, Director of IEEE Book and Information Services (BIS) www.it-ebooks.info Solving Enterprise Applications Performance Puzzles Queuing Models to the Rescue Leonid Grinshpan IEEE PRESS A John Wiley & Sons, Inc., Publication www.it-ebooks.info Copyright © 2012 by the Institute of Electrical and Electronics Engineers. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the Web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifi cally disclaim any implied warranties of merchantability or fi tness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profi t or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our website at www.wiley.com. Library of Congress Cataloging-in-Publication Data: Grinshpan, L. A. (Leonid Abramovich) Solving enterprise applications performance puzzles : queuing models to the rescue / Leonid Grinshpan. – 1st ed. p. cm. ISBN 978-1-118-06157-2 (pbk.) 1. Queuing theory. I. Title. T57.9.G75 2011 658.4'034–dc23 2011020123 Printed in the United States of America. 10 9 8 7 6 5 4 3 2 1 www.it-ebooks.info Contents v Acknowledgments ix Preface xi 1. Queuing Networks as Applications Models 1 1.1. Enterprise Applications—What Do They Have in Common?, 1 1.2. Key Performance Indicator—Transaction Time, 6 1.3. What Is Application Tuning and Sizing?, 8 1.4. Queuing Models of Enterprise Application, 9 1.5. Transaction Response Time and Transaction Profi le, 19 1.6. Network of Highways as an Analogy of the Queuing Model, 22 Take Away from the Chapter, 24 2. Building and Solving Application Models 25 2.1. Building Models, 25 Hardware Specifi cation, 26 Model Topology, 28 A Model’s Input Data, 29 Model Calibration, 31 2.2. Essentials of Queuing Networks Theory, 34 2.3. Solving Models, 39 2.4. Interpretation of Modeling Results, 47 Hardware Utilization, 47 Server Queue Length, Transaction Time, System Throughput, 51 Take Away from the Chapter, 54 3. Workload Characterization and Transaction Profi ling 57 3.1. What Is Application Workload?, 57 3.2. Workload Characterization, 60 www.it-ebooks.info vi Contents Transaction Rate and User Think Time, 61 Think Time Model, 65 Take Away from the Think Time Model, 68 Workload Deviations, 68 “Garbage in, Garbage out” Models, 68 Realistic Workload, 69 Users’ Redistribution, 72 Changing Number of Users, 72 Transaction Rate Variation, 75 Take Away from “Garbage in, Garbage out” Models, 78 Number of Application Users, 78 User Concurrency Model, 80 Take Away from User Concurrency Model, 81 3.3. Business Process Analysis, 81 3.4. Mining Transactional Data from Production Applications, 88 Profi ling Transactions Using Operating System Monitors and Utilities, 88 Application Log Files, 90 Transaction Monitors, 91 Take Away from the Chapter, 93 4. Servers, CPUs, and Other Building Blocks of Application Scalability 94 4.1. Application Scalability, 94 4.2. Bottleneck Identifi cation, 95 CPU Bottleneck, 97 CPU Bottleneck Models, 97 CPU Bottleneck Identifi cation, 97 Additional CPUs, 100 Additional Servers, 100 Faster CPUs, 100 Take Away from the CPU Bottleneck Model, 104 I/O Bottleneck, 105 I/O Bottleneck Models, 106 I/O Bottleneck Identifi cation, 106 Additional Disks, 107 Faster Disks, 108 www.it-ebooks.info Contents vii Take Away from the I/O Bottleneck Model, 111 Take Away from the Chapter, 113 5. Operating System Overhead 114 5.1. Components of an Operating System, 114 5.2. Operating System Overhead, 118 System Time Models, 122 Impact of System Overhead on Transaction Time, 123 Impact of System Overhead on Hardware Utilization, 124 Take Away from the Chapter, 125 6. Software Bottlenecks 127 6.1. What Is a Software Bottleneck?, 127 6.2. Memory Bottleneck, 131 Memory Bottleneck Models, 133 Preset Upper Memory Limit, 133 Paging Effect, 138 Take Away from the Memory Bottleneck Model, 143 6.3. Thread Optimization, 144 Thread Optimization Models, 145 Thread Bottleneck Identifi cation, 145 Correlation Among Transaction Time, CPU Utilization, and the Number of Threads, 148 Optimal Number of Threads, 150 Take Away from Thread Optimization Model, 151 6.4. Other Causes of Software Bottlenecks, 152 Transaction Affi nity, 152 Connections to Database; User Sessions, 152 Limited Wait Time and Limited Wait Space, 154 Software Locks, 155 Take Away from the Chapter, 155 7. Performance and Capacity of Virtual Systems 157 7.1. What Is Virtualization?, 157 7.2. Hardware Virtualization, 160 Non-Virtualized Hosts, 161 Virtualized Hosts, 165 www.it-ebooks.info viii Contents Queuing Theory Explains It All, 167 Virtualized Hosts Sizing After Lesson Learned, 169 7.3. Methodology of Virtual Machines Sizing, 171 Take Away from the Chapter, 172 8. Model-Based Application Sizing: Say Good-Bye to Guessing 173 8.1. Why Model-Based Sizing?, 173 8.2. A Model’s Input Data, 177 Workload and Expected Transaction Time, 177 How to Obtain a Transaction Profi le, 179 Hardware Platform, 182 8.3. Mapping a System into a Model, 186 8.4. Model Deliverables and What-If Scenarios, 188 Take Away from the Chapter, 193 9. Modeling Different Application Confi gurations 194 9.1. Geographical Distribution of Users, 194 Remote Offi ce Models, 196 Users’ Locations, 196 Network Latency, 197 Take Away from Remote Offi ce Models, 198 9.2. Accounting for the Time on End-User Computers, 198 9.3. Remote Terminal Services, 200 9.4. Cross-Platform Modeling, 201 9.5. Load Balancing and Server Farms, 203 9.6. Transaction Parallel Processing Models, 205 Concurrent Transaction Processing by a Few Servers, 205 Concurrent Transaction Processing by the Same Server, 209 Take Away from Transaction Parallel Processing Models, 213 Take Away from the Chapter, 214 Glossary 215 References 220 Index 223 www.it-ebooks.info Acknowledgments ix My career as a computer professional started in the USSR in the 1960s when I was admitted to engineering college and decided to major in an obscure area offi cially called “ Mathematical and Computational Tools and Devices. ” Time proved that I made the right bet — computers became the major driver of civilization ’ s progress, and (for better or for worse) they have developed into a vital component of our social lives. As I witnessed permanent innovations in my beloved occupation, I was always intrigued by the question: What does it take for such a colossal complex combination of hardware and software to provide acceptable services to its users (which is the ultimate goal of any application, no matter what task it carries out), what is its architecture, software technol- ogy, user base, etc.? My research lead me to queuing theory; in a few years I completed a dissertation on queuing models of computer systems and received a Ph.D. from the Academy of Science of the USSR. Navigating the charted and uncharted waters of science and engi- neering, I wrote many articles on computer system modeling that were published in leading Soviet scientifi c journals and reprinted in the United States, as well as a book titled Mathematical Methods for Queuing Network Models of Computer Systems . I contributed to the scientifi c community by volunteering for many years as a reviewer for the computer science section of Mathematical Reviews , published by American Mathematical Society. My professional life took me through the major generations of architectures and technologies, and I was fortunate to have multiple incarnations along the way: hardware engineer, software developer, microprocessor system programmer, system architect, performance analyst, project manager, scientist, etc. Each “ embodiment ” contributed to my vision of a computer system as an amazingly complex universe living by its own laws that have to be discovered in order to ensure that the system delivers on expectations. When perestroika transformed the Soviet Union to Soviet Dis- union, I came to work in the United States. For the past 15 years as an Oracle consultant, I was hands - on engaged in performance tuning and sizing of enterprise applications for Oracle ’ s customers and prospects. www.it-ebooks.info x Acknowledgments I executed hundreds of projects for corporations such as Dell, Citibank, Verizon, Clorox, Bank of America, AT & T, Best Buy, Aetna, Hallibur- ton, etc. Many times I was requested to save failing performance proj- ects in the shortest time possible, and every time the reason for the failure was a lack of understanding of the fundamental relationships among enterprise application architecture, workload generated by users, and software design by engineers who executed system sizing and tuning. I began collecting enterprise application performance prob- lems, and over time I found that I had a suffi cient assortment to write a book that could assist my colleagues with problem troubleshooting. I want to express my gratitude to people as well as acknowledge the facts and the entities that directly or indirectly contributed to this book. My appreciation goes to: • Knowledgeable and honest Soviet engineers and scientists I was very fortunate to work with; they always remained Homo sapiens despite tremendous pressure from the system to make them Homo sovieticus . • The Soviet educational system with its emphasis on mathematics and physics. • The overwhelming scarcity of everything except communist demagogy in the Soviet Union; as the latter was of no use, the former was a great enabler of innovative approaches to problem solving (for example, if the computer is slow and has limited memory, the only way to meet requirements is to devise a very effi cient algorithm). • U.S. employers who opened for me the world of enterprise appli- cations fi lled with performance puzzles. • Performance engineers who drove tuning and sizing projects to failures— I learned how they did it, and I did what was necessary to prevent it; along the way I collected real - life cases. • Reviewers who reconsidered their own priorities and accepted publishers ’ proposals to examine raw manuscripts; the recipes they recommended made it edible. • My family for the obvious and the most important reason — because of their presence, I have those to love and those to take care of. L.G. www.it-ebooks.info [...]... Indicator—Transaction Time 7 that the first one returns a number saved in a database, while the second one is actually an ad hoc request requiring analytical calculations on the fly On the other hand, if the system appears to the users to be slow and getting even slower and slower as more users log in, then the performance analyst has to use the tools of his trade to fix the issues Application sizing and the. .. functions, and their performance is critical for successful execution of business tasks Consider as an example the failure of a company to deliver a timely quarterly earnings report to its shareholders and Wall Street due to a bottleneck in one of the system servers, which had crashed and brought the application down Solving Enterprise Applications Performance Puzzles: Queuing Models to the Rescue, First... foundation under the performance engineering of enterprise applications based on their queuing models I have successfully used the modeling approach to identify and solve performance issues; I hope a book on modeling methodology can be as helpful to the performance engineering community as it was of great assistance to me for many years SUBJECT Enterprise applications are the information backbones of today’s... available to the users 8 A significant factor influencing the workload of enterprise applications is the rate of the requests submitted by the users—a www.it-ebooks.info 1.1 Enterprise Applications What Do They Have in Common? 3 number of requests per given time interval, usually per one work hour Pacing defines an intensity of requests from the users and by the same token utilization of system resources Enterprise. .. and therefore carried out successfully and efficiently 1.4 QUEUING MODELS OF ENTERPRISE APPLICATION In dealing with such intricate objects as enterprise applications we have to see the forest, not the trees That is exactly what models help us to do: they shield our brains from numerous nonimportant and distracting details and allow us to concentrate on the fundamental processes in applications Models. .. mostly using their names according to Windows Performance Monitor terminology Similar counters exist in all UNIX operating systems, but their names might differ, and a reader should consult documentation to find the matching equivalent Leonid Grinshpan www.it-ebooks.info C H AP T E R 1 Queuing Networks as Applications Models In this chapter: characteristics of enterprise applications and their key performance. .. volume Enterprise applications permanently evolve as they have to stay in sync with the ever-changing businesses they support That creates a constant need for application tuning and sizing due to the changes in the number of users, volume of data, and complexity of business transactions Enterprise applications are very intricate objects Usually they are hosted on server farms and provide services to a... financial data on bonuses paid to employees in the first quarter of the current year • Load into the database data on all the items sold in January across company stores in New York State A user ’s request forces the application to perform a particular amount of work to generate a response This unit of work comprises the transaction; the time needed to complete the work is called transaction response time... key performance indicators; what is application sizing and tuning; why queuing models are representative abstractions of enterprise applications; what is transaction response time and transaction profile 1.1 ENTERPRISE APPLICATIONS WHAT DO THEY HAVE IN COMMON? Enterprise applications have a number of characteristics essential from a performance engineering perspective 1 Enterprise applications support... interest to product managers—they can use it to include in product documentation application sizing guides with estimates of the hardware needed to deploy applications Such guides also are greatly sought by sales professionals who work with prospects and customers Students majoring in computer science will find numerous examples of queuing models of enterprise applications as well as an introduction into . Abramovich) Solving enterprise applications performance puzzles : queuing models to the rescue / Leonid Grinshpan. – 1st ed. p. cm. ISBN 978-1-118-06157-2 (pbk.) 1. Queuing theory. I. Title. . Zobrist Kenneth Moore, Director of IEEE Book and Information Services (BIS) www.it-ebooks.info Solving Enterprise Applications Performance Puzzles Queuing Models to the Rescue Leonid Grinshpan IEEE. matching equivalent. Leonid Grinshpan www.it-ebooks.info Solving Enterprise Applications Performance Puzzles: Queuing Models to the Rescue, First Edition. Leonid Grinshpan. © 2012 Institute