www.it-ebooks.info The Grid www.it-ebooks.info www.it-ebooks.info The Grid Core Technologies Maozhen Li Brunel University, UK Mark Baker University of Portsmouth, UK www.it-ebooks.info Copyright © 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777 Email (for orders and customer service enquiries): cs-books@wiley.co.uk Visit our Home Page on www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to permreq@wiley.co.uk, or faxed to +44 1243 770620. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging in Publication Data Li, Maozhen. Core technologies /Maozhen Li, Mark Baker. p. cm. ISBN-13 978-0-470-09417-4 (PB) ISBN-10 0-470-09417-6 (PB) 1. Computational grids (Computer systems) 2. Electronic data processing—Distributed processing. I. Baker, Mark. II. Title. QA76.9.C58L5 2005 005.3  6—dc22 2005002378 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN-13 978-0-470-09417-4 (PB) ISBN-10 0-470-09417-6 (PB) Typeset in 11/13pt Palatino by Integra Software Services Pvt. Ltd, Pondicherry, India Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production. www.it-ebooks.info Contents About the Authors xiii Preface xv Acknowledgements xix List of Abbreviations xxi 1 An Introduction to the Grid 1 1.1 Introduction 1 1.2 Characterization of the Grid 1 1.3 Grid-Related Standards Bodies 4 1.4 The Architecture of the Grid 5 1.5 References 6 Part One System Infrastructure 9 2 OGSA and WSRF 11 Learning Objectives 11 Chapter Outline 11 2.1 Introduction 12 2.2 Traditional Paradigms for Distributed Computing 13 2.2.1 Socket programming 14 2.2.2 RPC 15 2.2.3 Java RMI 16 2.2.4 DCOM 18 2.2.5 CORBA 19 2.2.6 A summary on Java RMI, DCOM and CORBA 20 2.3 Web Services 21 2.3.1 SOAP 23 2.3.2 WSDL 24 2.3.3 UDDI 26 2.3.4 WS-Inspection 27 2.3.5 WS-Inspection and UDDI 28 2.3.6 Web services implementations 29 2.3.7 How Web services benefit the Grid 33 www.it-ebooks.info vi CONTENTS 2.4 OGSA 34 2.4.1 Service instance semantics 35 2.4.2 Service data semantics 37 2.4.3 OGSA portTypes 38 2.4.4 A further discussion on OGSA 40 2.5 The Globus Toolkit 3 (GT3) 40 2.5.1 Host environment 41 2.5.2 Web services engine 42 2.5.3 Grid services container 42 2.5.4 GT3 core services 43 2.5.5 GT3 base services 44 2.5.6 The GT3 programming model 50 2.6 OGSA-DAI 53 2.6.1 OGSA-DAI portTypes 54 2.6.2 OGSA-DAI functionality 56 2.6.3 Services interaction in the OGSA-DAI 58 2.6.4 OGSA-DAI and DAIS 59 2.7 WSRF 60 2.7.1 An introduction to WSRF 60 2.7.2 WSRF and OGSI/GT3 66 2.7.3 WSRF and OGSA 69 2.7.4 A summary of WSRF 70 2.8 Chapter Summary 70 2.9 Further Reading and Testing 72 2.10 Key Points 72 2.11 References 73 3 The Semantic Grid and Autonomic Computing 77 Learning Outcomes 77 Chapter Outline 77 3.1 Introduction 78 3.2 Metadata and Ontology in the Semantic Web 79 3.2.1 RDF 81 3.2.2 Ontology languages 83 3.2.3 Ontology editors 87 3.2.4 A summary of Web ontology languages 88 3.3 Semantic Web Services 88 3.3.1 DAML-S 89 3.3.2 OWL-S 90 3.4 A Layered Structure of the Semantic Grid 91 3.5 Semantic Grid Activities 92 3.5.1 Ontology-based Grid resource matching 93 3.5.2 Semantic workflow registration and discovery in myGrid 94 3.5.3 Semantic workflow enactment in Geodise 95 3.5.4 Semantic service annotation and adaptation in ICENI 98 3.5.5 PortalLab – A Semantic Grid portal toolkit 99 3.5.6 Data provenance on the Grid 106 3.5.7 A summary on the Semantic Grid 107 www.it-ebooks.info CONTENTS vii 3.6 Autonomic Computing 108 3.6.1 What is autonomic computing? 108 3.6.2 Features of autonomic computing systems 109 3.6.3 Autonomic computing projects 110 3.6.4 A vision of autonomic Grid services 113 3.7 Chapter Summary 114 3.8 Further Reading and Testing 115 3.9 Key Points 116 3.10 References 116 Part Two Basic Services 121 4 Grid Security 123 4.1 Introduction 123 4.2 A Brief Security Primer 124 4.3 Cryptography 127 4.3.1 Introduction 127 4.3.2 Symmetric cryptosystems 128 4.3.3 Asymmetric cryptosystems 129 4.3.4 Digital signatures 130 4.3.5 Public-key certificate 130 4.3.6 Certification Authority (CA) 132 4.3.7 Firewalls 133 4.4 Grid Security 134 4.4.1 The Grid Security Infrastructure (GSI) 134 4.4.2 Authorization modes in GSI 136 4.5 Putting it all Together 140 4.5.1 Getting an e-Science certificate 140 4.5.2 Managing credentials in Globus 146 4.5.3 Generate a client proxy 148 4.5.4 Firewall traversal 148 4.6 Possible Vulnerabilities 149 4.6.1 Authentication 149 4.6.2 Proxies 149 4.6.3 Authorization 150 4.7 Summary 151 4.8 Acknowledgements 151 4.9 Further Reading 151 4.10 References 152 5 Grid Monitoring 153 5.1 Introduction 153 5.2 Grid Monitoring Architecture (GMA) 154 5.2.1 Consumer 155 5.2.2 The Directory Service 156 5.2.3 Producers 157 5.2.4 Monitoring data 159 www.it-ebooks.info viii CONTENTS 5.3 Review Criteria 161 5.3.1 Scalable wide-area monitoring 161 5.3.2 Resource monitoring 161 5.3.3 Cross-API monitoring 161 5.3.4 Homogeneous data presentation 162 5.3.5 Information searching 162 5.3.6 Run-time extensibility 162 5.3.7 Filtering/fusing of data 163 5.3.8 Open and standard protocols 163 5.3.9 Security 163 5.3.10 Software availability and dependencies 163 5.3.11 Projects that are active and supported; plus licensing 163 5.4 An Overview of Grid Monitoring Systems 164 5.4.1 Autopilot 164 5.4.2 Control and Observation in Distributed Environments (CODE) 168 5.4.3 GridICE 172 5.4.4 Grid Portals Information Repository (GPIR) 176 5.4.5 GridRM 180 5.4.6 Hawkeye 185 5.4.7 Java Agents for Monitoring and Management (JAMM) 189 5.4.8 MapCenter 192 5.4.9 Monitoring and Discovery Service (MDS3) 196 5.4.10 Mercury 201 5.4.11 Network Weather Service 205 5.4.12 The Relational Grid Monitoring Architecture (R-GMA) 209 5.4.13 visPerf 214 5.5 Other Monitoring Systems 217 5.5.1 Ganglia 217 5.5.2 GridMon 219 5.5.3 GRM/PROVE 220 5.5.4 Nagios 221 5.5.5 NetLogger 222 5.5.6 SCALEA-G 223 5.6 Summary 225 5.6.1 Resource categories 225 5.6.2 Native agents 225 5.6.3 Architecture 226 5.6.4 Interoperability 226 5.6.5 Homogeneous data presentation 226 5.6.6 Intrusiveness of monitoring 227 5.6.7 Information searching and retrieval 231 5.7 Chapter Summary 233 5.8 Further Reading and Testing 236 5.9 Key Points 236 5.10 References 236 www.it-ebooks.info CONTENTS ix Part Three Job Management and User Interaction 241 6 Grid Scheduling and Resource Management 243 Learning Objectives 243 Chapter Outline 243 6.1 Introduction 244 6.2 Scheduling Paradigms 245 6.2.1 Centralized scheduling 245 6.2.2 Distributed scheduling 246 6.2.3 Hierarchical scheduling 248 6.3 How Scheduling Works 248 6.3.1 Resource discovery 248 6.3.2 Resource selection 251 6.3.3 Schedule generation 251 6.3.4 Job execution 254 6.4 A Review of Condor, SGE, PBS and LSF 254 6.4.1 Condor 254 6.4.2 Sun Grid Engine 269 6.4.3 The Portable Batch System (PBS) 274 6.4.4 LSF 279 6.4.5 A comparison of Condor, SGE, PBS and LSF 288 6.5 Grid Scheduling with QoS 290 6.5.1 AppLeS 291 6.5.2 Scheduling in GrADS 293 6.5.3 Nimrod/G 293 6.5.4 Rescheduling 295 6.5.5 Scheduling with heuristics 296 6.6 Chapter Summary 297 6.7 Further Reading and Testing 298 6.8 Key Points 298 6.9 References 299 7 Workflow Management for the Grid 301 Learning Outcomes 301 Chapter Outline 301 7.1 Introduction 302 7.2 The Workflow Management Coalition 303 7.2.1 The workflow enactment service 305 7.2.2 The workflow engine 306 7.2.3 WfMC interfaces 308 7.2.4 Other components in the WfMC reference model 309 7.2.5 A summary of WfMC reference model 310 7.3 Web Services-Oriented Flow Languages 310 7.3.1 XLANG 311 7.3.2 Web services flow language 311 7.3.3 WSCI 313 7.3.4 BPEL4WS 315 7.3.5 BPML 317 7.3.6 A summary of Web services flow languages 318 www.it-ebooks.info [...]... assist the reader label topics more easily and hopefully help them get to grips with the content more easily The first section, “system infrastructure”, contains the chapters that discuss and outline the current architecture, services and instantiations of the Grid These chapters provide the underpinning information that the proceeding chapters build on The second section, “basic services”, contains the. .. learning about the core technologies that make up the Grid today The material being developed for the companion Web site will supplement the book’s content We intend that the book, along with Web content, will provide sufficient material to allow a complete self-study course of all the components addressed The book takes a bottom-up approach, addressing lower-level components first, then mid-level frameworks... The purpose of this book is not to convince the reader that one framework, technology or specification is better than another; rather its purpose is to expose the reader to a wide variety of what we call core technologies so that they can determine which is best for their own use This book is intended for postgraduate students and researchers from various fields who are interested in learning about the. .. utilities that can change their working environment (in the case of a Portal), or manage and schedule their jobs (in the case of workflow and scheduling systems) Finally, the last section of the book is called “Applications”; here we discuss a number of representative Grid-based applications that highlight the technologies and components discussed in the earlier chapters of the book www.it-ebooks.info... Grid-related technologies, tools and utilities to be taken up widely by the community at large, it is vital that developers design their software to conform to the relevant standards For the Grid community, the most important standards organizations are the Global Grid Forum (GGF) [10], which is the primary standards setting organization for the Grid, and OASIS [11], a notfor-profit consortium that drives the. .. found on a PC-based cluster This view then raises the question, what is the difference between a distributed system and the Grid? Obviously the Grid is a type of distributed system, but this does not really answer the question So, perhaps we should try and establish “What is a Grid?” In 1998, Ian Foster and Carl Kesselman provided an initial definition in their book The Grid: Blueprint for a New Computing... step-by-step approach to describe the middleware components that make up this virtual environment which is now called the Grid 1.2 CHARACTERIZATION OF THE GRID Before we go any further we need to somehow define and characterize what can be seen as a Grid infrastructure To start with, let us think about the execution of a distributed application Here The Grid: Core Technologies Maozhen Li and Mark Baker... Obviously, from the inception of book to its publication and distribution, the landscape that we describe will have undulated some more, so the book is a snapshot of the technologies during mid– late 2004 We believe that we can overcome some of the gaps that may appear in the book’s coverage of material by adding the appropriate content to the companion Web site www.it-ebooks.info List of Abbreviations Abbreviation... Management 7 Workflow Management for the Grid 8 Grid Portals Applications 9 Grid Applications – Case Studies Figure P.1 Organization of the book www.it-ebooks.info PREFACE xvii ORGANIZATION OF THE BOOK The organization of the book is shown in Figure P.P.1 We have organized the book into four general parts, which reflect the bottom-up view that we use to address the topics covered We know that certain... reviewed the book as a whole and fed back many useful comments about its presentation and content We would like to say a special thanks to Birgit Gruber, our Wiley editor, who worked closely with us through the production of the book, and generally made the effort involved a pleasant one COMPANION WEB SITE We have set up a Web site (coregridtechnologies.org) containing companion material to the book . in learning about the core technologies that make up the Grid today. The material being developed for the companion Web site will supplement the book’s content. We intend that the book, along with. specification is better than another; rather its purpose is to expose the reader to a wide variety of what we call core technologies so that they can determine which is best for their own use. This book. 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West