This page intentionally left blank Distributed Computing Principles, Algorithms, and Systems Distributed computing deals with all forms of computing, information access, and information exchange across multiple processing platforms connected by computer networks Design of distributed computing systems is a complex task It requires a solid understanding of the design issues and an in-depth understanding of the theoretical and practical aspects of their solutions This comprehensive textbook covers the fundamental principles and models underlying the theory, algorithms, and systems aspects of distributed computing Broad and detailed coverage of the theory is balanced with practical systems-related problems such as mutual exclusion, deadlock detection, authentication, and failure recovery Algorithms are carefully selected, lucidly presented, and described without complex proofs Simple explanations and illustrations are used to elucidate the algorithms Emerging topics of significant impact, such as peer-to-peer networks and network security, are also covered With state-of-the-art algorithms, numerous illustrations, examples, and homework problems, this textbook is invaluable for advanced undergraduate and graduate students of electrical and computer engineering and computer science Practitioners in data networking and sensor networks will also find this a valuable resource Ajay D Kshemkalyani is an Associate Professor in the Department of Computer Science, at the University of Illinois at Chicago He was awarded his Ph.D in Computer and Information Science in 1991 from The Ohio State University Before moving to academia, he spent several years working on computer networks at IBM Research Triangle Park In 1999, he received the National Science Foundation’s CAREER Award He is a Senior Member of the IEEE, and his principal areas of research include distributed computing, algorithms, computer networks, and concurrent systems He currently serves on the editorial board of Computer Networks Mukesh Singhal is Full Professor and Gartner Group Endowed Chair in Network Engineering in the Department of Computer Science at the University of Kentucky He was awarded his Ph.D in Computer Science in 1986 from the University of Maryland, College Park In 2003, he received the IEEE Technical Achievement Award, and currently serves on the editorial boards for the IEEE Transactions on Parallel and Distributed Systems and the IEEE Transactions on Computers He is a Fellow of the IEEE, and his principal areas of research include distributed systems, computer networks, wireless and mobile computing systems, performance evaluation, and computer security Distributed Computing Principles, Algorithms, and Systems Ajay D Kshemkalyani University of Illinois at Chicago, Chicago and Mukesh Singhal University of Kentucky, Lexington CAMBRIDGE UNIVERSITY PRESS Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521876346 © Cambridge University Press 2008 This publication is in copyright Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press First published in print format 2008 ISBN-13 978-0-511-39341-9 eBook (EBL) ISBN-13 hardback 978-0-521-87634-6 Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate To my father Shri Digambar and my mother Shrimati Vimala Ajay D Kshemkalyani To my mother Chandra Prabha Singhal, my father Brij Mohan Singhal, and my daughters Meenakshi, Malvika, and Priyanka Mukesh Singhal Contents Preface 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 page xv Introduction Definition Relation to computer system components Motivation Relation to parallel multiprocessor/multicomputer systems Message-passing systems versus shared memory systems Primitives for distributed communication Synchronous versus asynchronous executions Design issues and challenges Selection and coverage of topics Chapter summary Exercises Notes on references References 1 13 14 19 22 33 34 35 36 37 A model of distributed computations A distributed program A model of distributed executions Models of communication networks Global state of a distributed system Cuts of a distributed computation Past and future cones of an event Models of process communications Chapter summary Exercises Notes on references References 39 39 40 42 43 45 46 47 48 48 48 49 viii Contents 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 Logical time Introduction A framework for a system of logical clocks Scalar time Vector time Efficient implementations of vector clocks Jard–Jourdan’s adaptive technique Matrix time Virtual time Physical clock synchronization: NTP Chapter summary Exercises Notes on references References Global state and snapshot recording algorithms Introduction System model and definitions Snapshot algorithms for FIFO channels Variations of the Chandy–Lamport algorithm Snapshot algorithms for non-FIFO channels Snapshots in a causal delivery system Monitoring global state Necessary and sufficient conditions for consistent global snapshots Finding consistent global snapshots in a distributed computation Chapter summary Exercises Notes on references References Terminology and basic algorithms Topology abstraction and overlays Classifications and basic concepts Complexity measures and metrics Program structure Elementary graph algorithms Synchronizers Maximal independent set (MIS) Connected dominating set Compact routing tables Leader election 50 50 52 53 55 59 65 68 69 78 81 84 84 84 87 87 90 93 97 101 106 109 110 114 121 122 122 123 126 126 128 135 137 138 163 169 171 172 174 722 Figure 18.16 The simple Barabasi–Albert model [7] Peer-to-peer computing and overlay graphs Initially, there are m0 isolated nodes At each sequential step, perform one of the following operations: Growth Add a new node with m edges, (where m ≤ m0 ), that link the new node to m different nodes already in the system Preferential attachment The probability that the new node will be connected to node i depends on the degree ki , such that: ki (18.12) ki = j kj • Rather than begin with a constant number of nodes n that are then randomly connected or rewired, real networks (e.g., WWW, INTERNET) exhibit growth by the addition of nodes and edges • Rather than assume that the probability of adding (or rewiring) an edge between two nodes is a constant, real networks exhibit the property of preferential attachment, where the probability of connecting to a node depends on the node degree The simple Barabasi–Albert model [7], which captures growth and preferential attachment, is described in Figure 18.16 After t time steps, there are t + m0 nodes and mt edges Numerically, it is verified that the degree distribution follows a power law with degree =3, that is independent of the parameter m Two techniques to analyze the degree distribution of models are now described in the context of the BA model The master-equation approach was introduced by Dogorotsev et al [13] and the rate-equation approach was introduced by Krapivsky et al [22] 18.13.1 Master-equation approach Let p k ti t denote the probability that, at time t, a node i that was added at time ti has degree k When a new node with m edges is added to the graph, the degree of node i increases by one with probability m · k = k/2t Hence, we have [4, 13]: p k ti t + = k−1 k · p k − ti t − − · p k ti t 2t 2t (18.13) The first term is the probability that a node with k − degree gets a new edge; the second term is the probability that a node with degree k does not get a new edge Based on this formulation, the degree distribution can be expressed as: P k = limitt→ p k ti t /t ti (18.14) 723 18.14 Evolving networks From Eq (18.13), it can be shown that: Pk = k−1 P k+2 m+2 k−1 if k ≥ m + (18.15) if k = m This solves as: Pk = 2m m + k k+1 k+2 (18.16) 18.13.2 Rate-equation approach Let nk t be the average number of nodes having k edges at time t When a new node is added, nk t changes as follows New edges are added to some nodes with degree k − 1, new edges are added to some nodes with degree k, and new nodes with m edges are added These three changes affect nk t in the following manner: dnk = m· dt k · nk t k − · nk−1 t − k knk t k knk t + km (18.17) By taking the asymptotic limit, nk t = t · P k , and k knk t = 2mt This yields the same recursive Eq (18.15) obtained using the master-equation approach 18.14 Evolving networks The BA algorithm in Figure 18.16 represents a basic model that cannot fully capture real network properties For example, the BA model has a fixed exponent of for the power law, independent of the parameter m Real networks have an exponent that varies, typically between and Some real networks sometimes have exponential cutoffs that are not within the power law regime The study of more general and flexible models that can accurately capture real networks has lead to several notable directions of investigation: • Preferential attachment The BA model assumed that the probability k that a new node connects to a node i is proportional to the degree ki This implied that k is linearly proportional to k It has been shown analytically that for sublinear preferential attachment as well as for superlinear preferential attachment, the scale-free nature of the network cannot be preserved In real networks, there is a finite probability that a new node attaches to an isolated node, i.e., = and k = C + k , where C denotes the intial 724 Peer-to-peer computing and overlay graphs • • • • • Figure 18.17 The extended Barabasi–Albert model [3] attractiveness It can be seen that initial attractiveness changes the degree exponent but preserves the scale-free nature of the degree distribution Growth The BA model assumed that the rate of addition of nodes and edges was uniform Many real networks, such as INTNET, AS, WEB, SUBSTRATE, and WORDOCC, have the property that the number of edges increases faster than the number of nodes, implying an increase in the average degree as the number of nodes increases It has been shown analytically that accelerated growth does not affect the power law nature although the exponent degree is altered Local events Real networks undergo local (microscopic) changes to the topology, such as node addition and node deletion, edge addition and edge deletion A popular model that explores the properties of such local events is the extended Barabasi–Albert model [3], shown in Figure 18.17 Growth constraints Real networks often have bounded capacity for the number of edges (e.g., connections at a router) or a finite lifetime for the nodes (as in social networks) In the electrical power distribution network which exhibits an exponential distribution, there are practical reasons why the node degree is bounded In the actors network, which exhibits a power law with an exponential cutoff for large k, ageing limits the accrual of new edges Thus, ageing and finite capacity need to explicitly captured in a good model for such networks Competition Real-world networks exhibit competition, wherein some nodes can attract more edges (e.g., via advertising) at the cost of other nodes This feature can be modeled by a fitness parameter Similarly, a new node may inherit edges belonging to some other node or nodes (e.g., modifying a replica of a Web page) This needs to be explicitly modeled Induced preferential attachment Various local-level mechanisms, such as the copying mechanism (copy edges of another node as in Web Initially, there are m0 isolated nodes At each sequential step, perform one of the following operations: With probability p, add m, where m ≤ m0 , new edges For each new edge, one end is randomly selected, the other end with probability ki = ki + j kj + (18.18) With probability q, rewire m edges To rewire an edge, randomly select node i, delete some edge i w , add edge i x to node x that is chosen with probability kx as per Eq (18.18) With probability − p − q, insert a new node Add m new edges to the new node, such that with probability ki , an edge connects to a node i already present before this step 725 18.14 Evolving networks pages), and tracing selected walks (as in recursively following the citation trail in a citation network), need to be modeled because they implicitly introduce preferential attachment 18.14.1 Extended Barabasi–Albert model The extended BA model [3] is an example model for evolving networks Continuum theory analysis In continuum theory, it is assumed that ki changes continuously and the probability ki then represents the rate at which ki changes Each of the three possible events in a sequential step can affect the rate at which ki changes as follows [3]: With probability p, m new links are added For each link, one end is randomly chosen, leading to a change in ki of pm/n For each link, the second end attaches preferentially, leading to a change in ki of pm· kik+1+1 j j Hence, ki + j kj + dki = pm + pm dt n (18.19) With probability q, m existing links are rewired For each rewired link, a randomly chosen node loses one incident edge, which then attaches preferentially Thus, the impact on ki is: dki = −qm + qm dt n ki + j kj + (18.20) With probability − p − q , a new node is added with m links Each of the m links connects preferentially, thus: dki = 1−p−q m dt ki + j kj + (18.21) Summing the three effects, we have: dki = p−q m +m dt n ki + j kj + (18.22) As the system size and topology varies with time, we have: n t = m0 + − p − q t kj = 2mt − q − m j (18.23) 726 Peer-to-peer computing and overlay graphs As t increases, the constants m and m0 can be deleted Further, for a node added at ti , we have that ki ti = m (the initialization step) Exercise 18.8 asks you to show that the solution to Eq (18.22) has the form ki t = A p q m + m + A p q m = p−q t ti 1/B p q m 2m − q +1 1−p−q −A p q m −1 Bp q m = (18.24) 2m − q + − p − q m (18.25) Based on further algebraic derivations, Albert and Barabasi [3] showed that: k+ Pk p q m − pqm where p q m = A p q m + and p q m = B p q m +1 (18.26) Equation (18.26) is valid if, for a fixed p and m, q < qmax = − p − p + m / + 2m There are now two cases: q < qmax : Eq (18.26) is valid and the degree distribution is a power law and is scale-free q > qmax : Eq (18.26) is invalid, and P k can be shown to behave like an exponential distribution The model now behaves like the ER and WS models This is similar to the behavior seen in real networks – some networks show a power law while others show an exponential tail – and a single model can capture both behaviors by tuning the parameter q The scale-free regime 1.0 E q SF 0 p 1.0 Figure 18.18 Phase diagram for the extended Barabasi–Albert model [3] SF denotes the scale-free regime, which is enclosed by the thick border E denotes the exponential regime that exists in the remainder of the lower diagonal region of the graph The plain line shows the boundary for m = 1, having a y-axis intercept at 0.67 727 18.16 Exercises and the exponential regime are marked in the graph in Figure 18.18 The boundary between the two regimes depends on the value of m and has slope −m/ + 2m The area enclosed by thick lines shows the scale-free regime; the dashed line is its boundary when m → and the dotted line is its boundary when m → 18.15 Chapter summary Peer-to-peer (P2P) networks allow equal participation and resource sharing among the users This chapter first analyzed the different types of P2P networks Unstructured P2P networks are like Gnutella and BitTorrent We studied different search mechanisms – flooding, constrained flooding, and blind search – for such unstructured networks We also examined some data replication strategies, and their impact on the search performance The chapter then studied three classical structured P2P networks – Chord, CAN, Tapestry – all of which use the distributed hash table concept in their implementations Although all the three mechanisms differ, they are similar in that they represent different tradeoffs in search efficiency, i.e., path length, and the amount of local storage for implementing the hash tables The spectrum of P2P networks from unstructured to structured offer a wide range of tradeoffs for user requirements The chapter also examined issues such as fairness and trust management These issues are important because, in the P2P environment where there is no control authority, the system must be able to autonomously alllow for fairness The Internet, AS-AS level internets, and Web (WWW) overlays exhibit some interesting properties about how they grow and evolve Many network overlays outside of computer science also exhibit the same properties The chapter studied several properties of the Internet and Web graphs Then, in a more general setting, the chapter examined random networks, small-world networks, node degree distributions, scale-free networks, and the impact of error and attack tolerance on such networks Networks grow in an uncontrolled fashion, yet there seems to be some underlying basis for such growth Of the several proposals to model the growth of networks, we studied the Barabasi– Albert model, which appears to be promising in its applicability to not just computer science networks, but also to networks in other disciplines and natural phenomena 18.16 Exercises Exercise 18.1 (Replication) Derive the values of average search size A, Ai , and utilization ui for square-root replication The derived answers should match the entries in Table 18.3 728 Peer-to-peer computing and overlay graphs Exercise 18.2 (Fault-tolerance in Chord) Adapt the code in Algorithm 18.3 so that the nodes manage a successor list of successors, rather than a single successor Exercise 18.3 (Chord) In the Chord protocol, assume that the successor list at each node has = log n nodes Show the following: If a Chord ring is initially stable, and if the probability of subsequent failure of each node is 0.5, then Locate_Successor returns the closest functional successor node to the key being searched with high probability If a Chord ring is initially stable, and if the probability of subsequent failure of each node is 0.5, it takes O log n average-case time for Locate_Successor to complete Exercise 18.4 (CAN) Compute the time and message complexity of the distributed region reassignment protocol that is run periodically by the CAN protocol Exercise 18.5 (CAN) Identify all the changes to the base CAN protocol to accommodate the optimization of overloading coordinate regions, discussed in Section 18.5.5 Exercise 18.6 (Power law in the Internet [31]) Show that the number of edges l in the Internet graph that obeys the power law for the rank exponent is given as follows Let the graph have n nodes and rank exponent Then: l∼ 1 − +1 n +1 n Exercise 18.7 Show that Eq (18.15) using the master-equation approach for the degree distribution in the extended BA model can be solved as Eq (18.16) Exercise 18.8 Show that the solution to Eq (18.22) for the degree distribution in the extended BA model using continuum theory analysis is given by Eq (18.25) 18.17 Notes on references The introduction is based on the survey by Risson and Moors [29] and AndroutsellisTheotokis and Spinellis [6] The discussion on replication and search in unstructured networks is based on Cohen and Shenker [12], and on Lv et al [23], respectively Gnutella [16,17], Napster [25], and Freenet [10] are widely implemented commercial P2P protocols The Chord protocol was proposed by Stoica et al [32] The content addressible network (CAN) was proposed by Ratnasamy et al [27] The design of Tapestry [20, 21, 35, 36] and the related Pastry [30] overlay was based on the ideas of Plaxton trees proposed by Plaxton et al [26] Tapestry built on the Plaxton trees by providing better fault-tolerance and resilience in the face of node joins and departures The discussion on fundamental tradeoffs between routing table size and network diameter is based on Xu et al [34] and Ratnasamy et al [28] The BitTorrent system was initially proposed by Cohen [11] The discussion of trust management is based on Gupta et al [18, 19] and Aberer and Despotovic [1] The discussion on the graph structures of complex networks is structured and based on the excellent survey by Albert and Barabasi [4] The discussion on power laws and Zipf’s law is taken from the tutorial by Adamic [2] The power laws for the Internet 729 References were discovered by Siganos and the Faloutsos brothers [31] The discussion on the betweenness centrality metric for graphs is based on the work by Goh et al [15] The random graphs model was proposed and analyzed by Erdos and Renyi [14] Further results on the properties on random graphs were given by Bollobas [8, 9] The small worlds model was proposed by Watts and Strogatz [33] The extended Barabasi–Albert model for graph evolution was given by Albert and Barabasi [3] The analysis of error and attack tolerance on exponential networks and on scale-free networks was done by Albert et al [5] References [1] K Aberer and Z Despotovic, Managing trust in a peer-to-peer information system, Proceedings of the 10th International Conference on Information and Knowledge Management, Atlanta, Georgia, USA, November 2001, 310–317 [2] L Adamic, Zipf, Power-Laws, and Pareto – A Ranking Tutorial, available online at: www.hpl.hp.com/research/idl/papers/ranking/ranking.html [3] R Albert and A.-L Barabasi, Topology of evolving networks: local events and universality, Physical Review Letters, 85(24), 2000, 5234–5237 [4] R Albert and A.-L Barabasi, Statistical mechanics of complex networks, Review of Modern Physics, 74(1), 2002, 47–97 [5] R Albert, H Jeong, and A Barabasi, Error and attack tolerance of complex networks, Nature, 406, 2000, 378–381 [6] S Androutsellis-Theotokis and D Spinellis, A survey of peer-to-peer content distribution technologies, ACM Computing Surveys, 36(4), 2004, 335–371 [7] A.-L Barabasi and R Albert, Emergence of scaling in random networks, Science, 286, 1999, 509–512 [8] B Bollobas, Degree sequences of random graphs, Discrete Math, 33, 1981, 1–9 [9] B Bollobas, Random Graphs, London, Academic Press, 1985 [10] I Clarke, O Sandberg, B Wiley, and T W Hong, Freenet: a distributed anonymous information storage and retrieval system, Workshop on Design Issues in Anonymity and Unobservability, Berkeley, CA, July 2000, 46–66 [11] B Cohen, Incentives Build Robustness in BitTorrent, available online at: www.bittorrent.com/bittorrentecon.pdf [12] E Cohen and S Shenker, Replication strategies in unstructured peer-to-peer networks, ACM SIGCOMM, 2002, 177–190 [13] S Dogorotsev, J Mendes, and A Samukhin, Structure of growing networks: exact solution of the Barabasi–Albert model, Physical Review Letters, 85, 2000, 4633–4636 [14] P Erdos and A Renyi, Random graphs 6, 1959, 290– [15] K Goh, E Oh, H Jeong, B Kahng, and D Kim, Classification of scale-free networks, Proceedings of the National Academy of Sciences, 2002 [16] Gnutella, www.gnutella.com/ [17] The Gnutella protocol specification, available online at: www9.limewire com/developer/gnutella_protocol_0.4.pdf [18] M Gupta, P Judge, and M Ammar, A reputation system for peer-to-peer networks, Proceedings of the 13th International Workshop on Network and Operating Systems Support for Digital Audio and Video, Monterey, CA, June 2003, 144–152 730 Peer-to-peer computing and overlay graphs [19] M Gupta, M H Ammar, and M Ahamad, Trade-offs between reliability and overheads in peer-to-peer reputation tracking, Computer Networks, 50(4), 2006, 501–522 [20] K Hildrum, J Kubiatowicz, S Rao, and B Y Zhao, Distributed object location in a dynamic network, Proceedings of ACM SPAA 2002, 41–52 [21] K Hildrum, J Kubiatowicz, S Rao, and B Y Zhao, Distributed object location in a dynamic network, Theory of Computing Systems, 37, 2004, 405–440 [22] P Krapivsky, S Redner, and F Leyvraz, Connectivity of growing random networks, Physical Review Letters, 85, 2000, 4629–4632 [23] Q Lv, P Cao, E Cohen, K Li, and S Shenker, Search and replication in unstructured peer-to-peer networks, International Conference on Supercomputing, 2002, 84–95 [24] S Milgram, The small world problem, Psychology Today, 1(2), 1967, 60–67 [25] Napster, www.napster.com/ [26] C G Plaxton, R Rajaraman, and A W Richa, Accessing nearby copies of replicated objects in a distributed environment, Proceedings of ACM SPAA 1997, 311–320 [27] S Ratnasamy, P Francis, M Handley, R.M Karp, and S Shenker, A scalable content-addressable network, Proceedings of ACM SIGCOMM 2001, 161–172 [28] S Ratnasamy, I Stoica, and S Shenker, Routing algorithms for DHTs: some open questions, Proceedings of IPTPS 2002, 45–52 [29] J Risson and T Moors, Survey of research towards robust peer-to-peer networks: search methods, Computer Networks, 50(17), 2006, 3485–3521 [30] A Rowstron and P Druschel, Pastry: scalable, distributed object location and routing for large-scale peer-to-peer systems, Proceedings of the IFIP/ACM Middleware 2001, Heidelberg, Germany, November 2001, 329–350 [31] G Siganos, M Faloutsos, P Faloutsos, and C Faloutsos, Power laws and the ASlevel internet topology, IEEE/ACM Transactions on Networking, 11(4), 2003, 514–524 [32] I Stoica, R Morris, D Liben-Nowell, D Karger, M.F Kaashoek, F Dabek, and H Balakrishnan, Chord: a scalable peer-to-peer lookup service for internet applications, IEEE Transactions on Networking, 11(1), 2003, 17–31 [33] D J Watts and S H Strogatz, Collective dynamics of “Small World” networks, Nature, No 393, 1998, 440–442 [34] J Xu, A Kumar, and X Yu, On the fundamental tradeoffs between routing table size and network diameter in peer-to-peer networks, IEEE Journal on Selected Areas in Communications, 22(1), 2004, 151–163 [35] B Y Zhao, L Huang, J Stribling, S Rhea, A Joseph, and J Kubiatowicz, Tapestry: a resilient global-scale overlay for service deployment, IEEE Journal on Selected Areas in Communications, 22(1), 2004, 41–53 [36] B Y Zhao, J D Kubiatowicz, and A D Joseph, Tapestry: An Infrastructure for Fault-Resilient Wide-Area Location and Routing, Technical Report UC Berkeley, CSD-01-1141, University of California at Berkeley, Berkeley, CA, 2001 Index Compare&Swap, 548 Fetch&Increment, 548 Swap, 432 Test&Set, 432 Definitely, 383 Possibly, 383 Read-Modify-Write, 551 Abadi, M, 601, 627 accuracy properties, 571 Acharya, 107 adaptive algorithms, 130 Afek, Y, 655, 656 Agarwal, D, 331 Agrawala, AK, 312 agreement failure-free system, 515 Alagar, 108 Alvisi, L, 507 anonymous algorithms, 130 antimessages, 73 Arora, A, 635, 648, 655 asynchronous execution, 19 asynchronous system, 132 atomic broadcast, 583 atomic registers, 436 authentication, 599 authentication protocol failures, 625 authentication protocols with asymmetric cryptosystem, 615 with symmetric cryptosystem, 602 authentication server, 605 Badrinath, 107 Baldoni, R, 507 731 Barabasi-Albert model extended Barabasi-Albert model, 725 Bellovin, SM, 623 Bhargava, B, 506 Bremler, A, 656 Briatico, D, 470 broadcast, 148 Burrows, M, 627 Byzantine agreement, 512 exponential tree algorithm, 519 upper bound, 517 Cao, G, 467 causal delivery, 106 causal order, 206 optimal algorithm, 208 Raynal-Sciper-Toueg algorithm, 207 causal ordering, 43 causal path, 112 causal precedence relation, 41 Chakrabarti, S, 628 Chandra, TD, 568, 570, 573, 578, 583, 584 Chandrasekaran, S, 253 Chandy, 93, 362, 364 Chandy, KM, 375 channel state recording, 107 checkpoint, 110, 458 global, 459 local, 458 checkpointing communication-induced, 468 coordinated, 465 uncoordinated, 464 checkpointing algorithm Helary-Mostefaoui-Netzer-Raynal protocol, 499 732 Index checkpointing algorithm (cont.) Juang and Venkatesan algorithm, 478 Koo-Toueg, 476 Manivannan-Singhal algorithm, 483 Peterson-Kearns algorithm, 492 Chord, 688 churn, 691 clock inaccuracies, 80 clock offset, 79 clock skew, 79 clocks matrix, 68 physical, 78 scalar, 53 vector, 55 closure, 634 clustering, 713 common clock primitives, 648 common knowledge, 292 concurrent common knowledge, 293 Epsilon common knowledge, 292 eventual common knowledge, 292 protocols for concurrent common knowledge, 295 timestamped common knowledge, 293 communication asynchronous, 47 synchronous, 47 communication primitives, 14 asynchronous, 15 blocking, 15 non-blocking, 15 synchronous, 15 Compare&Swap, 550 completeness properties, 570 complex networks Barabasi-Albert model, 722 error and attack tolerance, 718 graph structures, 712 Internet, 715 Internet graph, 714 complexity metrics, 135 concurrency, 12 concurrency measure, 51 consistent cut, 91 conjunctive predicate detection interval-based piggybacking algorithm, 401 interval based algorithm, 389 interval-based token algorithm, 397 state-based token algorithm, 395 conjunctive predicate detetion state-based algorithm, 392 consensus, 513 k-set consensus, 532 approximate agreement, 533 impossibility in shared memory asynchronous systems, 544 impossibility result for asynchronous systems, 529 phase king algorithm, 526 reliable broadcast, 544 renaming problem, 538 shared memory k-set consensus, 556 terminating reliable broadcast, 531 transaction commit, 532 wait-free renaming using splitters, 560 wait-free shared memory renaming, 557 consensus hierarchy, 547 consensus problem, 577 solution using eventually strong FD, 580 solution using strong FD, 578 consensus under crash failures, 517 Consistent global snapshots, 113 consistent global snapshots necessary and sufficient conditions, 110 consistent global state, 91 consistent global states, 44 content-addressible networks (CAN), 695 convergecast, 148 convergence, 634 crown, 197 cryptographic protocols design principles, 601 cut, 91 cut in a ditributed computation, 45 data indexing, 679 deadlock avoidance, 354 Chandy-Misra-Haas algorithm, 362, 364 detection, 354 Kshemkalyani-Singhal algorithm, 365 Mitchell-Merritt algorithm, 360 phantom, 355 prevention, 353 resolution, 355 deadlock detection, 354 deadlocks, 330 diffusing computations based algorithms, 359 edge-chasing algorithm, 359 733 Index global state detection based algorithms, 359 path-pushing algorithms, 358 degree distributions, 713 delayed messages, 461 Delporte-Gallet, C, 586 deterministic execution, 131 dictionary attack, 623 diffusion computation, 359 Dijkstra, E, 631, 636, 662 distributed deadlock, 352 distributed discrete event simulations, 77 Distributed Program, 39 distributed reset, 648 distributed systems characteristics, design issues, 22 Dolev, S, 652, 660 duplicate messages, 461 dynamic termination detection, 261 El Abbadi, A, 331 Elnozahy, EN, 507 emulations, 21 message-passing, 14 shared memory, 14 synchronous system, 21 Encrypted Key Exchange (EKE) protocol, 623 enumerating consistent snapshots, 118 event counting, 54 eventual accuracy properties, 571 evolving networks, 723 executions realizable with synchronous communication, 196 timestamps, 198 failure detector adaptive, 591 implementation, 589 realistic, 586 weakest, 588, 589 failure detectors, 569 reducibility, 572 types, 572 failure pattern, 569 failure recovery, 462 Fowler, 62 free-riding, 708 Fuchs, WK, 469 future cone of an event, 46 Garg, V, 596 Gartner, F, 652 generalized deadlocks, 365 generalized random graph networks, 720 Gligor, V, 375 global state, 43, 92 consistent, 91 global virtual time, 75 Gnutella, 682 Gouda, M, 635, 648, 649, 655, 662 graph algorithms, 138 maximal independent set (MIS), 169 all sources shortest paths, 151 compact routing tables, 172 connected dominating set (CDS), 171 constrained flooding, 155 delay bounded Steiner trees, 233 distance vector rouitng, 150 leader election, 174 minimum weight spanning tree, 157, 162 reverse path forwarding, 230 single source shortest path, 149, 151 spanning tree, 138, 140, 143, 146 Steiner trees, 231 group communication, 205 fault-tolerant, 228 multicast, 220 Guerraoui, R, 596 Haas, L, 362, 364 Helary, 100 Helary, JM, 499 Herman, T, 375 Huang, ST, 656 illegitimate state, 634 impersonation attack, 618 incarnation number, 486 incremental snapshot, 99 inhibition, 131 interactive consistency, 513 interconnection networks, 6, 679 Israeli, A, 652 Jard, 65 Johnson, D, 507 Jourdan, 65 Juang, 478 Kaminsky, M, 628 Kasami, 336 Katz, S, 664 Kearns, 97, 109 734 Index Kearns, P, 492 Kerberos authentication service, 611 authenticator, 613 Kerberos authentication service, 611, 613 Kim, KH, 507 Knapp, E, 358, 375 knowledge agreement, 291 asynchronous system, 290 logic, 283 multi-dimensional clocks, 300 operators, 283 properties, 288 transfer, 298 Koo, R, 476 Kripke structures, 285 Kshemkalyani, AD, 60, 321, 365, 375 Kutten, S, 655 Lai, 102 Lam, S, 627 Lamport, 93, 309 Lamport’s happens before relation, 41 layering, 647 lazy failure detection protocol, 592 legitimate state, 634 Lodha, 321 log-based rollback recovery, 470 logging causal, 474 optimistic, 473 pessimistic, 472 logical clocks, 52 lost messages, 461 Maekawa, M, 328 Manivannan, 118, 483 Marzullo, K, 506 matrix clocks, 68 matrix time, 68 Mattern, F, 105, 263 memory consistency, 413 atomic consistency, 414 causal consistency, 420 hierarchy, 424 linearizability, 414 pipelined RAM (PRAM), 422 processor consistency, 422 sequential consistency, 417 slow memory, 423 Menasce, D, 360, 375 Merritt, M, 360, 623 message ordering, 190 asynchronous executions, 190 hierarchy, 199 synchronous executions, 194 message ordering paradigms causal order, 191 FIFO executions, 191 Misra, 362, 364 Mitchell, 360 modularization, 647 monitoring global state, 109 Moran, S, 652 Mostefaoui, 499 muddy children puzzle, 282 multicast, 220 core-based trees, 235 destination agreement based, 227 fixed sequencer based, 227 history based, 226 moving sequencer based, 227 privilege based, 226 propagation trees, 221 Muntz, R, 375 mutual exclusion Agarwal-El Abbadi algorithm, 331 fast mutual exclusion, 429 hardware-assisted, 432 Lamport’s algorithm, 309 Lamport’s bakery algorithm, 427 Lodha-Kshemkalyani algorithm, 321 Maekawa’s algorithm, 328 quorum-based algorithms, 327 Raymond’s algorithm, 339 Ricart-Agrawala algorithm, 312 Singhal’s dynamic algorithm, 315 Suzuki-Kasami algorithm, 336 token-based algorithms, 336 Napster, 678 Needham and Schroeder protocol, 617 Needham, R, 601, 617, 627 Network Time Protocol (NTP), 80 Netzer, R, 113, 118, 499 non-blocking universal algorithm, 553 nonce, 604 non-deterministic execution, 131 object replication, 176 one-time password, 606 735 Index orphan messages, 461 Otway-Rees protocol, 609 overlays, 126, 679 structured, 680 unstructured, 681 parallel system, coupling, 11 Flynn’s taxonomy, 10 interconnection networks, multiprocessor, parallelism, 12 Pareto law, 715 partial synchrony, 590 partially synchronous models, 590 past cone of an event, 46 path causal, 112 zigzag, 112 peer-to-peer flooding, 683 proportional replication, 686 random walk, 684 replication, 686 square-root replication, 686 uniform replication, 686 Peterson, SL, 492 physical clock synchronization, 78 physical clocks, 78 power law, 714 Prakash, R, 507 predicates, 380 conjunctive, 388 disjunctive, 404 modalities, 382 observer-independent, 405 relational, 384 stable, 380 unstable, 382 Prisoners’ dilemma, 708 probabilistic self-stabilization, 636 probe message, 363 program structure, 137 progress, 355 pseudo stabilization, 669 pseudo-stabilizing system, 636 public key certificate, 616 R-graph, 119 Ramamoorthy, CV, 375 randomized self-stabilization, 635 Raymond, K, 339 Raynal, M, 499, 596 Reducing Weak FD to a Strong FD, 573 register hierarchy, 434 regular registers, 436 relational predicate detection, 384 rendezvous, 201 reputation management, 709 Ricart, 312 Richard, G, 506 safe registers, 435 safety, 355 scalar time, 53 scale-free networks, 717, 719, 721 Schiper, A, 596 Schneider, M, 662 Schroeder, MD, 617 Secure Remote Password (SRP) protocol, 624 secure sockets layer, 619 self-stabilization, 634 cost, 646 for fault folerance, 665 role of compilers, 662 self-stabilizing algorithm for 1-maximal independent set, 657 self-stabilizing distributed spanning trees, 650 self-stabilizing token ring, 636 shared memory, 410, 456 shared memory mutual exclusion, 427 simultaneous regions, 109 Singhal, 60, 118, 315, 365, 375, 467, 483 Sistla, P, 506 small-world networks, 713, 720 snapshots, 274 solution to atomic broadcast, 584 spanning tree, 247 Spezialetti, 97, 109 splitters, 560 SSL Protocol, 619 stable property, 97 stable storage, 456–458 starvation, 345 state lattice, 384 static termination detection, 259 Strom, 506 surface of the future cone, 47 surface of the past cone, 47 Suzuki, 336 736 Index symmetry, 667 synchronizers, 163 synchronizer, 165 synchronizer, 166 synchronizer, 166 simple synchronizer, 164 synchronous execution, 19 synchronous order, 202 synchronous system, 132 Tapestry, 701 termination detection, 243, 368 atomic computation model, 263 channel counting method, 270 distributed snapshots, 243 faulty distributed system, 272 four counter method, 265 message-optimal, 253 spanning-tree based, 247 vector counters method, 268 very general model, 257 weight throwing, 245 time matrix, 68 physical, 78 scalar, 53 vector, 55 virtual, 69 time warp mechanism, 72 time-space diagram, 40 topology based primitives, 648 total order, 215 centralized algorithm, 216 three-phase distributed algorithm, 216 total order property, 583 total ordering, 54 Toueg, S, 476, 568, 570, 573, 578, 583, 584 transient failure, 635 tree-structured quorum, 331 Tseng, 273 uniform algorithms, 130 uniform consensus, 586 universality of consensus objects, 552 useless checkpoints, 464, 469 vector clocks, 55 efficient implementations, 59 size, 57 vector clocks size, 57 vector time, 55, 492 Venkatesan, S, 99, 108, 253, 478 virtual time, 69 wait-for-graph (WFG), 353 wait-free algorithms, 134 wait-free atomic snapshot, 447 wait-free consensus Compare&Swap, 550 wait-free register simulations, 437–440, 442, 444, 445 wait-free simulations, 434 wait-free universal algorithm, 556 wait-freedom, 434 Wang,Y-M, 115 Watts Strogatz model, 721 weight-throwing scheme, 273 Welch, J, 506 wide-mouth frog protocol, 605 Woo, T, 627 Wu, TD, 624 Xu, 113 Yang, 102 Yemini, 506 Yung, M, 655 zigzag cycle, 112 zigzag path, 112 Zipf’s law, 715 Zwaenepoel, 62 ...This page intentionally left blank Distributed Computing Principles, Algorithms, and Systems Distributed computing deals with all forms of computing, information access, and information... fundamental principles and models underlying all aspects of distributed computing It will address the principles underlying the theory, algorithms, and systems aspects of distributed computing. .. 36 37 A model of distributed computations A distributed program A model of distributed executions Models of communication networks Global state of a distributed system Cuts of a distributed computation