“replication-strategies” — 2007/4/24 — 10:56 — page 1 — #1 Research Report No. 2007:03 Replication Strategies for Streaming Media David Erman Department of Telecommunication Systems, School of Engineering, Blekinge Institute of Technology, S–371 79 Karlskrona, Sweden “replication-strategies” — 2007/4/24 — 10:56 — page 2 — #2 c  2007 by David Erman. All rights reserved. Blekinge Institute of Technology Research Report No. 2007:03 ISSN 1103-1581 Published 2007. Printed by Kaserntryckeriet AB. Karlskrona 2007, Sweden. This publication was typeset using L A T E X. “replication-strategies” — 2007/4/24 — 10:56 — page i — #3 Abstract Large-scale, real-time multimedia distribution over the Internet has been the subject of research for a substantial amount of time. A large number of mechanisms, policies, methods and schemes have been proposed for media coding, scheduling and distribution. Internet Protocol (IP) multicast was expected to be the primary transport mechanism for this, though it was never deployed to the expected extent. Recent developments in overlay networks has reactualized the research on multicast, with the consequence that many of the previous mechanisms and schemes are being re-evaluated. This report provides a brief overview of several important techniques for media broad- casting and stream merging, as well as a discussion of traditional IP multicast and overlay multicast. Additionally, we present a proposal for a new distribution system, based on the broadcast and stream merging algorithms in the BitTorrent distribution and repli- cation system. “replication-strategies” — 2007/4/24 — 10:56 — page ii — #4 ii “replication-strategies” — 2007/4/24 — 10:56 — page iii — #5 CONTENTS Contents 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Multicast 5 2.1 IP Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Application Layer Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3 Broadcasting Strategies 19 3.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2 Conventional Broadcasting . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Staggered Broadcasting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.4 Pyramid Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5 Staircase Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.6 Harmonic Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.7 Hybrid Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4 Stream Merging Strategies 25 4.1 Batching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 Piggybacking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.3 Patching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 Chaining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.5 Hierarchical and Hybrid Merging . . . . . . . . . . . . . . . . . . . . . . . 31 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5 Caching Strategies 33 5.1 Replacement Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5.2 Segment-based Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.3 Smoothing and Pre-fetching . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 iii “replication-strategies” — 2007/4/24 — 10:56 — page iv — #6 CONTENTS 6 BitTorrent Streaming 39 6.1 BitTorrent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.2 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6.3 Streaming Extensions for BitTorrent . . . . . . . . . . . . . . . . . . . . . 42 6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 7 Summary and Future Work 47 7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 iv “replication-strategies” — 2007/4/24 — 10:56 — page v — #7 LIST OF FIGURES List of Figures 2.1 Group Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Multicast architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1 Stream parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.1 Batching methods for a single video object. . . . . . . . . . . . . . . . . . 26 4.2 Piggybacking system state . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3 Chaining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 v “replication-strategies” — 2007/4/24 — 10:56 — page vi — #8 LIST OF FIGURES vi “replication-strategies” — 2007/4/24 — 10:56 — page vii — #9 LIST OF TABLES List of Tables 2.1 Group communication types. . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.1 Pagoda segment-to-channel mapping . . . . . . . . . . . . . . . . . . . . . 23 vii “replication-strategies” — 2007/4/24 — 10:56 — page viii — #10 LIST OF TABLES viii [...]... addressing and forwarding) necessary for media distribution to multiple clients • Broadcast strategies concern mechanisms for the segmentation of media objects and scheduling of media streams • Stream merging strategies concern mechanisms for the reduction of bandwidth consumption, typically by caching stream data in application for later redistribution • Caching strategies concern mechanisms for the buffering... “Broadcasting Strategies , several broadcasting schemes for streaming video are presented This is followed by Chapter 4: “Stream Merging Strategies , where we present methods and mechanisms for merging temporally disjoint media streams In Chapter 5: “Caching Strategies , we discuss caching mechanisms, and how caching of streaming objects relate to caching of Web objects Next, Chapter 6: “BitTorrent Streaming ,... large-scale media distribution Broadcasting strategies are considered because of the scheduling aspects of multimedia transmissions Stream merging strategies are discussed because of their bandwidth-conserving capability and relation to both broadcasting and caching We 2 1.2 OUTLINE also consider caching strategies, as these are important for decreasing bandwidth consumption, as well as for ALM to perform... Video-on-Demand (VoD), there is little information on applying the ideas and mechanisms from the former to the latter In this report, we provide an overview of four related topics: multicast systems, broadcasting strategies, stream merging strategies and caching mechanisms These form a foundation for a further discussion on using them in a BitTorrent-based system for VoD We discuss multicast, as this... when forwarding, since forwarding end hosts use unicast links for this However, using intelligent caching algorithms, ALM systems can significantly decrease media server bandwidth requirements [14] An additional issue for ALM is churn, where nodes frequently join or leave the system In addition to the specific issues of IPMC and ALM, there are more general issues with all media multicast systems For instance,... 25] Group communication as used by Internet users today is taken more or less for granted Forums and special interest groups abound, and the term “social networking” has become a popular buzzword These forums are typically formed as virtual meeting points for people with similar interests, that is, they act as focal points for social groups In this section, we discuss the technical aspects of group... most popular P2P applications [18], and proposals for adapting it to provide streaming services have been put forth While the original BitTorrent distribution model was designed for distributing large files in an efficient way, researchers have designed adaptations to the BitTorrent protocols and mechanisms so as to be able to use them as foundations for streaming systems [19, 20] 1.1 Motivation This research... same time, both joins and leaves should be performed expediently, i e., nodes should not have to wait for too long before joining or leaving a group Another important issue for RP-based protocols is the selection of an appropriate rendezvous point As the RP becomes an traffic aggregation point and single point of failure, it is also important to have a mechanism for quickly selecting a replacement RP in... thus considered as being an integral part of the next generation telecommunication networks, and the Internet as the primary distribution channel for this media The IMS is not the first proposed media- related killer application for the Internet A multitude of media applications were suggested in connection with the appearance of Internet Protocol Multicast (IPMC) [2–4] IPMC provided a method to send IP... playout rate, the disk I/O streams for the object on the server are also changed Each arriving client request for a specific video object causes a new I/O stream be allocated In essence, if a request for a video object arrives from client A at time t0 , and another request from client B arrives at time t1 , the display rate is increased for client B and decreased for client A until the I/O streams can . replication- strategies — 2007/4/24 — 10:56 — page 1 — #1 Research Report No. 2007:03 Replication Strategies for Streaming Media David Erman. addressing and forwarding) necessary for media distribution to multiple clients. • Broadcast strategies concern mechanisms for the segmentation of media objects