TCOM 501: Networking Theory & Fundamentals Lecture February 19, 2003 Prof Yannis A Korilis 6-2 Topics Time-Reversal of Markov Chains „ Reversibility „ Truncating a Reversible Markov Chain „ Burke’s Theorem „ Queues in Tandem „ 6-3 Time-Reversed Markov Chains „ {Xn: n=0,1,…} irreducible aperiodic Markov chain with transition probabilities Pij ∑ „ ∞ j =0 Pij = 1, i = 0,1, Unique stationary distribution (πj > 0) if and only if: π j = ∑ i =0 π i Pij , ∞ „ j = 0,1, Process in steady state: Pr{ X n = j} = π j = lim Pr{ X n = j | X = i} n →∞ „ „ „ Starts at n=-∞, that is {Xn: n = …,-1,0,1,…} Choose initial state according to the stationary distribution How does {Xn} look “reversed” in time? 6-4 Time-Reversed Markov Chains Define Yn=Xτ-n, for arbitrary τ>0 „ {Yn} is the reversed process Proposition 1: „ {Yn} is a Markov chain with transition probabilities: „ Pij* = „ π j Pji πi , i, j = 0,1, {Yn} has the same stationary distribution πj with the forward chain {Xn} 6-5 Time-Reversed Markov Chains Proof of Proposition 1: Pij* = P{Ym = j | Ym −1 = i, Ym − = i2 , … , Ym − k = ik } = P{ X τ − m = j | X τ − m +1 = i, X τ − m + = i2 , … , X τ − m + k = ik } = P{ X n = j | X n +1 = i, X n + = i2 , … , X n + k = ik } = P{ X n = j , X n +1 = i, X n + = i2 , … , X n + k = ik } P{ X n +1 = i, X n + = i2 , … , X n + k = ik } = P{ X n + = i2 , … , X n + k = ik | X n = j , X n +1 = i}P{ X n = j , X n +1 = i} P{ X n + = i2 , … , X n + k = ik | X n +1 = i}P{ X n +1 = i} = P{ X n = j , X n +1 = i} = P{ X n = j | X n +1 = i} = P{Ym = j | Ym −1 = i} P{ X n +1 = i} = P{ X n +1 = i | X n = j}P{ X n = j} Pji π j = P{ X n +1 = i} πi ∑ i =0 π P = ∑ i =0 πi ∞ * i ij ∞ π j Pji πi = π j ∑ i =0 Pji = π j ∞ Reversibility 6-6 Stochastic process {X(t)} is called reversible if „ (X(t1), X(t2),…, X(tn)) and (X(τ-t1), X(τ-t2),…, X(τ-tn)) „ have the same probability distribution, for all τ, t1,…, tn „ „ Markov chain {Xn} is reversible if and only if the transition probabilities of forward and reversed chains are equal Pij = Pij* or equivalently, if and only if π i Pij = π j Pji , i, j = 0,1, Detailed Balance Equations ↔ Reversibility Reversibility – Discrete-Time Chains 6-7 „ Theorem 1: If there exists a set of positive numbers {πj}, that sum up to and satisfy: π i Pij = π j Pji , i, j = 0,1, Then: {πj} is the unique stationary distribution The Markov chain is reversible „ Example: Discrete-time birth-death processes are reversible, since they satisfy the DBE 6-8 Example: Birth-Death Process Pn −1,n P01 Sc Pn ,n +1 n Pn ,n −1 P10 P00 S Pn ,n n+1 Pn +1,n One-dimensional Markov chain with transitions only between neighboring states: Pij=0, if |i-j|>1 „ Detailed Balance Equations (DBE) „ π n Pn ,n +1 = π n +1Pn +1,n „ n = 0,1, Proof: GBE with S ={0,1,…,n} give: n ∞ n ∞ ∑∑π P =∑∑πP j =0 i = n +1 j ji j = i = n +1 i ij ⇒ π n Pn ,n +1 = π n +1Pn +1,n Time-Reversed Markov Chains (Revisited) 6-9 „ Theorem 2: Irreducible Markov chain with transition probabilities Pij If there exist: „ A set of transition probabilities Qij, with ∑j Qij=1, i ≥ 0, and „ A set of positive numbers {πj}, that sum up to 1, such that πi Pij = π j Q ji , „ „ i, j = 0,1, (1) Then: „ Qij are the transition probabilities of the reversed chain, and „ {πj} is the stationary distribution of the forward and the reversed chains Remark: Use to find the stationary distribution, by guessing the transition probabilities of the reversed chain – even if the process is not reversible 6-10 Continuous-Time Markov Chains {X(t): -∞< t 0) if and only if: „ p j ∑ i ≠ j q ji = ∑ i ≠ j pi qij , „ j = 0,1, Process in steady state – e.g., started at t =-∞: Pr{ X (t ) = j} = p j = lim Pr{ X (t ) = j | X (0) = i} t →∞ „ If {πj}, is the stationary distribution of the embedded discrete-time chain: pj = π j /ν j ∑ i π i /ν i , ν j ≡ ∑ i ≠ j q ji , j = 0,1, 6-19 Example: M/M/2 Queue with Heterogeneous Servers αλ (1 − α )λ „ „ „ λ 1A µA µB µB µA 1B λ µ A + µB µ A + µB λ M/M/2 queue Servers A and B with service rates µA and µB respectively When the system empty, arrivals go to A with probability α and to B with probability 1-α Otherwise, the head of the queue takes the first free server Need to keep track of which server is busy when there is customer in the system Denote the two possible states by: 1A and 1B Reversibility: we only need to check the loop 0→1A→2→1B→0: q0,1 A q1 A,2 q2,1B q1B ,0 = αλ ⋅ λ ⋅ µ A ⋅ µ B „ λ q0,1B q1B ,2 q2,1 A q1 A,0 = (1 − α )λ ⋅ λ ⋅ µ B ⋅ µ A Reversible if and only if α=1/2 What happens when µA=µB, and α≠1/2? 6-20 Example: M/M/2 Queue with Heterogeneous Servers αλ S3 λ 1A µA µB µB µA (1 − α )λ 1B λ µ A + µB µ A + µB λ S2 S1  λ  pn = p2    µ A + µB  λ n−2 , n = 2,3, λ p0 = µ A p1 A + µ B p1B   ( µ A + µ B ) p2 = λ ( p1 A + p1B )  ⇒ ( µ A + λ ) p1 A = αλ p0 + µ B p2  λ λ + α (µ A + µB ) µ A 2λ + µ A + µ B λ λ + (1 − α )( µ A + µ B ) p1B = p0 µB 2λ + µ A + µ B p1 A = p0 p2 = p0 λ λ + (1 − α )µ A + αµ B µ AµB 2λ + µ A + µ B  λ λ λ + (1 − α ) µ A + αµ B  p0 + p1 A + p1B + ∑ n =2 pn = ⇒ p0 = 1 +  2λ + µ A + µB  µ A + µB − λ µ AµB  ∞ −1 ... 6- 2 Topics Time-Reversal of Markov Chains „ Reversibility „ Truncating a Reversible Markov Chain „ Burke’s Theorem „ Queues in Tandem „ 6- 3 Time-Reversed Markov Chains... πi = π j ∑ i =0 Pji = π j ∞ Reversibility 6- 6 Stochastic process {X(t)} is called reversible if „ (X(t1), X(t2),…, X(tn)) and (X(τ-t1), X(τ-t2),…, X(τ-tn)) „ have the same probability distribution,... Example: Discrete-time birth-death processes are reversible, since they satisfy the DBE 6- 8 Example: Birth-Death Process Pn −1,n P01 Sc Pn ,n +1 n Pn ,n −1 P10 P00 S Pn ,n n+1 Pn +1,n One-dimensional