ECE 307 – Techniques for Engineering Decisions Dynamic Programming George Gross Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved DYNAMIC PROGRAMMING ‰ Systematic approach to solving sequential decision making problems ‰ Salient problem characteristic: ability to separate the problem into stages ‰ Multi-stage problem solving technique © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved STAGES AND STATES ‰ We consider the problem to be composed of multiple stages ‰ A stage is the “point” in time, space, geographic location or structural element at which we make a decision; this “point” is associated with one or more states ‰ A state of the system describes a possible configuration of the system in a given stage © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved STAGES AND STATES dn sn state (input) stage n decision variable (decision) s n state (output) © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved RETURN FUNCTION ‰ A decision d n in the stage n transforms the state s n in the stage n into the state s n + in the stage n + ‰ The state s n and the decision d n have an impact on the objective function; the effect is measured in terms of the return function denoted by rn (s n , d n ) ‰ The optimal decision at stage n is the decision d *n that optimizes the return function for the state s n © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved RETURN FUNCTION decision variable (decision) sn state (input) dn state (output) stage n s n rn ( s n , d n ) return function © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved ROAD TRIP EXAMPLE ‰ A poor student is traveling from NY to LA ‰ To minimize costs, the student plans to sleep at friends’ houses each night in cities along the trip ‰ Based on past experience he can reach  Columbus, Nashville or Louisville after day  Kansas City, Omaha or Dallas after days  San Antonio or Denver after days  LA after days © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved ROAD TRIP EXAMPLE Columbus 680 K City 610 580 550 Denver 540 1030 790 790 NY 900 Nashville 760 Omaha 10 LA 790 700 770 Louisville day 1050 660 510 day 830 Dallas day 940 270 1390 S Antonio day © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved day ROAD TRIP ‰ The student wishes to minimize the number of miles driven and so he wishes to determine the shortest path from NY to LA ‰ To solve the problem, he works backwards ‰ We adopt the following notation c ij = distance between states i and j f k( i ) = distance of the shortest path to LA from state i in the stage k © 2006 – 2009 George Gross, University of Illinois at Urbana-Champaign, All Rights Reserved ROAD TRIP EXAMPLE CALCULATIONS day : day : f (8) = 1,030 f (9) = 1,390 ⎧ ⎫ ⎪ ⎪ f (5) = ⎨(610 + 1,030),(790 + 1,390)⎬ = 1,640   ⎪⎩ ⎪⎭ 2,220 2,330 2,320 ⎧ ⎫ ⎪ ⎪ f (4) = ⎨(510 + 1,640) , (700 + 1,570) , ( 830 + 1,660 ) ⎬ = 2,150