MULTIPLE KNAPSACK PROBLEM WITH INTER-RELATED ITEMS AND ITS APPLICATIONS TO REAL WORLD PROBLEMS ANG JUAY CHIN (B.Sc (Computer and Information Sciences), NUS) A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF COMPUTER SCIENCE NATIONAL UNIVERSITY OF SINGAPORE 2003 Abstract Motivated by a real world application, we study a variant of the classic knapsack problem, which we call the Multiple Knapsack Problem with Inter-related Items (MKPIR) We are given a set of items and a set of knapsacks of limited capacity For each item, a set of knapsacks that can hold that item is specified In addition, binary relationships may exist between the items These relationships affect the profit of having that item in the knapsack in consideration In this thesis, we adopted a few heuristics and test them on the venue assignment problem, instance of MKPIR, using actual data and randomly generated test instances Keywords: Multiple knapsack problem, Venue assignment problem Acknowledgements I would like to express my sincere gratitude to my supervisor A/P Andrew Lim for his sound advice through the course of my study I would also like to thank my cosupervisor Dr Tan Sun Teck for his guidance Lastly, I would like to express my heart felt love and appreciation to my family especially my husband Wee Kit for their support, patience and love during this period i Table of Contents Chapter Introduction 1.1 Motivation 1.2 Examination Timetabling Problem 1.3 Venue Assignment Problem 1.4 Thesis Outline Chapter Multiple Knapsack Problem 2.1 Multiple Knapsack Problem with Inter-related Items 2.2 Formulation of VAP as MIN_MKPIR 10 2.3 Related Work 13 Chapter System User Interface 14 3.1 Exam Scheduler 14 3.2 Venue 16 3.3 Venue Assignment 17 3.4 Seat Assignment 18 Chapter Heuristics 19 4.1 Greedy Method 20 4.2 Tabu Search 20 4.3 Simulated Annealing 22 4.4 Operators 26 ii 4.4.1 Relocate Operator 26 4.4.2 Exchange Operator 27 4.4.3 Multi-Exchange Operator 29 4.5 “Squeaky Wheel” Optimization 30 4.6 Combining “Squeaky Wheel” Optimization and Tabu Search 34 Chapter Experimental Results 36 5.1 Experimental Data 36 5.2 Evaluation of the Assignment Quality 42 5.3 Implementation Matters 45 5.4 Results 46 Chapter Conclusions 60 6.1 Project Summary 60 6.2 Directions for Future Work 60 Bibliography 63 iii List of Figures Figure 1: Venue Assignment Problem as an instance of a MKPIR 12 Figure 2: Exam Scheduler 14 Figure 3: Edit Venue Layout 16 Figure 4: Venue Assignment 17 Figure 5: Seat Assignment 18 Figure 6: Tabu Search 21 Figure 7: Simulated Annealing 25 Figure 8: Relocate Operator 26 Figure 9: Exchange Operator 28 Figure 10: The Construct/Analyze/Prioritize cycle 30 Figure 11: Relationship Cost for 0102_sem2 50 Figure 12: Relationship Cost for 0203_sem1 50 Figure 13: Relationship Cost for 0203_sem2 51 Figure 14: Relationship Cost for many relationships test instances 55 Figure 15: Relationship Cost for many items, few knapsacks test instances 57 Figure 16: Relationship Cost for few items, many knapsacks test instances 57 iv List of Tables Table 1: Characteristics of 0102_sem2 37 Table 2: Characteristics of 0203_sem1 37 Table 3: Characteristics of 0203_sem2 37 Table 4: List of Examination Venues for 0102_sem2 38 Table 5: List of Examination Venues for 0203_sem1 39 Table 6: List of Examination Venues for 0203_sem2 40 Table 7: Generated Problem Instances 42 Table 8: Results for 0102_sem2 47 Table 9: Results for 0203_sem1 48 Table 10: Results for 0203_sem2 49 Table 11: Computational Time for 0102_sem2 52 Table 12: Computational Time for 0203_sem1 52 Table 13: Computational Time for 0203_sem2 52 Table 14: Results for many relationships test instances 53 Table 15: Results for many items, few knapsacks test instances 54 Table 16: Results for few items, many knapsacks test instances 54 Table 17: Computational Time for many relationships test instances 57 Table 18: Computational Time for many items, few knapsacks test instances 58 Table 19: Computational Time for few items, many knapsacks test instances 58 v Summary In the Multiple Knapsack Problem, N items of different sizes have to be packed into M knapsacks with limited volume Each item i has an associated profit p i and weight wi The problem is to select N disjoint subsets of items, such that subset i fits into knapsack i and total profit of the selected items is maximized Motivated by a real world application, we study a variant of MKP, the Multiple Knapsack Problem with Inter-related Items (MKPIR) In MKPIR, binary relationships may exist between items Consider two items, i and j, packed in knapsacks x and y respectively If there exists a relationship rij between the two items, then the value of the relationship between i and j is rij ⋅ k xy where k xy is the distance measure between knapsacks x and y Accordingly, item i has profit pix = ∑ j =1 rij k xy N when considered for inclusion in knapsack x Thus, unlike most other variants of MKP in which the profit/cost of item i is fixed, p i , the value of item i in MKPIR is a variable In addition, an assignment restriction is imposed on the items For each item i , a set of knapsacks that can hold item i is specified In a feasible assignment of items to knapsacks, each item is assigned to at most one knapsack, assignment restrictions are satisfied, and knapsack capacities are not exceeded, with the objective of M maximizing N ∑∑ p x =1 i =1 ix z ix , where zix = if item i is assigned to knapsack x and zix = otherwise The Venue Assignment Problem (VAP) is a sub-problem in an exam-timetable scheduling application In this application, examinations for modules are assigned exam timeslots based on the estimated enrolment figures, disregarding the actual vi examination venue Specific venues are only allocated when the actual enrolment figures for modules are known after student registration Each day of the examination period is divided into a few non-overlapping exam sessions Venues may be physically far apart, and some candidates may be required to take more than one exam a day and have to travel between venues Thus, in addition to assigning examinations to venues without violating the venue capacity constraint, we would like to minimize the distance that candidates have to travel between their examinations each day The problem of assigning exams to venues so as to minimize student movement can be formulated as an instance of MIN_MKPIR (a minimization version of MKPIR) The assignment of exams to each exam timeslot is restricted to only venues available during that timeslot The distance that needs to be traveled by candidates of exam i to his next exam j on the same day will be translated into cost for both exam i and exam j Each exam day may be viewed as a separate VAP, since exams scheduled on different days are independent Various heuristics such as Tabu Search, Simulated Annealing and “Squeaky Wheel Optimization” are experimented on the MKPIR using actual data from the National University of Singapore and generated test data Preliminary results based on our experimentations shows that Simulated Annealing and “Squeaky Wheel Optimization” with Tabu Search post-optimization produce satisfactory results Although more work still need to be done on solving the MKPIR efficiently, we believe that our heuristics will be useful for solving VAP and other real world problems of similar characteristics vii Chapter Introduction 1.1 Motivation Our variant of the multiple knapsack problem, multiple knapsack problem with inter-related items was motivated by a sub-problem in the timetabling problem in the National University of Singapore (NUS) NUS conducts more than 1500 modules for about 25 000 students every semester In 1993, a modular course system was adopted by NUS Under this modular system, students progress at their own pace and choose the modules that they wished to study, subject to timetable arrangements, in order to complete their degree requirements In addition, students can choose from a wide range of modules, called cross-faculty modules, offered by different faculties As a result of this flexibility for students, the task of scheduling examination timetables in NUS became much more complex Previously, the scheduling of examination timetable in NUS was done manually This was a tedious and error-prone process that typically took months to process Critical conflicts where a student was scheduled to take examinations at the same time were not always detected, and correcting these mistakes was cumbersome and expensive A team was thus funded by NUS to develop an automated campuswide examination timetabling system, UTTSExam [Lim et al 2000] CHAPTER EXPERIMENTAL RESULTS 400 Relationship Cost 350 300 Greedy 250 Tabu Search 200 SA 150 SWO SWO+Tabu Search 100 50 10 11 12 Day Figure 11: Relationship Cost for 0102_sem2 160 Relationship Cost 140 120 Greedy Tabu Search SA SWO 100 80 60 SWO+Tabu Search 40 20 10 11 12 Day Figure 12: Relationship Cost for 0203_sem1 50 CHAPTER EXPERIMENTAL RESULTS 250 Relationship Cost 200 Greedy 150 Tabu Search SA SWO 100 SWO+Tabu Search 50 10 11 12 Day Figure 13: Relationship Cost for 0203_sem2 that could be held in one single venue is not great for all three actual data sets As the ratio between the capacity made available and capacity required is rather high, this makes it relatively easy for the heuristics to fit in most of the average size examinations An examination whose size is greater than the largest available venues remains unassigned since these examinations are only assigned to multiple venues after the termination of our heuristics and is not reflected in the results Computational time taken by the various heuristics for each of the actual data sets is reflected in Table 11 to Table 13 For all three data sets, though the greedy method requires the least amount of time, it clearly performs badly when its results is compared with that that is found by the other heuristics 51 CHAPTER EXPERIMENTAL RESULTS Day 10 11 12 Greedy 1 1 1 1 1 1 Tabu Search 51 34 38 36 31 55 52 45 33 25 17 15 SA 61 29 37 29 27 39 43 47 32 19 19 15 SWO 13 10 14 12 10 20 17 14 11 4 SWO + Tabu Search 23 19 22 20 17 31 27 22 19 17 10 Table 11: Computational Time for 0102_sem2 Day 10 11 12 Greedy 1 1 1 1 1 1 Tabu Search 141 68 97 281 21 24 91 122 112 104 15 17 SA 74 29 40 92 14 40 82 48 18 SWO 7 10 10 10 SWO + Tabu Search 24 16 18 23 19 22 27 22 7 Table 12: Computational Time for 0203_sem1 Day 10 11 12 Greedy 1 1 1 1 1 1 Tabu Search 336 549 484 808 121 141 607 490 510 148 129 291 SA 104 134 196 163 57 64 274 232 232 50 45 27 SWO 11 12 15 19 15 16 15 10 11 SWO + Tabu Search 74 70 76 72 30 31 69 82 58 28 30 54 Table 13: Computational Time for 0203_sem2 52 CHAPTER EXPERIMENTAL RESULTS Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 #not assign 224 62 70 55 32 32 11 0 Greedy Size not assign 14 3 0 Relation cost 13566 20978 20599 19903 19034 18553 20615 19722 18998 19097 #not assign 198 26 16 0 0 0 Tabu Search Size not Relation assign cost 12 15005 22217 19584 18624 16920 16907 15160 15287 12965 12128 #not assign 198 10 70 0 0 0 SA Size not assign 12 0 0 0 Relation cost 15005 23060 20599 17598 16561 14564 13298 14445 12369 12491 #not assign 220 50 11 13 0 0 SWO Size not assign 14 1 0 0 Relation cost 12043 16661 22976 23046 18736 17112 20438 17806 18245 18843 SWO + Tabu Search #not Size not Relation assign assign cost 84 20346 0 19890 0 19281 0 18930 0 16218 0 16443 0 16866 0 14925 0 13479 0 12046 Table 14: Results for many relationships test instances Table 14 to Table 16 show the results achieved for the generated problem instances using the heuristics The relationship cost incurred by the various heuristics is shown graphically in Figure 14 to Figure 16 First we observe how the difference in admissibility density of knapsacks affects the performance of the various heuristics We see that when the admissibility density of knapsacks is low, the number of items not assigned is high This is reasonable since each item is only allowed to go into few choices of knapsacks, and thus there will be a higher number of unassigned items Generally, as the admissibility density increases, 53 CHAPTER EXPERIMENTAL RESULTS Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Size not assign 1679 1142 777 389 238 159 51 40 0 Greedy #not assign 176 169 114 61 50 31 0 Relation cost 58723 61157 61137 63295 65716 66366 64731 62563 60705 60861 Tabu Search Size not #not Relation assign assign cost 157 33 63045 0 52913 0 44524 0 40365 0 34290 0 30795 0 29730 0 26438 0 25488 0 23513 Size not assign 0 0 0 0 0 SA #not assign 0 0 0 0 0 Relation cost 64671 52347 43880 39033 34606 30650 28521 26677 25862 22128 Size not assign 1043 388 103 30 0 0 0 SWO #not assign 106 37 24 0 0 0 Relation cost 56074 56577 53008 50725 45610 41086 38608 34750 32773 32190 SWO + Tabu Search Size not #not Relation assign assign cost 0 61989 0 52874 47070 0 38526 0 36878 0 33249 0 32409 0 28642 0 27304 0 24741 Relation cost 27065 22910 21066 19986 20330 19827 19068 17357 18945 17824 SWO + Tabu Search Size not #not Relation assign assign cost 293 27566 293 22608 293 20744 293 19495 0 19520 0 19136 0 18798 0 17204 0 18468 0 16370 Table 15: Results for many items, few knapsacks test instances Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Size not assign 548 347 293 293 0 0 0 Greedy #not assign 1 0 0 0 Relation cost 30067 33927 38778 37111 37130 36690 39192 37714 35953 36109 Tabu Search Size not #not Relation assign assign cost 494 29633 293 29362 293 27552 293 24277 0 24451 0 24701 0 21211 0 21106 0 20674 0 19589 Size not assign 494 293 293 293 0 0 0 SA #not assign 1 0 0 0 Relation cost 29088 25754 22068 22207 20468 18900 18983 18342 16647 17096 Size not assign 470 293 293 293 0 0 0 SWO #not assign 1 0 0 0 Table 16: Results for few items, many knapsacks test instances 54 CHAPTER EXPERIMENTAL RESULTS 24000 22000 Relationship Cost 20000 Greedy Tabu Search 18000 SA SWO 16000 SWO+Tabu Search 14000 12000 10000 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Admissibility Density of Knapsacks Figure 14: Relationship Cost for many relationships test instances most, if not all, items are able to pack into the available knapsacks Hence, the heuristics mainly focus on reducing the relationship cost as the admissibility density gets higher We also see the general trend that as the admissibility density figure gets higher; the relationship cost tends to be lower Though in some cases, the relationship cost appear to be quite low when the admissibility density of knapsacks is very low That’s because the restrictive choice of knapsacks that an item may be assigned to has caused the number of items that could not be assigned a knapsack to be high These items that are not assigned a knapsack not incur relationship cost As the choices of knapsacks that are made available to the items increases, the chance of being assigned to a knapsack increases These items when assigned to a knapsack may result in a 55 CHAPTER EXPERIMENTAL RESULTS 70000 65000 Relationship Cost 60000 55000 Greedy 50000 Tabu Search SA 45000 SWO 40000 SWO+Tabu Search 35000 30000 25000 20000 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Admissibility Density of Knapsacks Figure 15 : Relationship Cost for many items, few knapsacks test instances relationship cost if it is assigned to a different knapsack from those items it has a relationship with This leads to an increase in the overall relationship cost When there are more choices for the items to be assigned to, each item can basically be assigned to any knapsack as long as it is within the capacity limit As a result, the chance of matching an item to a less costly knapsack gets higher, causing the overall relationship cost to be lowered However because of this added flexibility, the solution search space of the problem increases This in turn increases the computational time required by the search heuristics during their search process as can be seen in Table 17 to Table 19 56 CHAPTER EXPERIMENTAL RESULTS 40000 Relationship Cost 35000 Greedy 30000 Tabu Search SA SWO 25000 SWO+Tabu Search 20000 15000 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Admissibility Density of Knapsacks Figure 16: Relationship Cost for few items, many knapsacks test instances Admissibility Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Greedy 1 1 1 1 1 Tabu Search 14 17 19 20 24 21 21 23 25 25 SA 45 22 14 76 64 146 85 74 73 60 SWO 23 25 28 31 35 42 43 47 51 50 SWO + Tabu Search 28 31 35 38 43 49 52 54 60 58 Table 17: Computational Time for many relationships test instances 57 CHAPTER EXPERIMENTAL RESULTS Admissibility Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Greedy 1 1 1 1 1 Tabu Search 1091 1232 1358 1368 1353 1354 1564 1554 1423 1577 SA 599 617 618 588 1086 881 680 1096 1370 915 SWO 145 173 201 225 245 250 266 286 302 311 SWO + Tabu Search 694 749 737 1009 990 943 1091 1217 1066 1276 Table 18: Computational Time for many items, few knapsacks test instances Admissibility Density 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Greedy 1 1 1 1 1 Tabu Search 47 56 52 48 58 56 53 56 51 68 SA 76 68 63 64 72 79 94 71 84 57 SWO 11 15 17 21 24 29 32 34 38 SWO + Tabu Search 15 18 22 24 28 31 36 38 42 45 Table 19: Computational Time for few items, many knapsacks test instances For the many items, few knapsacks test instance, we noticed that Tabu Search actually requires substantially more computational time than the rest of the heuristics This is mainly due to our implementation of Tabu Search in which the tabu operators will find all possible moves before selecting the best move to be applied in every iteration Hence, in this test instance where there are many items, the large number of possible moves that need to be computed slows down the search process of Tabu Search considerably Similarly, the computational time required by the hybrid approach also increases for this test instance since Tabu Search is being used as a postoptimization process 58 CHAPTER EXPERIMENTAL RESULTS Taking all the results into account, it appears that the hybrid approach performs relatively well in the sense that it is able to efficiently reduce the number and size of items not assigned when the admissibility density of knapsack is low However as the admissibility density increase, SA appears to perform better although Tabu Search and the hybrid approach appear not to perform too badly either This may suggest that SA may be good in reducing the relationship cost One problem with Tabu Search when applied on MKPIR is that it may take a long time when the number of items is huge 59 Chapter Conclusions 6.1 Project Summary In this thesis, we have introduced the multiple knapsack problem with inter- related items, a variant of the multiple knapsack problem that allows assignment restriction and where relationship which exists between items affects the profit of an item in a knapsack We have also formulated the Venue Assignment Problem, a subproblem of the examination timetabling problem in NUS as an instance of MKPIR We have performed a few heuristics on the MKPIR using both actual data from the National University of Singapore and generated test instances and presented our results Our experimentation shows that the combination of SWO with TS produces promising results in solving MKPIR To the best of our knowledge, no similar experiment has been conducted on MKPIR or VAP As can be seen from the example of the venue assignment problem, MKPIR models allocation and packing problems with interrelated items, which is highly relevant in real world applications We believe that an algorithm that provides an efficient assignment to VAP and the MKPIR will bring great benefits to many other real world problems with similar characteristics 6.2 Directions for Future Work In MKP, often, even if the total available size of the knapsacks is greater than the total size of items, it may not be possible to pack all items into the knapsacks Due 60 to the nature of the venue assignment problem, when an examination could not be wholly assigned to a venue, it should be allowed to be assigned to multiple venues This makes the problem to be similar to fractional multiple knapsack problem However in the VAP, practically the number of examinations that need to be held in multiple venues should be minimal In addition, the number of venues in which an examination is being split to should be kept to the minimum Besides causing an increase in manpower, an examination being conducted in multiple venues often leads to confusion for candidates Candidates may turn up at the wrong exam venue This differentiates our problem from the classical fractional knapsack problem where the number of splits is not a concern For all of our heuristics, we however not attempt to minimize the number of items that need to be held in multiple knapsacks nor the number of knapsack that the item is being split to within the algorithm Instead, these items are split to knapsacks that are nearby and that have available capacity after the algorithm terminates This however does not guarantee that splits are minimized Allowing an item to be assigned to multiple knapsacks together with the mentioned considerations would add another dimension of difficulty to MKPIR Which items should be split? How many parts should the item be split into? How big should each part be? Future efforts could possibly take into consideration the minimizing of splitting an item to multiple knapsacks within the heuristics One possible approach is dynamic programming Our implementation of SWO uses a relatively simple blaming system based on the objectives we seek When the results of SWO is post-optimized using Tabu Search, 61 CHAPTER CONCLUSIONS the final results obtained is encouraging In future works, a more complex blaming system could possibly be employed Combining the action of SWO with various heuristics that complements the pitfall of each other would be interesting 62 Bibliography Dawande M., P Keskinocak, R Ravi and F.S Salman, Approximation Algorithms for the Multiple Knapsack Problem with Assignment Restrictions, Journal of Combinatorial Optimization, Vol 4, 2000, pg 171-186 Dawande M and J Kalagnanam, The Multiple Knapsack Problem with Color Constraints, IBM Research Report, 1998 Garey M R and D S Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, Freeman, San Francisco, 1979 Glover F., Future Paths for Integer Programming and Links to Artificial Intelligence, Computer & Operational Research, Vol 5, 1986, pg 533-549 Glover F and M Laguna, Tabu Search, In Reeves (Reeves 1993), chapter 3, 1993 Hung M S and J C Fisk, An Algorithm for 0-1 Multiple Knapsack Problems, Naval Research Logistics Quarterly, Vol 24, pg 571-579, 1978 Joslin D and D Clement, “Squeaky Wheel” Optimization, In Proceedings of AAAI 1998, pg 340-346 Joslin D and D Clement, “Squeaky Wheel” Optimization, Journal of Artificial Intelligence Research, Vol 10, 1999, pg 353-373 Kirkpatrick S., C.D Gelatt and M.P Vechhi, Optimization by Simulated Annealing, Science, Vol 220 1983, pg 671-680 63 Lim A., J.C Ang, W.K Ho and W.C Oon, A Campus-Wide University Examination Timetabling Application, Innovative Applications in Artificial Intelligence (AAAI/IAAI) 2000, pg 1020-1025 Lim A., J.C Ang, W.K Ho and W.C Oon, UTTSExam: A Campus-Wide University Examination Timetabling System, Innovative Applications in Artificial Intelligence (AAAI/IAAI) 2002, pg 838-844 Martello S and P Toth, Knapsack Problems: Algorithms and Computer Implementations, 1990, Wiley, Chichester, England Pisinger D and P Toth, Handbook of Combinatorial Optimization – Knapsack Problems, Vol 1, pg 299 – 428, 1998, Kluwer Academic Publishers Rayward-Smith V J., I.H Osman, C.R Reeves and G.G Smith, Modern Heuristic Search Methods, Wiley, New York, 1996 Reeves C.R., Modern Heuristic Techniques for Combinatorial Problems, Blackwell Scientific Publications, Oxford, 1993 Shachnai H and T Tamir, On Two Class-Constrained Versions of the Multiple Knapsack Problem, Algorithmica, Vol 29, 2001, pg 442-467 Yang M., An Efficient Algorithm to Allocate Shelf Space, European Journal of Operational Research, Vol 131, 2001, pg 107-118 64 ... MULTIPLE KNAPSACK PROBLEM 2.1 Multiple Knapsack Problem with Inter- related Items In Multiple Knapsack Problems with Inter- related Items (MKPIR), we are given a set of N items to be packed into a... VAP and other real world problems of similar characteristics vii Chapter Introduction 1.1 Motivation Our variant of the multiple knapsack problem, multiple knapsack problem with inter- related items. .. by a real world application, we study a variant of the classic knapsack problem, which we call the Multiple Knapsack Problem with Inter- related Items (MKPIR) We are given a set of items and a