348 Cheng A modi®ed version of incidence matrix A is formed as P À1 0 1 0 T 0 À1 0 1 T  à T 1 0 À1 0 AXI ˆT T 0 1 0 À1 T R À1 À1 0 0 0 0 1 1 Figure 6 A Petri net model 3 4 Check the rank of A: r = Partition A as A11= 1, (À10†T; andA22 ˆ …10†T: Use Eq (16) to determine    1 1 Bp ˆ À À1; 0 0 rank(A) = 1 A12 = À1, A21 = Bp ! 0 1 ˆ 1 0 1 0 0 1 ! Check Bp Ám, where Ám ˆ mf À m0 ˆ …3 À3 0†T P Q ! 3 ! 1 1 0 R Sˆ 0 Bp ˆ À3 0 0 1 0 0 According to Theorem 1, mf can be reachable from m0 The solution to Eq (9) may become more complicated when one includes the constraint that the elements of x must be nonnegative integers In this case, an ef®cient algorithm [9] for computing the invariant of Petri net is proposed The basic idea of the method can be illustrated by getting the minimal set of P-invariants through the following example Example 7 The incidence matrix of the Petri net in Fig 6 is P À1 0 T 0 À1 T T 1 0 AˆT T 0 1 T R À1 À1 0 0 1 0 0 1 À1 0 0 À1 0 0 1 1 Copyright © 2000 Marcel Dekker, Inc Q 0 0 U U 0 U U 0 U U 1 S À1 j j j j j j 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 Q 0 0U U 0U U 0U U 0S 1 We will add rows to eliminate a nonzero element in each row of A Speci®cally, (1) adding the third row to the ®rst row; (2) the fourth row to the second row; (3) adding the fourth, ®fth, and sixth rows to the third row Determine if mf = (3 0 1)T is reachable from initial marking m0 =(0 3 1)T Solution: 1 2 0 0 0 0 1 À1 P 0 T 0 T T 1 T T 0 T R À1 0 0 1 0 À1 0 1 0 0 0 0 1 À1 j j j j j j 1 0 0 0 0 0 0 0 0 0 T 0 0 0 0 T T 1 0 À1 0 T T 0 1 0 À1 T R À1 À1 0 0 0 0 1 1 P 0 0 0 0 T 0 0 0 0 T T 0 0 0 0 T T 0 1 0 À1 T R À1 À1 0 0 0 0 1 1 0 0 0 0 1 À1 j j j j j j 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 P 0 À1 0 1 À1 0 0 0 À1 0 0 1 0 0 0 0 1 À1 j j j j j j 1 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 0 0 0 1 0 Q 0 0U U 0U UA 0U U 0S 1 Q 0 0 0 0U U 0 0U UA 0 0U U 1 0S 0 1 Q 0 0 0 0U U 1 1U U 0 0U U 1 0S 0 1 The three P-invariants are x1 = (1 0 1 0 0 0)T, x2 = (0 1 0 1 0 0)T, and x3 = (0 0 1 1 1 1)T They are actually the ®rst three rows of the modi®ed identity matrix This is because their associated three rows in the ®nal version of modi®ed A have all zero elements 4.7.3.4 Invariant Analysis of a Pure Petri Net The following example illustrates how Petri net invariants can aid the analysis of a pure Petri net Example 8 Consider the Petri net in Fig 6 as a model that describes two concurrent processes (i.e., process 1 and process 2), each of which needs a dedicated Petri Nets resource to do the operations in the process The tokens at p1 and p2, models the initial availability of each dedicated resource Both processes also share a common resource to aid their own process This shared resource is modeled as a token initially at p5 The initial marking of the model is m0= (2 1 0 0 3 0)T Applying three P-invariant results obtained from Example 7 to Eq (10), then m(p1) + m(p3) = 2 m(p2) + m(p4) = 1 m(p3) + m(p4) + m(p5) + m(p6) = 3 The ®rst equation implies that the total number of resources for process 1 is 2 The second equation implies that the total number of resources for process 2 is 1 The last equation implies that the three shared resources are in a mutual exclusion situation serving for either process 1 or process 2 The invariant analysis of a pure Petri net also includes that: 1 2 3 4 4.8 A structural bounded Petri net must have an …n Â1† vector x of positive integer such that xT A 0 A conservative Petri net must have an …n  1† vector x of positive integers such that xTA = 0 A repetitive Petri net must have an …m  1† vector y of positive integers such that Ay ! 0 It is partially repetitive if y contains some zero elements A consistent Petri net must have an …m  1† vector y of positive integers such that Ay = 0 It is partially consistent if y contains some zero elements TIMED PETRI NET MODELS FOR PERFORMANCE ANALYSIS Timed Petri net (TPN) models have been developed for studying the temporal relationships and constraints of a DES They also form the basis of system performance analysis that includes the calculation of process cycles, resource utilization, operation throughput rate, and others There are two approaches for modeling the time information associated with the occurrence of events in a Petri net A timed place Petri net (TPPN) associates time information with places A timed tran- Copyright © 2000 Marcel Dekker, Inc 349 sition Petri net (TTPN) associates time information with transitions Timed Petri nets can be further classi®ed according to the time parameters assigned in the net If all time parameters in a net are deterministic, the net is called deterministic timed Petri net (DTPN) If all time parameters in a net are exponentially distributed random variables, the net is called a stochastic timed Petri net (STPN or SPN) 4.8.1 4.8.1.1 Deterministic Timed Petri Nets TPPN Approach In a TPPN, every place is assigned a deterministic time parameter that indicates how long an entered token remains unavailable in that place before it can enable a transition It is possible that during the unavailable period of a token in a place another token may arrive in the place Only available tokens in a marking can enable a transition The ®ring of a transition is carried out with zero time delay as it is in ordinary Petri nets The matrix-based methods can be used for the analysis of a TPPN From Eq (7), the marking at instant time t can be expressed as m…t† ˆ m…t0 † ‡ Au…t† where: m(t0) is the initial marking at instant t0 (t0 < t) u(t) represents the ®ring vector at instant t Considering Át ˆ t À t0 Tˆ 0, m…t† À m…t0 † ˆ Au…t†, then Ám…t†=Át ˆ Au…t†=Át ˆ Af where: and Ám…t† ˆ …17† Ám…t†=Át represents the average change in tokens over period Át f represents the average ®ring frequency of the transitions, called current vector A direct application of Eq (17) is to study the periodic behavior of a consistent Petri net According to De®nitions 19 and 20, a consistent Petri net has some ®ring sequence that returns a marking m back to itself In the case of a consistent Petri net modeled as a TPPN, it has m…t† À m…t0 † ˆ 0 Substituting this fact into Eq (17), then Af ˆ 0; f > 0 …18† Actually, solving Eq (18) for a TPPN is equivalent to determining the T-invariants with the exception of the time scaling factor The performance of the model such as throughput and resource utilization can be deter- 350 Cheng mined using the current vectors that are associated with the places representing the resource Through P-invariants a relationship can also be established, relating the initial marking m0, the deterministic time delays in timed places, and the ®ring frequencies of the transitions [2]: xT m0 ˆ xT DA‡ f …19† where: x is a P-invariant D contains the deterministic time delays for timed places, and D = diag{di} for i = 1, 2, , n A+ is the output part of the incidence matrix A In Eq (19), A+f indicates the average frequency of token arrivals and DA+f indicates the average number of tokens due to the delay restrictions If Eqs (18) and (19) are satis®ed for all P-invariants, the TPPN model functions at its the maximum rate The applications of this approach can be found in Sifakis [10] Example 9 Figure 6 shows a TPPN model, where T T T T ‡ DA f ˆ T T T T R QP 0 UT 0 UT UT UT 1 UT UT 0 UT UT SR 0 d2 d3 d4 d5 d6 d2 f4 d3 f3 d4 f2 0 d5 f5 QP Q 0 1 0 0 f1 0 0 1 0 U T f2 U UT U UT U 0 0 0 0 U T f3 U UT U 1 0 0 0 U T f4 U UT U UT U 0 0 0 1 S R f5 S 0 1 1 0 d6 …f4 ‡ f3 † f6 ÃT Given the initial marking m0 as [K1 K2 0 0 K5 0]T, where: K1 represents resource) at K2 represents resource) at K5 represents resource) at For a maximum rate, the minimum number of shared resource (K5) required in the model must be 10 4.8.1.2 TTPN Approach In a TTPN framework [11], deterministic time parameters are assigned to transitions A timed transition is enabled by removing appropriate number of tokens from each input place The enabled transition ®res after a speci®ed amount of time by releasing the tokens to its output places A direct application of TTPN is the computation of cycle time in a marked graph [2] De®nition 36 The cycle time Ci of transition ti is de®ned as ni 3I d1  ˆ d1 f1 f1 ˆ 1=2; f2 ˆ 1=4 K5 ˆ …6 ‡ 4†…f2 ‡ f1 † ‡ 2f1 ‡ 3f2 ˆ 9:25 Ci ˆ lim Si …ni †=ni ; D ˆ f2; 1; 2; 3; 6; 4g P These three equations provide a relation that relates the initial markings, the time delays, and the ®ring frequencies Let K1 = 2, K2 = 1, then the number of tokens (dedicated p 1 the number of tokens (dedicated p 2 the number of tokens (shared p 5 Applying the P-invariants results obtained in Example 7 to Eq (19), then K1 ˆ …d1 ‡ d3 †f1 K2 ˆ …d2 ‡ d4 †f2 K5 ˆ …d5 ‡ d6 †…f2 ‡ f1 † ‡ d3 f1 ‡ d4 f2 Copyright © 2000 Marcel Dekker, Inc …20† where Si …ni † is the time at which transition ti initiates its ni th execution Theorem 4 A marked graph has the same cycle time for all transitions Theorem 5 The minimum cycle time (maximum performance) Cm of a marked graph is Cm ˆ maxfTk =Kk g for k ˆ 1; :::; c k …21† where: Tk is the sum of transition delays in a circuit k Kk is the sum of the tokens in a circuit k c is the number of circuits in the model The processing rate (or throughput) can be easily determined from the cycle time by  ˆ 1=Cm ˆ minfKk =Tk ; k ˆ 1; 2; :::; cg …22† Petri Nets 351 Example 10 Let us determine the minimum cycle time for the marked graph in Fig 4, where {di}= {5, 20, 4, 3, 6} Using the elementary circuit results in Example 4 and apply Eq (21) to each elementary circuit, then Circuit Circuit Circuit Circuit 1: 2: 3: 4: T1/K1 T2/K2 T3/K3 T4/K4 = = = = (5 (5 (5 (5 + + + + 20 + 3 + 6) / 2 = 17 4 + 3) / 1 = 12 20 + 3+ 6) / 2 = 17 4 + 3) / 1 = 12 Therefore, the minimum cycle time Cm = max{17, 12, 17, 12} = 17 4.8.2 SPN and GSPN Stochastic Petri nets (SPNs) have been developed [12,13] to model the nondeterministic behavior of a DES De®nition 37 A continuous-time stochastic Petri net, SPN, is de®ned as a TTPN with a set of stochastic timed transitions The ®ring time of transition ti is an exponentially distributed variable with ®ring rate i > 0 Generalized stochastic Petri nets (GSPNs) [14] incorporate both stochastic timed transitions and immediate transitions The immediate transitions ®re in zero time Additional modeling capabilities are introduced to GSPNs without destroying the equivalence with Markov chains They are inhibitor arcs, priority functions, and random switches An inhibitor arc in the net prevents a transition from ®ring when certain conditions are true A priority function speci®es a rule for the marking in which both timed and immediate transitions are enabled The random switch, as a discrete probability distribution, resolves con¯icts between two or more immediate transitions The generation of a Markov chain can be greatly simpli®ed through SPN and GSPNs approaches Speci®cally, one needs to: 1 2 Model the system with a SPN or GSPN Check the liveness and boundedness of the model by examining the underlying Petri net model with either reachability tree or invariants analysis The liveness and boundedness properties are related to the existence of the steadystate probabilities distribution of the equivalent Markov chain [13] Copyright © 2000 Marcel Dekker, Inc 3 Obtain the equivalent Markov chain from the reachability tree 4 Solve the equivalent Markov chain by a set of linear algebraic equations, i.e., the steady-state probabilities The steady-state probabilities obtained from the Markov chain could be used to compute (1) the expected number of tokens in a place; (2) the probability that a place is not empty; (3) the probability that a transition is enabled; and (4) performance measures such as average production rate, average in-process inventory, and average resource utilization It is interesting to note that the solution of a GSPN may be obtained with less effort than what is required to solve the corresponding SPN, especially if many immediate transitions are involved [14] 4.9 PETRI NET MODEL SYNTHESIS TECHNIQUES AND PROCEDURES Modeling a practical DES with Petri nets can be done using stepwise re®nement technologies [2] In this approach, Petri net modeling starts with a simple, coarsely detailed model that can be easily veri®ed as a live, bounded, and reversible one Then, the transitions and places of the initial model are replaced by special subnets step by step to capture more details about the system Each successive re®nement will guarantee the preservation of the desired properties of the initial model This process is to be repeated until the required modeling detail is achieved Through this approach, the computational dif®culties of checking a large Petri net model for liveness, boundedness, and reversibility are avoided 4.9.1 Initial Petri Net Model Figure 7 shows an initial Petri net model that contains n + k places and two transitions, where: Places p1, p2, , pn are n operation places that represent n concurrently working subsystems Places pn+1, pn+2, , pn+k are k resource places Transition t1 represents the beginning of the working of the system Transition t2, represents the end of the working of the system Petri Nets 353 Figure 9 An example of (a) choice Petri net and (b) a mutual exclusion Petri net 4.9.3 Modeling Dedicated Resources Example 15 Given a subnet as shown in Fig 10a that is a live, reversible, and safe Petri net with respect to an initial marking m0, one may add a dedicated resource (i.e., tokens at place pd) to the subnet as shown in Fig 10b It has been veri®ed [2] that the new Petri net is also safe, live, and reversible with respect to new initial marking M0 4.9.4 Stepwise Re®nement of Transitions Re®nements of transitions in a Petri net use the concepts of a block Petri net, an associated Petri net of a block Petri net, and a well-formed block Petri net [15] as shown in Fig 11 De®nition 38 A block Petri net is a Petri net that starts always from one initial transition, tin, and ends with one ®nal transition, tf ” De®nition 39 An associated Petri net, PN , of a block Petri net is obtained by adding an idle place p0 to the block Petri net such that (1) tin is the only output transition of p0; (2) tf is the only input transition to p0; (3) ” the initial marking of the associated Petri net is m0 and ” m0 …p0 † ˆ 1 Figure 10 Petri net (a) is augmented into Petri net (b) Copyright © 2000 Marcel Dekker, Inc De®nition 40 A well-formed Petri net block must be ” ” a live associated Petri net PN with m0 ˆ m0 …p† ” ˆ m0 …p0 † ˆ 1 Petri Nets the concurrent groups of operation processes are successive 4.9.5.2 Sequential Mutual Exclusion (SME) One typical example of a SME is shown in Fig 12b It has a shared resource place p6 and a group of sets of transition pairs (t1, t2) and (t3, t4) The token initially marked at p6 models a single shared resource, and the groups of transitions model the processes that need the shared resource sequentially This implies that there is a sequential relationship and a mutual dependency between the ®ring of a transition in one group and the ®ring of a transition in another group The properties of an SME such as liveness are related to a concept called token capacity De®nition 41 The maximum number of ®rings of ti from the initial marking without ®ring tj is the token capacity c…ti ; tj † of an SME The value of c(ti, tj) depends on the structure and the initial marking of an SME It has been shown [16] that when the initial marking (tokens) on dedicated resource places is less than or equal to c(ti, tj), the net with and without the shared resource exhibits the same properties For example, in Fig 12b, p1 is a dedicated resource place and the initial marking of the net is (3 0 0 0 2 1) It is easy to see that t2 can only ®re at most two times before t3 must be ®red to release two lost tokens at p5 Otherwise, no processes can continue Thus, the token capacity of the net is 2 As long as 1 m0 …p1 † 2, the net is live, bounded, and reversible 4.9.6 1 2 Petri Net Synthesis Technique Procedure [17] Start an initial Petri net model that is live, bounded, and reversible This model should be a macrolevel model that captures important system interactions in terms of major activities, choices, and precedence relations All places are either operation places, ®xed resource places, or variable resource places Use stepwise re®nement to decompose the operation places using basic design modules until all the operations cannot be divided or until one reaches a point where additional detail is not needed At each stage, add the dedicated Copyright © 2000 Marcel Dekker, Inc 355 resource places before proceeding with additional decomposition 3 Add shared resources using bottom-up approach At this stage, the Petri net model will be merged to form the ®nal net The place where the resource is shared by k parallel processes is speci®ed so that it forms a k-PME The place where the resource is shared by several sequentially related processes is added such that the added place and its related transitions form an SME 4.10 EXTENSIONS TO ORIGINAL PETRI NETS Based on the original Petri nets concept, researchers have developed different kinds of extended Petri nets (EPNs) for different purposes The key step for developing EPNs is the developments of the theory that supports the extensions de®ned in the nets As an example, timed Petri nets are well-developed EPNs that are used for system performance analysis Similarly, to aid the modeling of the ¯ow of control, resources, parts, and information through complex systems such as CIM and FMS, multiple classes of places, arcs, and tokens are introduced to the original Petri net to form new EPNs With these extensions, system modeling can proceed through different levels of detail while preserving structural properties and avoiding deadlocks 4.10.1 Multiple Places Five types of places that are commonly used in EPNs are developed to model ®ve common classes of conditions that may arise in a real system They are status place, simple place, action place, subnet place, and switch place as shown in Fig 13 Each place may also have a type of procedure associated with it if the net is used as a controller A status place is equivalent to a place in an original Petri net Its only procedure is the enable check for the associated transitions A simple place has a simple procedure associated with it in addition to the transitionenable check An action place is used to represent procedures that take a long time to be executed Usually, these procedures are spawned off as subprocesses that are executed externally in parallel with the Petri netbased model, for example, on other control computers Petri Nets 12 JB Dugan Extended Stochastic Petri nets: applications and analysis PhD thesis, Duke University, July 1984 13 MK Molloy Performance analysis using stochastic Petri nets IEEE Trans Computers C-31: 913±917, 1982 14 MA Marsan, G Conte, G Balbo A class of generalized stochastic Petri nets for the performance evaluation of multiprocessor systems ACM Trans Comput Syst 2: 93±122, 1984 15 R Valette Analysis of Petri nets by stepwise re®nements J Computer Syst Sci 18: 35±46, 1979 Copyright © 2000 Marcel Dekker, Inc 357 16 M Zhou, F DiCesare Parallel and sequential mutual exclusions for Petri net modeling of manufacturing systems with shared resources IEEE Trans Robot Autom 7: 515±527, 1991 17 M Zhou, F DiCesare, AA Desrochers A hybrid methodology for synthesis of Petri net models for manufacturing systems IEEE Trans Robot Autom 8: 350±361, 1992 18 K Jenson Colored Petri Nets: Basic Concepts, Analysis Methods and Practical Use, vol 1 New York: Springer-Verlag, 1993 Chapter 4.5 Decision Analysis Hiroyuki Tamura Osaka University, Toyonaka, Osaka, Japan 5.1 INTRODUCTION dimensionality of the functions that are required to be assessed These conditions restrict the form of a multiattribute utility function in a decomposition theorem In this section, after a brief description of an expected utility model of von Neumann and Morgenstern [2], additive, multiplicative, and convex decompositions are described for multiattribute utility functions [1,3] This chapter attempts to show the central idea and results of decision analysis and related decision-making models without mathematical details Utility theory and value theory are described for modeling value perceptions of a decision maker under various situations, risky or riskless situations, and situation of single or multiple attributes An analytic hierarchy process (AHP) is also included, taking into account the behavioral nature of multiple criteria decision making 5.2 5.2.1 Let A ˆ fa; b; F F Fg be a set of alternative actions from which a decision maker must choose one action Suppose the choice of a P A results in a consequence xi with probability pi and the choice of b P A results in a consequence xi with probability qi , and so forth Let UTILITY THEORY Multiattribute utility theory is a powerful tool for multiobjective decision analysis, since it provides an ef®cient method of identifying von Neumann± Morgernstern utility functions of a decision maker The book by Keeney and Raiffa [1] describes in detail the standard approach The signi®cant advantage of the multiattribute utility theory is that it can handle both uncertainty and multiple con¯icting objectives: the uncertainty is handled by assessing the decision maker's attitude towards risk, and the con¯icting objectives are handled by making the utility function multidimensional (multiattribute) In many situations, it is practically impossible to assess directly a multiattribute utility function, so it is necessary to develop conditions that reduce the X ˆ fx1 ; x2 ; F F Fg be a set of all possible consequences In this case pi ! 0; qi ! 0; F F F Vi ˆ ˆ pi ˆ qi ˆ Á Á Á ˆ 1 i i Let a real function u be a utility function on X Then the expected utilities of actions a; b; F F F are written, respectively, as ˆ ˆ pi u…xi †; Eb ˆ qi u…xi †; F F F …1† Ea ˆ i 359 Copyright © 2000 Marcel Dekker, Inc Expected Utility Model i 360 Tamura The assertion that the decision maker chooses an alternative action as if he maximizes his expected utility is called the expected utility hypothesis of von Neumann and Morgenstern [2] In other words, the decision maker chooses an action according to the normative rule a 1 b D Ea > Eb a $ b D Ea ˆ Eb …2† where a 1 b denotes ``a is preferred to b,'' and a $ b denotes ``a is indifferent to b.'' This rule is called the expected utility rule A utility function which satis®es Eqs (1) and (2) is uniquely obtained within the class of positive linear transformations Figure 1 shows a decision tree and lotteries which explain the above-mentioned situation, where `a ; `b ; F F F denote lotteries which the decision maker comes across when he chooses the alternative action a; b; F F F ; respectively, and described as `a ˆ …x1 ; x2 ; F F F Y p1 ; p2 ; F F F† `b ˆ …x1 ; x2 ; F F F Y q1 ; q2 ; F F F† De®nition 1 A certainty equivalent of lottery `a is an ” amouint x such that the decision maker is indifferent ” between `a and x From the expected untility hypothesis we obtain ˆ ” pi u…xi † …3† u…x† ˆ u…Ea † ˆ i In a set X of all possible consequences, let x0 and xà be the worst and the best consequences, respectively Since the utility function is unique within the class of positive linear transformation, let us normalize the utility function as u…x0 † ˆ 0 u…xà † ˆ 1 Let hxà ; p; x0 i be a lottery yielding consequences xà and x0 with probabilities p and …1 À p†, respectively In particular, when p ˆ 0:5 this lottery is called the 50±50 lottery and is denoted as hxà ; x0 i Let x be a certainty equivalent of lottery hxà ; p; x0 i, that is, x $ hxà ; p; x0 i Then u…x† ˆ pu…xà † ‡ …1 À p† u…x0 † ˆ p It is easy to identify a single-attribute utility function of a decision maker by asking the decision maker about the certainty equivalents of some 50±50 lotteries and by means of a curve-®tting technique The attitude of a decision maker toward risk is described as follows De®nition 2 A decision maker is risk averse if he pre€ " fers the expected consequence x…ˆ i pi xi † of any lotteries to the lottery itself In this case ˆ " u…x† > pi u…xi † If a decision maker is risk averse, his utility function is concave The converse is also true A decision maker is risk neutral (prone) if and only if his utility function is linear (convex) 5.2.2 Multiattribute Utility Function The following results are the essential summary of Refs 1 and 3 Let a speci®c consequence x P X be characterized by n attributes (performance indices) X1 ; X2 ; F F F ; Xn (e.g., price, design, performance, etc., of cars, productivity, ¯exibility, reliability, etc., of manufacturing systems, and so on) In this case a speci®c consequence x P X is represented by x ˆ …x1 ; x2 ; F F F ; xn † Figure 1 A decision tree and lotteries Copyright © 2000 Marcel Dekker, Inc …4† i x1 P X1 ; x2 P X2 ; F F F ; xn P Xn A set of all possible consequences X can be written as a subset of an n-dimensional Euclidean space as X ˆ X1  X2  Á Á Á  Xn This consequence space is called n-attribute space An n-attribute utility function is de®ned on X ˆ X1  X2  Á Á Á  Xn as u X X1  X2  Á Á Á  Xn 3 R Let I be a subset of f1; 2; F F F ; ng with r …1 r < n† elements, and J be a complementary subset of I with …n À r† elements Suppose a set of n attributes fX1 ; X2 ; F F F ; Xn g is divided into two subsets fXi ; i P Ig and fXi ; i P Jg Let XI be an r-attribute space composed Decision Analysis 361 of fXi ; i P Ig, and XJ be an …n À r†-attribute space composed of fXi ; i P Jg Then X ˆ XI  XJ De®nition 3 Attribute XI is utility independent of attribute XJ , denoted XI …UI†XJ , if conditional preferences for lotteries on XI given xJ P XJ do not depend on the conditional level xJ P XJ Let us assume that x0 and xà are the worst level and I I the best level of the attribute XI , respectively De®nition 4 Given an arbitrary xJ P XJ , a normalized conditional utility function uI …xI j xJ † on XI is de®ned as uI …xI j xJ † Xˆ u…xI ; xJ † À u…x0 ; xJ † I u…xà ; xJ † À u…x0 ; xJ † I I …5† where it is assumed that u…xà ; xJ † > u…x0 ; xJ † I I From De®nition 4 it is obvious that uI …xà j xJ † ˆ 1 I uI …x0 j xJ † ˆ 0 I VxJ P XJ From De®nitions 3 and 4 the following equation holds, if XI …UI†XJ : uI …xI j xJ † ˆ uI …xI j x0 † J VxJ P XJ In other words, utility independence implies that the normalized conditional utility functions do not depend on the different conditional levels De®nition 5 Attributes X1 ; X2 ; F F F ; Xn are mutually utility independent, if XI …UI†XJ for any I & f1; 2; F F F ; ng and its complementary subset J Theorem 1 Attributes X1 ; X2 ; F F F ; Xn are mutually utility independent, if and only if u…x† ˆ u…x1 ; x2 ; F F F ; xn † ˆ n ˆ iˆ1 ki ui …xi † if n ˆ iˆ1 ki ˆ 1 …6† or ku…x† ‡ 1 ˆ n ‰ iˆ1 fkki ui …xi † ‡ 1g if n ˆ iˆ1 ki Tˆ 1 …7† holds, where u…x0 ; x0 ; F F F ; x0 † ˆ 0 1 2 n ui …xi † Xˆ ui …xi j x0c † i u…xà ; xà ; F F F ; xà † ˆ 1 1 2 n ic ˆ f1; F F F ; i À 1; i ‡ 1; F F F ; ng ki ˆ u…xà ; x0c † i i and k is a solution of Copyright © 2000 Marcel Dekker, Inc k‡1ˆ n ‰ iˆ1 …kki ‡ 1† De®nition 6 Attributes X1 ; X2 ; F F F ; Xn are additive independent if preferences over lotteries on X1 ; X2 ; F F F ; Xn depend only on their marginal probability distributions, not on their joint probability distribution Theorem 2 Attributes X1 ; X2 ; F F F ; Xn are additive independent if and only if Eq (6) holds From Theorems 1 and 2 the additive independence is a special case of mutual utility independence For notational simplicity we deal only with the twoattribute case …n ˆ 2† in the following discussion The cases with more than two attributes are discussed in Tamura and Nakamura [3] We deal with the case where u1 …x1 j x2 † Tˆ u1 …x1 † u2 …x2 j x1 † Tˆ u2 …x2 † for some x2 P X2 for some x1 P X1 that is, utility independence does not hold between the attributes X1 and X2 De®nition 7 Attribute X1 is mth-order convex dependent on attribute X2 , denoted X1 …CDm †X2 , if there exist distinct xj2 P X2 … j ˆ 0; 1; F F F ; m† and real functions j X X2 3 R … j ˆ 0; 1; F F F ; m† on X2 such that the normalized conditional utility function u1 …x1 j x2 † can be written as u1 …x1 j x2 † ˆ m ˆ jˆ0 j …x2 †u1 …x1 j xj2 † m ˆ jˆ1 j …x2 † ˆ 1 …8† for all x1 P X1 ; x2 P X2 , where m is the smallest nonnegative integer for which Eq (8) holds This de®nition says that, if X1 …CDm †X2 , then any normalized conditional utility function on X1 can be described as a convex combination of …m ‡ 1† normalized conditional utility functions with different conditional levels where the coef®cients j …x2 † are not necessarily nonnegative In De®nition 7, if m ˆ 0, then u1 …x1 j x2 † ˆ u1 …x1 j x0 † for all x2 P X2 This implies 2 X1 …CD0 †X2 A X1 …UI†X2 that is, zeroth-order convex dependence is nothing but the utility independence This notion shows that the Chapter 5.1 Sensors: Touch, Force, and Torque Richard M Crowder University of Southampton, Southampton, England 1.1 INTRODUCTION 1.2 The objective of any robotic sensing system is to acquire knowledge and resolve uncertainty about the robot's environment, including its relationship with the workpiece Prior to discussing the requirements and operation of speci®c sensors, the broad objectives of sensing need to be considered The control of a manipulator or industrial robot is based on the correct interpretation of sensory information This information can be obtained either internally to the robot (for example, joint positions and motor torques) or externally using a wide range of sensors The sensory information can be obtained from both vision and nonvision sensors A vision system allows the position and orientation of the workpiece to be acquired; however, its performance is dependent on lighting, perspective distortion, and the background A touch, force, or torque sensor will provide information regarding the contact between the sensor and workpiece, and is normally localized in nature It is recognized that these sensors will not only complement vision sensing, but offer a powerful sensing capability in their own right Vision may guide the robot arm through many manufacturing operations, but it is the sense of touch that will allow the robot to perform delicate manipulations and assembly tasks Touch and tactile sensors are devices which measure the parameters of a contact between the sensor and an object This interaction obtained is con®ned to a small de®ned region This contrasts with a force and torque sensor, which measures the total forces being applied to an object In the consideration of tactile and touch sensing, the following de®nitions are commonly used: Touch sensing This is the detection and measurement of a contact force at a de®ned point A touch sensor can also be restricted to binary information, namely, touch and no touch Tactile sensing This is the detection and measurement of the spatial distribution of forces perpendicular to a predetermined sensory area, and the subsequent interpretation of the spatial information A tactile sensing array can be considered to be a coordinated group of touch sensors Slip This is the measurement and detection of the movement of an object relative to the sensor This can be achieved either by a specially designed slip sensor or by the interpretation of the data from a touch sensor or a tactile array Tactile sensors can be used to sense a diverse range of stimuli, from detecting the presence or absence of a grasped object to a complete tactile image A tactile 377 Copyright © 2000 Marcel Dekker, Inc TOUCH AND TACTILE SENSING 378 sensor consists of an array of touch-sensitive sites; the sites may be capable of measuring more than one property The contact forces measured by a sensor are able to convey a large amount of information about the state of a grip Texture, slip, impact, and other contact conditions generate force and position signatures that can be used to identify the state of a manipulation This information can be determined by examination of the frequency domain, and is fully discussed in the literature [1] As there is no comprehensive theory available that de®nes the sensing requirements for a robotic system, much of the knowledge is drawn from investigation of human sensing, and the analysis of grasping and manipulation Study of the human sense of touch suggests that creating a gripper incorporating tactile sensing requires a wide range of sensors to fully determine the state of a grip The detailed speci®cation of a touch sensor will be a function of the actual task it is required to perform Currently, no general speci®cation of a touch or tactile sensor exists Reference 2, though dated, can be used as an excellent basis for de®ning the desirable characteristics of a touch or tactile sensor suitable for the majority of industrial applications: A touch sensor should ideally be a single-point contact, though the sensory area can be any size In practice, an area of 1±2 mm2 is considered a satisfactory compromise between the dif®culty of fabricating a subminiature sensing element and the coarseness of a large sensing element The sensitivity of the touch sensor is dependent on a number of variables determined by the sensor's basic physical characteristic In addition the sensitivity may also be the application, in particular any physical barrier between the sensor and the object A sensitivity within the range 0.4 to 10 N, together with an allowance for accidental mechanical overload, is considered satisfactory for most industrial applications The minimum sensor bandwidth should be 100 Hz The sensor's characteristics must be stable and repeatable with low hysteresis A linear response is not absolutely necessary, as information-processing techniques can be used to compensate for any moderate nonlinearities As the touch sensor will be used in an industrial application, it will need to be robust and protected from environmental damage If a tactile array is being considered, the majority of applications can be undertaken by an array 10± Copyright © 2000 Marcel Dekker, Inc Crowder 20 sensors square, with a spatial resolution of 1±2 mm In a dexterous end effector, the forces and relative motions between the grasped object and the ®ngers need to be controlled This can be achieved by using a set of sensors capable of determining in real time the magnitude, location, and orientation of the forces at the contact point This problem has been approached by using miniature force and torque sensors inside the ®ngertips, to provide a robot with an equivalent to the kinesthetic sense found in humans The integration of skinlike and kinesthetic-like sensing will result in robots being equipped with arti®cial haptic perceptions [3] The study of human touch and the use of perceived information indicates that other variables, such as hardness and thermal properties, can also be measured, and this allows greater ¯exibility in an automated process Human touch is of considerable complexity, with sensors that respond to a range of stimuli including temperature, pain, acceleration, velocity, and intensity The human touch sensors in the skin may have many purposes, but are predominantly protective to prevent self-in¯icted damage to the body The human touch sense is obtained by a combination of four sensors: a transient load detector, a continuous force sensor, a position transducer to give proprioceptive data, and an overload sensor (i.e., pain) reacting both to force and other external environmental conditions This combination of sensors is very sensitive, e.g., a ®ne surface texture can be detected, but there is poor spatial resolution; the dif®culty in reading Braille is readily apparent Humans are very good at learning about an unknown object from touch The information from the sensors is brought together through the nervous system to give us the sense of feel It should be noted that the sensory information is processed and interpreted both locally (peripheral nervous system) and centrally (spinal cord and the brain) 1.2.1 Touch Sensor Technology Many physical principles have been exploited in the development of tactile sensors As the technologies involved are very diverse, this chapter can only consider the generalities of the technology involved In most cases, the developments in tactile sensing technologies are application driven It should be recognized that the operation of a touch or tactile sensor is very dependent on the material of the object being gripped Sensors 379 The sensors discussed in this chapter are capable of working with rigid objects However, if nonrigid material is being handled, problems may arise Work has shown that conventional sensors can be modi®ed to operate with nonrigid materials [4] 1.2.1.1 Mechanically Based Sensors The simplest form of touch sensor is one where the applied force is applied to a conventional mechanical microswitch to form a binary touch sensor The force required to operate the switch will be determined by its actuating characteristics and any external constraints Other approaches are based on a mechanical movement activating a secondary device, such as a potentiometer or displacement transducer 1.2.1.2 Resistive-Based Sensors The use of compliant materials that have a de®ned force-resistance characteristics have received considerable attention in touch and tactile sensor research [5] The basic principle of this type of sensor is the measurement of the resistance of a conductive elastomer or foam between two points The majority of the sensors use an elastomer that consists of a carbon-doped rubber The resistance of the elastomer changes with the application of force, resulting from the deformation of the elastomer altering the particle density (Fig 1) If the resistance measurement is taken between opposing surfaces of the elastomer, the upper contacts have to be made using a ¯exible printed circuit to allow movement under the applied force Measurement from one side can easily be achieved by using a dot-and-ring arrangement on the substrate (Fig 2) Resistive sensors have also been developed using elastomer cords laid in a grid pattern, with the resistance measurements being taken at the points of intersection Arrays with 256 elements have been constructed [6] This type of sensor easily allows the construction of a tactile image of good resolution The conductive elastomer or foam-based sensor, while relatively simple, does suffer from a number of signi®cant disadvantages: An elastomer has a long nonlinear time constant In addition the time constant of the elastomer, when force is applied, is different from the time con- Figure 1 Resistive sensor based on a conductive foam or elastomer (a) Principle of operation (b) Normalized resistance against applied force Copyright © 2000 Marcel Dekker, Inc Sensors 381 To maximize the change in capacitance as force is applied, it is preferable to use a high-permittivity dielectric in a coaxial capacitor design Figure 3b shows the cross-section of the capacitive touch transducer in which the movement of one set of the capacitor's plates is used to resolve the displacement and hence applied force The use of a highly dielectric polymer such as polyvinylidene ¯uoride maximizes the change in capacitance From an application viewpoint, the coaxial design is better as its capacitance will give a greater increase for an applied force than the parallel plate design In both types of sensors, as the size is reduced to increase the spatial resolution, the sensor's absolute capacitance will decrease With the limitations imposed by the sensitivity of the measurement techniques, and the increasing domination of stray capacitance, there is an effective limit on the resolution of a capacitive array To measure the change in capacitance, a number of techniques can be, the most popular is based on the use of a precision current source The charging characteristic of the capacitive sensor is given by Iˆ C dv "A dV ˆ dt d dt …2† hence, the voltage across the sensor over a period of time is de®ned as I dt d dV ˆ "A …3† As the current source, I, and sampling period, dt, are de®ned, the capacitance and hence the applied force can be determined [7] A second approach is to use the sensor as part of a tuned or LC circuit, and measure the frequency response Signi®cant problems with capacitive sensors can be caused if they are in close proximity with the end effector's or robot's earthed metal structures, as this leads to stray capacitance This can be minimized by good circuit layout and mechanical design of the touch sensor It is possible to fabricate a parallel plate capacitor on a single silicon slice [8] This can give a very compact sensing device; this approach is discussed in Sec 1.2.1.10 1.2.1.5 Magnetic-Based Sensor There are two approaches to the design of touch or tactile sensors based on magnetic transduction Firstly, the movement of a small magnet by an applied force will cause the ¯ux density at the point of measurement to change The ¯ux measurement can be made by either a Hall effect or a magnetoresistive device Second, the core of the transformer or inductor Copyright © 2000 Marcel Dekker, Inc can be manufactured from a magnetoelastic material that will deform under pressure and cause the magnetic coupling between transformer windings, or a coil's inductance, to change A magnetoresistive or magnetoelastic material is a material whose magnetic characteristics are modi®ed when the material is subjected to changes in externally applied physical forces [9] The magnetorestrictive or magnetoelastic sensor has a number of advantages that include high sensitivity and dynamic range, no measurable mechanical hysteresis, a linear response, and physical robustness If a very small permanent magnet is held above the detection device by a compliant medium, the change in ¯ux caused by the magnet's movement due to an applied force can be detected and measured The ®eld intensity follows an inverse relationship, leading to a nonlinear response, which can be easily linearized by processing A one-dimensional sensor with a row of 20 Hall-effect devices placed opposite a magnet has been constructed [10] A tactile sensor using magnetoelastic material has been developed [11], where the material was bonded to a substrate, and then used as a core for an inductor As the core is stressed, the material's susceptibility changes; this is measured as a change in the coil's inductance 1.2.1.6 Optical Sensors The rapid expansion of optical technology in recent years has led to the development of a wide range of tactile sensors The operating principles of opticalbased sensors are well known and fall into two classes: Intrinsic, in which the optical phase, intensity, or polarization of transmitted light are modulated without interrupting the optical path Extrinsic, where the physical stimulus interacts with the light external to the primary light path Intrinsic and extrinsic optical sensors can be used for touch, torque, and force sensing For industrial applications, the most suitable will be that which requires the least optical processing For example, the detection of phase shift, using interferometry, is not considered a practical option for robotic touch and force sensors For robotic touch and force-sensing applications, the extrinsic sensor based on intensity measurement is the most widely used due to its simplicity of construction and the subsequent information processing The potential bene®ts of using optical sensors can be summarized as follows: Immunity to external electromagnetic interference, which is widespread in robotic applications 382 Crowder Intrinsically safe The use of optical ®ber allows the sensor to be located some distance from the optical source and receiver Low weight and volume Touch and tactile optical sensors have been developed using a range of optical technologies: Modulating the intensity of light by moving an obstruction into the light path The force sensitivity is determined by a spring or elastomer To prevent crosstalk from external sources, the sensor can be constructed around a deformable tube, resulting in a highly compact sensor (Fig 4a) A design approach for a re¯ective touch sensor is shown in Fig 4b, where the distance between the re¯ector and the plane of source and the detector is the variable The intensity of the received light is a function of distance, and hence the applied force The U-shaped spring was manufactured from spring steel, leading to a compact overall design This sensor has been successfully used in an anthropomorphic end effector [12] A re¯ective sensor can be constructed with source±receiver ®ber pairs embedded in a solid elastomer structure As shown in Fig 4c, above the ®ber is a layer of clear elastomer topped with a re¯ective silicone rubber layer The amount of light re¯ected to the receiver is determined by an applied force that changes the thickness of the clear elastomer For satisfactory operation the clear elastomer must have a lower compliance than the re¯ective layer By the use of a number of transmitter±receiver pairs arranged in a grid, the tactile image of the contact can be determined [13] (a) (b) (c) Figure 4 (a) Optical touch sensor based on obstructing the light path by a deformable tube (b) Optical re¯ective touch sensor (c) Optical re¯ective sensor based on two types of elastomer Copyright © 2000 Marcel Dekker, Inc Sensors 383 Photoelasticity is the phenomenon where stress or strain causes birefringence in optically transparent materials Light is passed through the photoelastic medium As the medium is stressed, it effectively rotates the plane of polarization, and hence the intensity of the light at the detector changes as a function of the applied force [14] A suitable sensor is discussed in Section 1.2.2.2 A change in optical density occurs at a boundary, and determines if total internal re¯ection may occur As shown in Fig 5, an elastomer membrane is separated by air from a rigid translucent medium that is side illuminated If the elastomer is not in contact with the surface, total internal re¯ection will occur and nothing will be visible to the detector However, as the membrane touches the top surface of the lower medium, the boundary conditions will change, thus preventing total internal re¯ection, and the light will be scattered Hence an image will be seen by the detector The generated image is highly suitable for analysis by a vision system [15] 1.2.1.7 Optical-Fiber-Based Sensors In the previous section, optical ®bers were used solely for the transmission of light to and from the sensor; however, tactile sensors can be constructed from the ®ber itself A number of tactile sensors have been developed using this approach In the majority of cases either the sensor structure was too big to be attached to the ®ngers of a robotic hand or the operation was too complex for use in the industrial environment A suitable design can be based on internal-state microbending of optical ®bers Microbending is the process of light attenuation in the core of ®ber where a mechanical bend or perturbation (of the order of few microns) is applied to the outer surface of the ®ber The degree of attenuation depends on the ®ber parameters as well as radius of curvature and spatial wavelength of the bend Research has demonstrated the feasibility of effecting microbending on an optical ®ber by the application of a force to a second orthogonal optical ®ber [16] One sensor design comprises four layers of ®bers, each layer overlapping orthogonally to form a rectangular grid pattern The two active layers are sandwiched between two corrugation layers, where the ®bers in adjcent layers are slightly staggered from each other for better microbending effect When the force is applied to a ®ber intersection, microbending appears in the stressed ®bers, attenuating the transmitted light The change in the light intensity provides the tactile information 1.2.1.8 Piezoelectric Sensors Although quartz and some ceramics have piezoelectric properties, polymeric materials that exhibit piezoelectric properties are suitable for use as touch or tactile sensors; polymers such as polyvinylidene ¯uoride Figure 5 Optical boundary sensor Copyright © 2000 Marcel Dekker, Inc 384 Crowder (PVDF) are normally used [17] Polyvinylidene ¯uoride is not piezoelectric in its raw state, but can be made piezoelectric by heating the PVDF within an electric ®eld Polyvinylidene ¯uoride is supplied as sheets between 5 mm and 2 mm thick, and has good mechanical properties A thin layer of metalization is applied to both sides of the sheet to collect the charge and permit electrical connections to be made In addition it can be molded, hence PVDF has number of attractions when considering tactile sensor material as an arti®cial skin As a sensing element the PVDF ®lm acts as a capacitor on which charge is produced in proportion to the applied stress The charge developed can be expressed in terms of the applied stress, r ˆ ‰1 ; 2 ; 3 ŠT , the piezoelectric constant, d ˆ ‰d1 ; d2 ; d3 ŠT , and the surface area, giving q ˆ A Á r …4† The piezoelectric touch transducer is most often used in conjunction with a charge ampli®er; this results in an output voltage that is proportional to the applied stress Using a high-impedance ®eld-effect transistor (a) (FET) input ampli®er (Fig 6), the ampli®er's output voltage is given by vˆ dq dr R ˆ ARf d Á dt f dt which can be calibrated to give a force measurement The piezoelectric sensors are essentially dynamic, and are not capable of detecting static forces In practice their use is restricted to specialist applications such as slip and texture detection The use of PVDF in piezoelectric sensors causes dif®culty in scanning an array of sensing elements, as PVDF exhibits pyroelectric effects Therefore some applications require a reference sensor of unstressed PVDF to allow the separation of the piezoelectric effect from the pyroelectric signal 1.2.1.9 Strain Gages in Tactile Sensors A strain gage, when attached to a surface, will detect the change in length of the material as it is subjected to external forces The strain gage is manufactured from either resistive elements (foil, wire, or resistive ink) or from semiconducting material A typical resistive gage consists of a resistive grid bonded to an epoxy backing ®lm If the strain gage is prestressed prior to the application of the backing medium, it is possible to measure both tensile and compressive stresses The semiconducting strain gage is fabricated from a suitably doped piece of silicone; in this case the mechanism used for the resistance change is the piezoresistive effect [18] When applied to robotic touch applications, the strain gage is normally used in two con®gurations: as a load cell, where the stress is measured directly at the point of contact, or with the strain gage positioned within the structure of the end effector 1.2.1.10 (b) Figure 6 PVDF touch sensor (a) De®nition used in the polarization of PVDF ®lm (b) Equivalent circuit of a sensor Copyright © 2000 Marcel Dekker, Inc …5† Silicon-Based Sensors Technologies for micromachining sensors are currently being developed worldwide The developments can be directly linked to the advanced processing capabilities of the integrated circuit industry, which has developed fabrication techniques that allow the interfacing of the nonelectronic environment to be integrated through microelectromechanical systems [19] Though not as dimensionally rigorous as the more mature silicon planar technology, micromachining is inherently more complex as it involves the manufacture of a threedimensional object Therefore the fabrication relies on additive layer techniques to produce the mechanical structure Sensors 385 The excellent characteristics of silicon, which have made micromachined sensors possible, are well known [20], and include a tensile strength comparable to steel, elasticity to breaking point, very little mechanical hysteresis in devices made from a single crystal, and a low thermal coef®cient of expansion To date it is apparent that microengineering has been applied most successfully to sensors Some sensor applications take advantage of the device-to-device or batch-to-batch repeatability of wafer-scale processing to remove expensive calibration procedures Current applications are restricted largely to pressure and acceleration sensors, though these in principle can be used as force sensors As the structure is very delicate, there are still problems in developing a suitable tactile sensor for industrial applications [21] 1.2.1.11 Smart Sensors The most sign®icant problem with the sensor systems discussed so far is that of signal processing Researchers are therefore looking to develop a complete sensing system rather than individual sensors, together with individual interfaces and interconnections This allows the signal processing to be brought as close as possible to the sensor itself or integrated with the sensor Such sensors are generally termed smart sensors It is the advances in silicon fabrication techniques which have enabled the recent developments in smart sensors There is no single de®nition of what a smart sensor should be capable of doing, mainly because interest in smart sensors is relatively new However, there is a strong feeling that the minimum requirements are that the sensing system should be capable of self-diagnostics, calibration, and testing As silicon can be machined to form moving parts such as diaphragms and beams, a tactile sensor can, in principle, be fabricated on single piece of silicon Very little commercial success has been obtained so far, largely due to the problems encountered in transferring the technology involved from the research laboratory to industry In all tactile sensors there is a major problem of information processing, and interconnection As an n Ân array has 2n connections and individual wires, any reduction in interconnection requirements is welcomed for ease of construction and increased reliability A number of researchers have been addressing the problem of integrating a tactile sensor with integral signal processing In this design the sensor's conductive elastomer sheet was placed over a substrate The signi®cant feature of this design is that the substrate Copyright © 2000 Marcel Dekker, Inc incorporates VLSI circuitry so that each sensing element not only measures its data but processes it as well Each site performs the measurements and processing operations in parallel The main dif®culty with this approach was the poor discrimination, and susceptibility to physical damage However, the VLSI approach was demonstrated to be viable, and alleviated the problems of wiring up each site and processing the data serially 1.2.1.12 Multistimuli Touch Sensors It has been assumed that all the touch sensors discussed in this section respond only to a force stimulus However, in practice most respond to other external stimuli, in particular, temperature If PVDF has to be used as a force sensor in an environment with a widely varying ambient temperature, there may be a requirement for a piece of unstressed PVDF to act as a temperature reference It is possible for a sensor to respond both to force and temperature changes This has a particular use for object recognition between materials that have different thermal conductivity, e.g., between a metal and a polymer [22] If the complexity of the interpretation of data from PVDF is unsuitable for an application, touch sensors incorporating a resistive elastomer for force, and thermistors for temperature measurement can be constructed By the use of two or more force-sensitive layers on the sensor, which have different characteristics (e.g., resistive elastomer and PVDF), it is possible to simulate the epidermal and dermal layers of human skin 1.2.2 Slip Sensors Slip may be regarded as the relative movement of one object's surface over another when in contact The relative movement ranges from simple translational motion to a combination of translational and rotational motions When handling an object, the detection of slip becomes necessary so as to prevent the object being dropped due to the application of a low grip force In an assembly operation, it is possible to test the occurrence of slip to indicate some predetermined contact forces between the object and the assembled part For the majority of applications some qualitative information on object slip may be suf®cient, and can be detected using a number of different approaches 386 1.2.2.1 Crowder Interpretation of Tactile-Array Information The output of a tactile-sensing array is the spatial distribution of the forces over the measurement area If the object is stationary, the tactile image will also remain stationary However, if the pattern moves with time, the object can be considered to be moving; this can be detected by processing the sensor's data 1.2.2.2 Slip Sensing Based on Touch-Sensing Information Most point-contact touch sensors are incapable of discrimination between relative movement and force However, as the surfaces of the tactile sensor and the object are not microscopically smooth, the movement of an object across the sensor will cause a high-frequency, low-amplitude vibration to be set up, which can be detected and interpreted as movement across the sensor This has been achieved by touch sensors based on the photoelastic effect [23] and piezoelectric [24] sensors In a photoelastic material the plane of polarization is a function of the material stress Figure 7a shows a sensor, developed at the University of Southampton, to detect slip The sensor uses the property of photoelastic material, where the plane of the material's polarization is rotated as the material is stressed In the sensor, light is ®rst passed through a polarizing ®lm (polarizer), the material, then a second polarizing ®lm (analyzer) As the stress applied to the material changes, the amount of received light varies Typical results are shown in Fig 7b; the changes in stress are caused by vibrations due to the photoelastic material slipping±sticking as the object moves relative to the sensor The sensitivity of the sensor can be increased by arti®cially roughening the surface area of the sensor In addition to slip detection, the information from the sensor can be used to determine information about the surface roughness of the gripped object by measurement of the vibration characteristics 1.2.2.3 erates vibrations which in turn stimulate a piezoelectric crystal The disadvantage of this approach is that it picks up external vibrations from the gripper and robot mechanics, and the needle frequently wears out The improved version of this sensor uses a steel ball at the end of the probe, with the piezoelectric crystal replaced by a permanent magnet and a coil enclosed in a damping medium To avoid the problem of interference signals from external vibrations, a range of interrupt-type slip sensors have been designed In one design, a rubber roller has a permanent magnet passing over a magnetic head which generates a voltage when slip occurs In a similar design the roller has a number of slits which interrupt an optical path; this allows an indication of slip to be obtained Though these sensors give a very good indication of the speed and direction of slip there are disadvantages with poor slip resolution and the possibility of jamming of the roller 1.2.3 Summary This section has discussed the technology available for touch, tactile, and slip sensors In the interpretation of a sensor's information, consideration should be taken of its design and use One aspect that is often overlooked is the mechanical ®ltering of the sensory information caused by the sensor's protective covering material Work has shown [25] that a cover of as little as 0.2 mm thick will degrade a sensor that is required to have a spatial resolution of less than 1 mm As shown in Fig 8, a point contact is diffused so that a number of sensors are stimulated The degree of ®ltering is a function of the covering material, its thickness, and its physical properties, and requires the use of ®nite-element tecniques to be analyzed fully For any application the characteristics of a number of sensors may need to be compared Table 1 presents a summary of the major advantages and disadvantages, allowing this comparison to be made between the transduction techniques Sensors to Speci®cally Detect Slip It is possible to develop sensors that will respond only to relative movement They are normally based on the principle of transduction discussed for touch sensors, but the sensors' stimulus comes from the relative movement of an area of the gripper Several methods to detect slip have been reported One sensor requries a sapphire needle protruding from a sensor surface to touch the slipping object; this gen- Copyright © 2000 Marcel Dekker, Inc 1.3 FORCE AND TORQUE MEASUREMENT As noted earlier, touch sensors operate at the point of contact If, however, it is required to measure the global forces and torques being exerted on an object by a robotic system, a multiaxis force measurement system is needed If an object ®xed in space is considered and Sensors 389 (a) (b) Figure 9 Four-beam wrist sensor (a) The generalized construction, showing the location of the strain gages and the robot and tool interfaces (b) The relationship between the strain gages and the six forces have high stiffness, to ensure that the disturbing forces are damped, allowing a high sampling rate The high stiffness will minimize the de¯ections of the wrist under the applied forces that lead to positional errors The sensor needs to be small so as not to restrict the movements of the robot within the workspace Within the individual joints of the robots by using joint torque sensing; however, inertia, gravity loading and joint friction present in a manipulator will complicate the determination of the forces The wrist sensor structure discussed above is effectively rigid; however, a force and torque sensor can be designed to have the fast error absorption of a passive compliance structure and the measurement capabilities of a multiaxis sensor The structure, including its sensing package, is normally termed the instrumented remote center compliance (IRCC) [30] In the IRCC, some or all of the remote center compliance de¯ections are measured From the de¯ections, the applied forces and torques can be determined and, if required, used in the robot control algorithms In the conventional IRCC the end effector interface plate is mounted on three shear blocks, with the base connected to the robot The applied forces cause the platform to move relative to the base and this movement gives the IRCC its remote center characteristics The movements can Copyright © 2000 Marcel Dekker, Inc be detected by the use of displacement transducers or by the use of two-dimensional optical position sensors In the position sensor, a light source ®xed to the platform illuminates the sensing element, allowing the position of the light source can be measured in two axes This, when combined with the outputs from the other position sensors, is used to calculate the applied force, both magnitude and direction It should be noted that the calculation of the applied forces using an IRCC is nontrivial due to the complex movement of the platform under load 1.3.1 External to the Robot The touch, torque, and force sensors that have been discussed are also suitable for mounting external to the robot In most assembly tasks, the robot is used to make two parts and the applied force can be measured by sensors attached to the workpiece This information can be used either to modify the robot's position, the applied forces, or to adjust the position of workpiece To achieve the latter, the workpiece has to be mounted on a table capable of multiaxis movement The construction of the table is similar to an IRCC and the same form of algorithms can be used It is possible to use any of the touch and tactile sensors external to the robot, the only limitation being the accuracy and resolution of the task being performed While the Maltese cross sensor is usually 390 Crowder placed within the robot's wrist, it can be placed at almost any point in the robot's structure After the wrist, the most common place is integral to the robot's base, where it is termed a pedestal sensor However, it is of only limited use at this point due to the complexity of the transformations 1.4 CONCLUDING COMMENTS The last decade has seen considerable effort applied to research and development activities related to the design of touch, force, and torque sensors, primarily for robotics applications This brief survey has not considered the processing of the measured data, sensory data fusion, and sensory-motor integration Research on these topics is rapidly expanding Most of the work related to the processing methodologies and algorithms have been focused on the analysis of static tactile images, following the lines developed in the ®eld of machine vision, some of which have been reported in the literature [31] This approach is limited in scope and does not consider a major asset of tactile sensing which lies in the processes of active touch and manipulation Planning active touch procedures and analyzing the pertinent sensor data was recognized early on as being important, but progress in this area has been quite slow As a ®nal observation, it should be noted that although the rapid growth in interest in this ®eld of sensing has initiated signi®cant progress in haptic technology, very little movement to real applications has occurred At present the market for these devices is still very marginal, despite some early optimistic forecasts Future widespread use of tactile and haptic systems is still foreseen, but the time scale for these events to occur should be realistically correlated with the great theoretical and technical dif®culties associated with this ®eld, and with the economic factors that ultimately drive the pace of its development REFERENCES 1 B Eberman, JK Salisbury Application of dynamic change detection to dynamic contact sensing Int J Robot Res 13(5): 369±394, 1994 2 LD Harmon Automated tactile sensing Int J Robot Res 1(2): 3±32, 1982 3 AM Okamura, ML Turner, MR Cutkosky Haptic exploitation of objects with rolling and sliding IEEE Conference on Robotics and Automation, 1997, New York, p 2485±2490 Copyright © 2000 Marcel Dekker, Inc 4 R Stone, P Brett A sensing technique for the measurement of tactile forces in the gripping of dough like material Proc Inst Mech Engrs 210(B3): 309±316, 1996 5 HR Nicholls Advanced Tactile Sensing for Robotics Singapore: World Scienti®c Publishing, 1992 6 BE Robertson, AJ Walkden Tactile sensor system for robotics In: A Pugh, ed Robot Sensors, vol 2: Tactile and Non-Vision Bedford, UK: IFS (Publications), 1986, p 89±97 7 BV Jayawant, MA Onori, J Watson Robotic tactile sensing: a new array sensor: robot sensors In: A Pugh, ed Robot Sensors, vol 2: Tactile and Non-Vision Bedford, UK: IFS (Publications), 1986, p 120±125 8 Y Lee, K Wise A batch fabricated silicone capacitive pressure transducer with low temperature sensitivity IEEE Trans Electron Dev ED-29(1): 1982, p 42±48 9 J Vranish, R Demoyer Outstanding potential shown by magnetoelastic force feedback sensors for robots Sensor Rev 2(4): 1982 10 G Kinoshita, T Hajika, K Hattori Multifunctional tactile sensors with multi-elements for ®ngers Proceedings of the International Conference on Advanced Robotics, 1983, pp 195±202 11 EE Mitchell, J Vranish Magnetoelastic force feedback sensors for robots and machine toolsÐan update Proceedings, 5th International Conference on Robot Vision and Sensory Controls, 1985, p 200±205 12 RM Crowder An anthropomorphic robotic end effector Robot Autonom Syst 1991, p 253±268 13 J Schneiter, T Sheridan An optical tactile sensor for manipulators Robot Computer Integr Manuf 1(1): 65±71, 1984 14 W Splillman, D McMahan Multimode Fibre Optic Sensor Based on Photoelastic Effect Sudby, MA: Sperry Research Centre, 1985 15 P Dario, D De Rossi Tactile sensors and the gripping challenge IEEE Spectrum 1985, p 46±52 16 D Jenstrom, C Chen A ®bre optic microbend tactile array sensor Sensors Actuators 20(3): 239±248, 1989 17 P Dario, R Bardelli, D de Rossi, L Wang Touch sensitive polymer skin uses piezoelectric properties to recognise orientation of objects Sensor Rev 2(4): 194±198, 1982 18 JM Borky, K Wise Integrates signal conditioning for silicone pressure sensing IEEE Trans Electron Dev ED26(12): 1906±1910, 1979 19 J Bryzek, K Petersen, W McCalley Micromachines on the march IEEE Spectrum 31(5): 20±31, 1994 20 L Vinto Can micromachining deliver? Solid State Technol 38: 57±59, 1995 21 J Smith, C Baron, J Fleming, S Montague, J Rodriguez, B Smith, J Sniegowski Micromachined sensors and actuator research at a microelectronics development laboratory Proceedings of the American Conference on Smart Structures and Materials, San Diego, 1995, pp 152±157 Sensors 22 D Siegel, I Garabieta, J Hollerbach An integrated tactile and thermal sensor IEEE Conference on Robotics and Automation, San Francisco, California, 1996, pp 1286±1291 23 F Kvansik, B Jones, MS Beck Photoelastic slip sensors with optical ®bre links for use in robotic grippers Institute of Physics Conference on Sensors, Southampton, UK, 1985, pp 58±59 24 R Howe, M Cutkosky Dynamic tactile sensing, perception of ®ne surface features with stress rate sensing IEEE Trans Robot Autom 9(2): 145±150, 1993 25 AM Shimojo Mechanical ®ltering effect of elastic covers for tactile sensing IEEE Trans Robot Autom 13(1): 128±132, 1997 Copyright © 2000 Marcel Dekker, Inc 391 26 H Van Brussel, H Berlien, H Thielemands Force sensing for advanced robotic control Robotics 2(2): 139±148, 1986 27 PM Will, D Grossman An experimental system for computer controlled assembly IEEE Trans Computers C-24(9) 1975, pp 879±87 28 B Roth, B Shimans On force sensing information and its use in controlling manipulators, Information Control problems in Manufacturing, Tokyo 1980, p 119±26 29 K Bejczy Smart sensors for smart hands Progr Astronaut Aeronaut 1980 30 T DeFazio, D Seltzer, DE Whitney The IRCC instrumented remote center compliance Rob Sens 2; 33±44, 1986 31 HR Nicholls, MH Lee A survey of robot tactile sensing technology Int J Robot Res 8(3): 3±30, 1989 Chapter 5.2 Machine Vision Fundamentals Prasanthi Guda, Jin Cao, Jeannine Gailey, and Ernest L Hall University of Cincinnati, Cincinnati, Ohio 2.1 INTRODUCTION tasks in a highly ef®cient manner that cannot yet be duplicated by machine vision However, the human visual system can also be confused by illusions The key element to vision is light, which is radiant energy in the narrow range of the electromagnetic spectrum, from about 350 nm (violet) to 780 nm (red) This energy, upon stimulation of the retina of the human eye, produces a visual sensation we call visible light Photometry and colorimetry are sciences that describe and quantify perceived brightness and color, and objective measurements of radiant energy The visual cortex of the human brain, as shown in Fig 1, is the central location for visual processing We believe that the inner layer, or white matter, of the brain consists mainly of connections, while the outer layer, or gray matter, contains most of the interconnections that provide neural processing The eyes function as an optical system whose basic components consist of the lens, the iris, and the retina The lens of the eye focuses the incoming light to form an inverted image on the retina at the rear wall of the eye The amount of light entering the eye is controlled by a muscle group called the iris The retina, as shown in Fig 2, consists of about 125 million light-sensitive receptors that, because of the many-to-one connections, have some processing capabilities These receptors consist of color-sensitive ``cones'' and brightness-sensitive ``rods.'' The central part of the retina is called the fovea It contains a dense cluster of between six and seven million cones that are sensitive to color and are connected directly to the brain via individual nerves The new machine vision industry that is emerging is already generating millions of dollars per year in thousands of successful applications Machine vision is becoming established as a useful tool for industrial automation, where the goal of 100% inspection of manufactured parts during production is becoming a reality The purpose of this chapter is to present an overview of the fundamentals of machine vision A review of human vision is presented ®rst to provide an understanding of what can and cannot be easily done with a machine vision system 2.2 HUMAN VISUAL SYSTEM Human beings receive at least 75% of their total sensory input through visual stimuli; our vision processes are executed in billions of parallel neural networks at high speeds, mainly in the visual cortex of the brain Even with the fastest supercomputers, it is still not possible for machines to duplicate all of the functions of human vision However, an understanding of the fundamentals of human image formation and perception provides a starting point for developing machine vision applications The human visual system comprises three main organs: the eyes, the optic nerve bundle, and the visual cortex of the brain This complex system processes a large amount of electrochemical data and performs its 393 Copyright © 2000 Marcel Dekker, Inc Machine Vision Fundamentals 395 input signal as shown in Eq (1) By the combination of linear and nonlinear processing, a model was developed which showed similar characteristics, as shown in Fig 3a An original image and the result of processing by the nonlinear model are shown in Fig 3b and c Note that the image is blurred slightly, but that a considerable edge enhancement is produced As may be observed in Fig 3, the reduction of irrelevant detail and enhancement of important edges in the dark regions of the image was achieved This effect could be analogous to night vision Perhaps early humans, living in caves or hunting and foraging at night, had required this survival ability to discern the outlines of predators in the dark shadows The neural network model may also be written to emphasize the fact that the response must be greater than a threshold, b, to produce an output: y ˆ f …g‰wT x À bŠ† …2† In this case, g represents the nonlinear, zero±one step function whose value is 1 when the threshold, b, is exceeded, and 0 otherwise Additionally, the function f may be another function that determines frequency sensitivity or recognition selectivity The overall neural network function is a composite function of a basic linear decision element combined with nonlinear function mappings that are characteristic of modern multilayer neural networks 2.3 Figure 3 Example of neural processing in the early stages of vision element followed by another network The inhibition equation may be written as ! ! ! ! y1 x1 0 w12 y1 ˆ À …1† y2 x2 w21 0 y2 where the output responses are yi , the input is xi and the coef®cients, wij , regulate the amount of inhibition This model was used to demonstrate the nonlinear nature of the frequency sensitivity of the human visual system [2] The human visual system was found to respond in a nonlinear manner to the contrast of an Copyright © 2000 Marcel Dekker, Inc MACHINE VISION HARDWARE COMPONENTS A machine vision system consists of hardware and software components The basic hardware components are of a light source, a solid state camera and lens, and a vision processor The usual desired output is data that is used to make an inspection decision or to permit a comparison with other data The key considerations for image formation are lighting and optics One of the ®rst considerations in a machine vision application is the type of illumination to be used Natural, or ambient, lighting is always available but rarely suf®cient Point, line, or area lighting sources may be used as an improvement over ambient light Spectral considerations should be taken into account in order to provide a suf®ciently high contrast between the objects and background Additionally, polarizing ®lters may be required to reduce glare or undesirable spectral re¯ections If a moving object is involved, a rapid shutter or strobe illumination can be used to capture an image without motion blur To obtain an 396 Guda et al excellent outline of an object's boundary, back lighting can provide an orthogonal projection used to silhouette an object Line illumination, produced with a cylindrical lens, has proven useful in many vision systems Laser illumination must be used with proper safety precautions, since high-intensity point illumination of the retina can cause permanent damage Another key consideration in imaging is selecting the appropriate camera and optics High-quality lenses must be selected for proper ®eld of view and depth of ®eld; automatic focus and zoom controls are available Cameras should be selected based on scanning format, geometrical precision, stability, bandwidth, spectral response, signal-to-noise ratio, automatic gain control, gain and offset stability, and response time A shutter speed or frame rate greater than one-thirtieth or one-sixtieth of a second should be used In fact, the image capture or digitization unit should have the capability of capturing an image in one frame time In addition, for camera positioning, the position, pan and tilt angles can be servo controlled Robot-mounted cameras are used in some applications Fortunately, with recent advances in solid-state technology, solid-state cameras are now available at a relatively lower cost Since the advent of the Internet and the World Wide Web (WWW), a great variety of images are now available to anyone This has also led to an increase in the variety of formats for image data interchange [3] Some of the most common image formats now are bitmaps (BMP), data-compressed JPEGs (JPG), and the GIF87a (GIF) ®le format The Graphics Interchange Format (GIF), shown in Table 1, was developed by CompuServe, and is used to store multiple bitmap Table 1 GIF Image Format Characteristics Header and color table information Image 1 Image 2 Image N Header Logical screen descriptor Global color table Local image descriptor Local color table Image data Local image descriptor Local color table Image data Local image descriptor Local color table Image data Trailer Copyright © 2000 Marcel Dekker, Inc images in a single ®le for exchange between platforms The image data is stored in a bitmap format in which numbers represent the values of the picture elements or pixels The bit depth determines the number of colors a pixel can represent For example, a 1-bit pixel can be one of two colors, whereas an 8-bit pixel can be one of 256 colors The maximum image size with the GIF format is 64,000 by 64,000 pixels The image data stored in a GIF ®le is always compressed using the Lempel±Ziv±Welch (LZW) technique The GIF data can also be interlaced up to 4:1 to permit images to display progressively instead of top down There are literally hundreds of various image ®le formats Table 2 lists some common formats as well as the extension, creator, and conversion ®lter(s) for each format 2.4 2.4.1 MACHINE VISION ALGORITHMS AND TECHNIQUES Image Functions and Characteristics As mentioned by Wagner [4], in manufacturing, human operators have traditionally performed the task of visual inspection Machine vision for automatic inspection provides relief to workers from the monotony of routine visual inspection, alleviates the problems due to lack of attentiveness and diligence, and in some cases improves overall safety Machine vision can even expand the range of human vision in the following ways: Improving resolution from optical to microscopic or electron microscopic Extending the useful spectrum from the visible to the x-ray and infrared or the entire electromagnetic range, and improving sensitivity to the level of individual photons Enhancing color detection from just red, green, and blue spectral bands to detecting individual frequencies Improving time response from about 30 frames per second to motion stopping strobe-lighted frame rates or very slow time lapse rates Modifying the point of view from the limited perspective of a person's head to locations like Mars, the top of a ®xture, under a conveyor or inside a running engine Another strength of machine vision systems is the ability to operate consistently, repetitively, and at a high rate of speed In addition, machine vision system components with proper packaging, especially solid-state ... x1 x2 Qà x3 x4 to mean that the difference of the strength -of- preference for x1 over x2 is greater than or equal to the difference of the strength -of- preference for x3 over x4 If it is assumed... advantage of the device-to-device or batch-to-batch repeatability of wafer-scale processing to remove expensive calibration procedures Current applications are restricted largely to pressure and acceleration... consists of about 125 million light-sensitive receptors that, because of the many-to-one connections, have some processing capabilities These receptors consist of color-sensitive ``cones'''' and brightness-sensitive