CHAPTER 5. SINGLE-LENS TRINOCULAR STEREOVISION In this chapter, we present a novel design for stereovision - a 3F filter (triprism) based single-lens trinocular stereovision system. This system can be considered as an extension of the single-lens binocular stereovision presented in Chapter 4. An image captured by this system is divided into three sub-images, called stereo image triplet, and these three sub-images can be taken as the images captured by three virtual cameras which are created by the 3F filter. The stereo image triplet is captured simultaneously by this system, and hence a dynamic scene will be handled by this system without any problem. Video rate image capturing is not a problem for this system too. The basic ideas of the two approaches used to study the previous single-lens binocular system are also applied here to model and determine this single-lens trinocular system: one based on calibration technique and another one based on geometrical analysis of ray sketching. The approach based on geometry analysis of ray sketching is still of the greater interest because of its significantly simpler implementation: it does not require the usual complicated calibration process but one simple field point test (see 4.1.3) to determine the whole system once the system is fixed and pin-hole camera model is used. In addition, greater consideration is given so that the mathematical analysis used in this approach can be generalized, with as minor modification as possible, to explain similar system which employ the prisms with similar pyramid-like structures but different number of faces (≥3), the so called single-lens multi-ocular stereovision systems which will be introduced in the next chapter. An implicit mathematical solution is given. Due to its complexity, this 50 solution can only be obtained numerically by computer programming, and is not like the explicit mathematical expressions for the single-lens binocular stereovision obtained by the geometrical analysis based approach. The mathematic method used by this approach is made generic to facilitate any comprehensive system analysis and also to provide a flexible way of analyzing any refractive ray problem involving any planar glass surface in 3-dimensional space. Experiments are conducted to test the feasibility of both approaches. Search of stereo correspondence is a difficult issue in stereovision. Trinocular stereovision, which enables to cross check the hypothesized correspondence using additional epipolar constraints, contributes to the solution of this problem. A short review on the epipolar geometry and its application to trinocular stereovision is given in Appendix A. Trinocular stereovision can also help to solve the problem of occlusion in stereovision and its redundant stereo information should lead to better accuracy in depth recovery. In 1986, the idea of trinocular stereovision was presented by Yachida, et al. [38]. Extensive discussions were given by Ayache [39][41][44]. A list of the pioneers of trinocular stereovision is given in [38]-[44]. The discussed trinocular vision systems appeared in the literature review include the orthogonal configuration and the non-orthogonal configuration. Because of the potential advantages of trinocular stereovision, researches are still carried out and different applications have been developed in recent years. A list of more recent works on trinocular stereovision is [45]-[52]. Chiou et al. [45] discussed the optimal camera geometry of trinocular stereovision with regards to system performance; Agrawal and Davis [52] studied the problem of shortest paths and the ordering constraint in correspondence searching of trinocular; Pollard et al. 51 [51] presented their application of trinocular stereo system on view synthesis. Discussions about trinocular stereovision can also be found in the book by Faugeras [5], Sonka et al. [7] and a discussion on its geometrical properties can be found in the book by Hartley and Zisserman [6]. Nevertheless, the price to pay for trinocular stereovision is due to the third camera which often increases the complexity of system setup, calibration and also camera synchronization. Developing a single-lens trinocular stereovision system may help to solve these problems, but very few works on single-lens trinocular stereovision systems that can perform simultaneous image capturing are reported. Some relevant but different works are presented by Kurada [53] and Ramsgaard [54]. Both systems need to employ mirrors in order to assist and can perform close range stereovision. The design of Kurada [53] uses a four-mirror setup such that the three views of a scene can be imaged onto a camera image plane side by side via a tri-split lens head. However its system configuration is relatively complex, and more importantly the three virtual camera optical axes are nearly co-planar, which results in the difficulty in the application of epipolar constraints for correspondence searching. The design of Ramsgaard [54] positions two rectangular mirrors to be perpendicular with each other and both are parallel with the real optical axis such that the camera can simultaneously detect one direct view of the object and also two reflected images of the object. However this system needs to capture an image via two reflections and the information from this system does not seem to be utilized easily because its quality depends on a perfect alignment. It also has the problem of inefficient CCD matrix usage. In our work, one alternative in building a single-lens trinocular stereovision system which can avoid the above problems is presented, and also detailed methods to model 52 the system are provided, including a method providing fast and efficient implementation. To our knowledge this design is novel. Part of the work reported in this chapter has been published in [55]. 5.1 Virtual Camera Generation The key issue to model and determine our single-lens trinocular system is on the determination of virtual cameras. If a 3F filter is vertically positioned in front of a CCD camera as shown in Figure 5.1, in which the shape of a 3F filter is also illustrated, the image plane of this camera will capture three different views of the same scene behind the filter in one shot. These three sub-images can be taken as the images captured by three virtual cameras which are generated by the 3F filter. One sample image captured by this system is given in Figure 5.2, from which significant differences among the three sub-images caused by different view angles and view scopes of the virtual cameras can be observed. It is assumed that each virtual camera consists of one unique optical center and one “planar” image plane. The challenge is to determine the properties of these virtual cameras, which mainly include their focal lengths, positions and orientations so that the disparity information on the sub-images can be exploited to perform depth recovery like a stereovision system. Furthermore, as these three views are captured simultaneously, this system theoretically possesses the merits of a typical trinocular stereovision system including its special properties on epipolar constraints, which provides a significant advantage in correspondence searching. Like the virtual camera model used for single-lens binocular stereovision system in the previous chapter, it is assumed that the Field of View (FOV) of each 53 virtual camera is constrained by two boundary lines (see Figure 5.4): one boundary line is the optical axis of the virtual camera which can be determined by backextending the refracted ray that is aligned with the real camera optical axis; and another FOV boundary line of the virtual camera can be determined by backextending the refracted ray that is aligned with the real camera FOV boundary line(s). The optical center of the virtual camera can be found at the intersection between these two FOV boundary lines. Thus, the generation of the virtual camera(s) is done by the proceeding method. The properties of each virtual camera can be determined either by calibration or by geometrical analysis of ray sketching, which are presented in next two sections. The basic requirements to build this system are: 1) the image plane of the CCD camera in use has consistent properties; 2) the 3F filter is exactly symmetrical with respect to all of its three apex edges and its center axis, which passes through the prism vertex and is normal with its back plane; 3) the back plane of the 3F filter is positioned in parallel with the real camera image plane, and; 4) the projection of the 3F filter vertex on the camera image plane is located at the camera principle point and the projection of one apex edge of the filter on the image plane bisects the camera image plane equally and vertically. With the above requirements satisfied, the camera optical axis will pass through the 3F filter vertex; the three virtual cameras will have identical properties and will be symmetrically located with respect to real camera optical axis. Thus the analysis of any one virtual camera would be sufficient as the results can be transposed to the other two virtual cameras. Now the three sub-regions of the image 54 plane (and also the three corresponding virtual cameras) can be differentiated by using label l, r and b which stand for left, right and bottom, as shown in Figure 5.1. l r b Figure 5.1 Positioning a 3F filter in front of a CCD camera Figure 5.2 One image captured by the single-lens trinocular system 55 5.1.1 Determining the Virtual Cameras by Calibration The calibration technique introduced in Chapter can also be used here to calibrate the virtual cameras, with slight modifications. Various coordinate systems can be created on the virtual cameras analogously, including the distorted virtual camera 2D image coordinate systems (Xd,l, Yd,l), (Xd,r, Yd,r) or (Xd,b, Yd,b), undistorted virtual camera 2D image coordinate systems (Xu,l, Yu,l), (Xu,r, Yu,r) or (Xu,b, Yu,b) and the Left Virtual Camera Coordinate System (LCCS), the Right Virtual Camera Coordinate System (RCCS), and the Bottom Virtual Camera Coordinate System (BCCS). (Xd,l, Yd,l), (Xd,r, Yd,r) and (Xd,b, Yd,b) can be linked to the computer image coordinates (Xf, Yf) via: X d ,l = (C x − X f ) • dx ' , Yd ,l = (C Y − Y f ) • dy ' ; X d , r = (C x − X f ) • dx' , Yd , r = (CY − Y f ) • dy ' ; Yd ,b = (C x − X f ) • dx ' , X d ,b = (C y − Y f ) • dy ' , (5.1) where dx′ and dy′ are the pixel size of the computer sampled images (images captured by computer and displayed on computer screen) and can be obtained by actual CCD pixel size times its resolution and then divided by the computer sampled image resolution in both x and y directions. becomes possible. Hence the calibration of virtual cameras Each virtual camera can be calibrated one-by-one using the information provided by its correspondent sub-image, from which the whole system can be determined. This system is now ready to perform depth recovery like a typical trinocular stereovision system using triangulation knowledge. From the coordinate setup for calibration the following equations can be obtained: 56 xl xw y l = Rl y w + Tl ; zl zw xb xw y b = Rb y w + Tb ; zb zw xr xw y r = R r y w + Tr , zr zw (5.2) where: rl1 rl rl Rl ≡ rl rl rl rl rl rl rb1 rb rb3 Rb ≡ rb rb rb5 rb8 rb rb rr1 rr rr Rr ≡ rr rr rr rr rr rr Tlx Tl = Tly , Tlz Tbx Tb = Tby , Tbz and, Trx Tr = Try . Trz The precondition for the proceeding equations to hold is that the world coordinate systems used in calibrating all the left, bottom and right virtual cameras must be the same coordinate system (same origin and orientation). From the calibration result, Rl, Tl, Rb, Tb, Rr and Tr can be obtained, and also fl, fb and fr. The details on the calibration procedure can be found in Tsai [37] and in Chapter of this thesis. It is also known that: 57 xl = X ul zl , fl yl = Yul zl ; fl xb = X ub zb , fb yb = Yub zb ; fb xr = X ur zr , fr yr = Yur zr . fr (5.3) Thus a set of linear equations can be obtained: X ul z l = rl1 x w + rl y w + rl z w + Tlx ; fl Yul z l = rl x w + rl y w + rl z w + Tly ; fl z l = rl x w + rl y w + rl z w + Tlz ; X ub z b = rb1 x w + rb y w + rb z w + Tbx ; fb Yub z b = rb x w + rb5 y w + rb z w + Tby ; fb z b = rb x w + rb8 y w + rb z w + Tbz ; X ur z r = rr1 x w + rr y w + rr z w + Trx ; fr Yur z r = rr x w + rr y w + rr z w + Try ; fr z r = rr x w + rr y w + rr z w + Trz . (5.4) The following equation can be obtained manipulating the proceeding equations: Ac = B (5.5) with: 58 A= rl1 rl rl rl rl rl rl rl rl X ul fl Yul − fl −1 rb1 rb rb3 rb rb5 rb rb rb8 rb9 rr1 rr rr rr c = [x w 0 X − ub fb Y − ub fb −1 rr rr rr rr rr yw [ − and B = − Tlx zw zl − Tly zb − Tlz 0 , X ur − fr Yur − fr −1 zr ] , T − Tbx − Tby − Tbz − Trx − Try ] − Trz . With the least square solution, c = ( AT A) −1 AT B (5.6) The redundant information obtained with three virtual cameras (as any two virtual cameras are enough for stereovision purpose) are handled by using the least square method, and the condition number appearing in calculating the matrix inverse is not a problem as shown by our calculation in the experiment. This is believed to be due to the fact that all the three virtual cameras are naturally symmetrically located (or in another word, evenly scattered) about the optical axis of the real camera and this situation leads to the possible maximum linear independence amongst the coordinate systems on the three virtual cameras that can be achieved in such a system design. (This explanation is equally valid in the calibration based approach and the single-lens multi-ocular stereovision system to be presented in the following sections and chapters). Now this system is ready for depth recovery. 59 As line KO″ is along the z axis of coordinate system C, it can be determined easily. Step B3: Find line O″J For this part, we draw an auxiliary line A″C″ passing through point O″ and parallel to line AC (see Figure 5.4). As z axis of the coordinate system C is normal to line AC, this z axis is also normal to A″C″. The angle between line KO and plane AO″C, say, φ, equals to (90°-∠O″DO), where point D is the mid-point of line AC. This angle has been determined in Appendix C of this thesis. Now we look at plane O″DO. Let NAO”C denote a normal to the plane AO″C passing through point O″. Line KO″, after refraction becomes line O″J, where point J is located on plane A′B′C′. According to the law of refraction: sin(90° − φ ) = nr , sin η where η is angle between line O″J and NAO”C, and can be found via, η = arcsin( sin(90° − φ ) ). nr Hence the direction of O″J is known. As point O″ is known, line O″J can be determined. Step B4: Find point J The angle between line O″J and the normal NA’B’C’ of plane A′B′C passing though point J is (90°-φ-η). From Figure 5.10, the following equation can be obtained: 78 b = tan(90° − φ − η ) . (h + t ) K NAO”C 90°-φ φ O″ R η 90-φ-η w J b χ NA’B’C’ S Figure 5.10 Plane KJS Hence b is b = tan(90° − φ − η ) × (h + t ) . (5.30) After knowing b, the position of point J with respect to the coordinate system C is known: x J = b × cos 30°, y J = −b × sin 30°, z J = d + h + t. (5.31) Step B5: Find line JS 79 Line O″J becomes line JS after refraction. To find the angle χ between line JS and NA’B’C’ which is a normal of plane A′B′C′ passing through point J: According to the law of refraction: sin( χ ) = nr . sin(90° − φ − η ) Hence χ can be determined through χ = arcsin(sin(90° − φ − η ) • nr ) . (5.32) Extending line JS in the reverse direction, it interests line O″O at point R. We can find the distance between point R and point O, denoted by w, through the following equation: w = b × tan(90° − χ ) . (5.33) After knowing w, the position of point R is also known, which is given by: x R = 0, y R = 0, z R = d + h + t − w. (5.34) After knowing point R, line RJ (also line JS) can be determined, which can be expressed in the following form: PRJ = PR + ( PJ − PR ) × k , (5.35) where PRJ represents any point on line RJ, and k is a variable corresponding to the chosen point on line RJ. 80 Line RJ (line JS) and line NL will help to form the triangle of virtual camera model. The intersection of lines RJ and NL, point F′ is the optical center of the virtual camera. Assuming that point K′ and point P′ are the corresponding points of point K and point P, respectively, and the distance F′K′ is taken to be the focal length of the virtual camera, with the knowledge that the virtual camera focal length can be considered identical with the real camera focal length (which has been proven in section 4.1.3, although the proof was used for the binocular system, but it can also be transposed to this trinocular system, as the principle of virtual camera generation is the same at this point), the virtual camera can be determined mathematically. As we have assumed the real camera is not calibrated, the exact value of the virtual camera focal length and the real camera focal length are not known. Hence the procedure of field point test described in section 4.1.3 is used with equation (5.46). Three virtual cameras are taken as identical and hence have the same focal length. The recovered focal length, which can be taken as a scale ratio between this system and the external world, completes the determination of the system. A problem of determining the exact position of the point F′ may be encountered in practice, or in more detail, it is about the position of F′ in the actual calculation. Theoretically line RJ and line NL should always intersect, however due to digitization errors, this may not be the case. Therefore an approximation of the coordinates of F′ may be necessary. The following method is about how to simulate and handle this situation. Two straight lines in 3D not intersect and are not parallel to each other have a unique shortest distance, which is probably the case that needs to be handled after getting the expression of line RJ and line NL. Figure 5.11 illustrates this 81 scenario, in which, two non-parallel and non-intersecting lines AB and CD are shown. The shortest distance between them is assumed to be given by EF. A D E F B C Figure 5.11 Illustration of the shortest segment connecting two non-intersecting, and nonparallel lines It is assumed that line AB and line CD not intersect and are also not parallel lines, and line EF is perpendicular to line AB and line CD and its length is the shortest distance between line AB and line CD. Line AB and line CD are represented by the following expressions: PAB = PA + K AB ( PB − PA ), PCD = PC + K CD ( PD − PC ), (5.36) where PAB and PCD are any points on line AB and line CD respectively, and KAB and KCD are corresponding parameters, the value of which depending on the chosen PAB and PCD respectively. Point E and point F can then be represented as: PE = PA + K AB ( PB − PA ), PF = PC + K CD ( PD − PC ). (5.37) As line EF is perpendicular to line AB and line CD, the following expression can be obtained: 82 ( PE − PF ) • ( PB − PA ) = 0, ( PE − PF ) • ( PD − PC ) = 0. (5.38) Replacing PE and PF in Equation (5.38) using the Equation (5.37): (( PA + K AB ( PB − PA )) − ( PC + K CD ( PD − PC ))) • ( PB − PA ) = 0, (( PA + K AB ( PB − PA )) − ( PC + K CD ( PD − PC ))) • ( PD − PC ) = 0. Solving the proceeding equations for the corresponding parameters KAB and KCD for point E on line AB and point F on line CD: K AB = K CD = M ACDC M DCBA − M ACBAM DCDC M BABA M DCDC − M DCBA , M ACDC + M DCBA • K AB , M DCDC (5.39) where M 1234 = ( x1 − x2 )( x3 − x4 ) + ( y1 − y )( y3 − y ) + ( z1 − z )( z − z ) . Once the corresponding parameters KAB and KCD for point E and F are found respectively, point E and F can be determined easily and the mid-point of segment EF is taken to be the lens center of virtual camera (i.e. point F′). Now we shall look at plane F′P′K′ (Figure 5.12, also refer to Figure 5.4). To determine point K′ which is the principle point of virtual camera, ∠P′F′K′ needs to be determined first. Let f′p′ denote the unit vector of line F′P′ and f′k′ denote the unit vector of line F′P′, respectively. As it is known that F ' K ' = f , hence point K′ can be found, which is given by PK ' = PF ' + f ' k ' F ' K ' . (5.40) 83 F′ P′ K′ Figure 5.12 Plane F′P′K′ Next point P′ needs to be found. Knowing angle ∠P′F′K′, the length of P′F′ can be determined: P' F ' = f . cos ∠P' F ' K ' Hence point P′ is given by: PP ' = PF ' + f ' p' P' F ' . (5.41) For better understanding the position relationship between the virtual camera and real camera, a coordinate system R, can be set on this virtual camera. Its origin is on point F′, its z axis zR is along line K′F′, its x axis xR is along line K′P′ and its y axis yR can be determined according to the right hand rule. With the reference to coordinate system C, its z axis zR can be given by the unit vector, k′f′, which is given by: xk ' f ' k ' f ' = yk ' f ' . zk ' f ' 84 The angle between zR and xC (x axis of coordinate system C) can be given by: ψ x = arccos( x k ' f ' ) . Similarly, the angle between zR and yC, zR and zC are given respectively by: ψ y = arccos( y k ' f ' ), ψ z = arccos( z k ' f ' ). The angle between k′f′ and yCzC plane (the smallest angle between any lines on yCzC plane and zR) can be determined via: ζ yz = arccos( y k2' f ' + z k2' f ' ). (5.42) Similarly, its angle with respect to xCzC plane and xCyC plane can be determined via: ζ xz = arccos( ζ xy = arccos( x k2' f ' + z k2' f ' x k2' f ' + y k2' f ' ), ). These angles are important to understand the position and orientation relationship between the real and virtual cameras. Another two important parameters that can be used to describe the relative position and orientation among different virtual cameras are λ (the distance between any two virtual camera optical centers) and γ (the angle between any two virtual camera optical axes). As the position and orientation of each virtual camera are known, determining these two parameters is straightforward. After knowing the relative position and orientation between the virtual cameras and the real camera, the position and orientation between virtual camera and the external world are easy to be determined if the world coordinate system is chosen 85 to be known with respect to the real camera coordinate located at its optical center. Hence the following equations can be obtained: xw xl y w = w Rl y l + wTl ; zw zl xw xb y w = Rb y b + wTb ; w zw zb xw xr y w = Rr y r + wTr , w zw zr (5.43) w where Rl ≡ w w w rl1 w rl rl w w w Rb ≡ w w w rl w rl rl w rb1 w rb rb w w w rl rl rl rb w rb rb8 w w w Tl = w Tly , Tlz w rb3 rb rb9 Tlx w Tb = w Tbx w Tby , Tbz w and, w w Rr ≡ rr1 w rr w rr w w rr w rr w rr w w rr rr w rr w Tr = Trx w Try . w Trz Obviously the same world coordinates (same origin and orientation) is used for each virtual camera coordinates to refer to. From the geometrical analysis wRl, wTl, w Rb, wTb, wRr and wTr can be obtained. Also the focal lengths for each virtual camera fl, fb and fr are taken as equal and can be obtained from a field point test as described previously in this section. 86 According to the relationship between undistorted 2D image coordinate system and 3D virtual camera coordinate system, the following equations can be obtained: xl = − X ul zl , fl xb = − X ub zb , fb yb = − Yub zb ; fb xr = − X ur zr , fr yr = − Yur zr . fr yl = − Yul zl ; fl (5.44) Please note the difference on the sign here with respect to that of calibration based approach. Thus a set of linear equations can be obtained: x w =− w rl1 X ul Y z l − w rl ul z l + w rl z l + wTlx , fl fl y w =− w rl X ul Y z l − w rl ul z l + w rl z l + wTly , fl fl z w =− w rl X ul Y z l − w rl ul z l + w rl z l + w Tlz , fl fl x w =− w rb1 X ub Y z b − w rb ub z b + w rb z b + w Tbx , fb fb y w =− w rb X ub Y z b − w rb5 ub z b + w rb z b + w Tby , fb fb z w =− w rb X ub Y z b − w rb8 ub z b + w rb9 z b + w Tbz , fb fb x w =− w rr1 X ur Y z r − w rr ur z r + w rr z r + wTrx , fr fr y w =− w rr X ur Y z r − w rr ur z r + w rr z r + wTry , fr fr z w =− w rr X ur Y z r − w rr ur z r + w rr z r + w Trz . fr fr (5.45) After manipulating the proceeding equation, we obtain: 87 A' c' = B ' , (5.46) where 0 0 X ul w Yul w + rl − rl fl fl X Y w rl ul + w rl ul − w rl fl fl X Y w rl ul + w rl ul − w rl fl fl w rl1 X ub w Yub w + rb − rb3 fb fb X Y w rb ub + w rb5 ub − w rb fb fb X Y w rb ub + w rb8 ub − w rb fb fb w 0 A' = 0 0 0 c' = [x w yw zw and B' = [T w w lx zl Tly zb w Tlz rb1 0 X ur w Yur w + rr − rr fr fr X Y w rr ur + w rr ur − w rr fr fr X Y w rr ur + w rr ur − w rr fr fr w rr1 zr ] , T w Tbx w Tby w Tbz w Trx w Try w ] Trz . The least square solution is: c' = ( A'T A' ) −1 A'T B ' . (5.47) All the elements in A′ and B′ can be known either from calculation or reading from the image captured. Once zl (also zb and zr) are found, the distance Zl (also Zb and Zr) between the real camera optical center and the point of interest can be determined, and the average of Zl, Zb and Zr are used. The condition number of ( A'T A' ) (which includes the 88 relationship between the coordinate of the three virtual cameras) is not a problem when finding its inverse, which has been shown in our experiments and has been explained in section 5.1.1. In this section, we have discussed following questions: 1) how to determine the virtual cameras using geometrical analysis of ray sketching; 2) how to a field point testing to find focal lengths of the real camera and virtual cameras; 3) how to determine focal point F′, and; 4) how to utilize the stereo information for depth recovery. 5.2 Experiment and Discussion Similar experimentation technique and devices used for single-lens binocular system (described in section 4.2) are adopted for this single-lens trinocular system with necessary modifications to test both approaches used to model this virtual stereovision system. The main 3F filter used has a diameter of 74.6mm, and its l is 65mm, a is 37.8mm, t is 6.8mm and h is 4mm (see Figure A. 7). experimentation can still be divided into three main steps. The The first step is to calibrate the real camera in order to determine its properties. The second step is to model and determine the virtual cameras either via calibration or via the geometrical analysis based approach which includes one field point testing. The third step is the depth recovery test. One image captured during calibration of virtual camera is shown in Figure 5.13, and another image captured during the depth recovery is shown in Figure 5.2. In Figure 5.13 the black circles appear blurred because the need of compromising the requirement on the depth recovery (requires the focus at the 89 smaller distance due to the small system working distance) and the requirement on the calibration (requires the focus at the larger distance for capturing more calibration patters). This explanation also applies to the calibration work for the work in next Chapter. In our experiments, the correspondence searching ends at pixel level and does not go into sub-pixels. The redundancy caused by the extra virtual camera during depth recovery (as any two virtual cameras would be enough for stereo) is handled by the least square method. Table 5.1 gives details on the depth recovery result (λ=40mm). For the depth ranged from 0.9m to 1.5m the geometrical analysis based approach can give an absolute depth recovery error of about 1% in average using a typical setup in the experiment (see equations (5.46) and (5.47)), while the calibration based approach can give an error of about 3% under the same condition (see equations (5.5) and (5.6)), according to which it is believed that both approaches are capable to model and determine this system. A comparison is made here with the results from a perfect binocular stereovision system: it is shown that under similar situation according to equation (4.29) a perfect binocular stereovision system will give a depth error of about 1% for a correspondence error of one pixel size if the searching line (epipolar line) is parallel with the CCD pixel rows or columns; and it will give a depth error of about 1.4% for a correspondence error of one pixel size if the searching line has 45° angle with respect to CCD pixel rows (or columns), with the assumption that CCD pixels are squares and has identical size. The main constraint that is encountered in our experiments is from the 3F filter that can be immediately found. The prism can only give a baseline value of about 40 to 60mm in the experiment, which obviously 90 limited the precision of stereovision, in particular, the depth recovery accuracy. And the view zone of each virtual camera is also limited, which affects the accuracy of calibration and hence the accuracy of the calibration based modeling. Efforts are now spent to acquire filters with larger size and better shape which can give larger baselines and common view zones for further testing. Figure 5.13 Calibration of virtual cameras Nevertheless, besides the advantage of the typical epipolar property of the trinocular stereovision system for correspondence searching, which has been reviewed in Appendices B, one property worth mentioning is that the image planes of these three virtual cameras can be approximately taken as co-planar as shown by both our analysis and experimentation. As a result, the epipolar lines of the system are approximately parallel to one of the lines joining the optical centres of any two virtual camera or perpendicular to the projections of the filter apex edges on camera 91 image plane. The epipolar lines of the correspondence triplets will form equilateral triangles, as shown in Figure 5.2. This fact obviously greatly facilitates the correspondence searching of this stereovision system besides the well known epipolar properties of a typical trinocular stereovision system. 5.3 Summary This chapter presented a single-lens trinocular stereovision system with a 3F filter which combines the advantages the single-lens stereovision and trinocular stereovision. One image acquired by this system can be split into three sub-images and these sub-images can be taken as the images simultaneously captured by three virtual cameras. Two different approaches were presented to determine the system: one based on calibration technique and another based on geometrical analysis of ray sketching. The latter method does not require complex calibration and has a much more simplified implementation process. Experiments showed the feasibility of both of these approaches. Comparing to the single-lens binocular system reported in CHAPTER 2, the modeling of this system is more complex, but it offers one significant advantage on the stereo correspondence checking as demonstrated in our experiment. Theoretically the redundant stereo information captured by this system should lead to better depth recovery accuracy. However, the view zone of each of the virtual cameras is reduced since the three virtual cameras now share the same CCD, as compared to two in the case of a single-lens binocular system reported in CHAPTER 4. A comparison between the binocular and the trinocular system would have been ideal to demonstrate the advantages of one system over the other. However, a fair comparison cannot be made due to the limitation in the hardware setup available in the laboratory. 92 The two setups give different working distance and view scopes thus affect the accuracy of the camera calibration. This is the results of the geometry of the different prisms used in this project. We would recommend that better hardware should be acquired for further study. Actual Depth (mm) 900 1000 1100 1200 1300 1400 1500 AVG (32,32) (276,453) (524,34) (136,93) (376,514) (625,93) Recovered Depth (mm, Cali based Approach) 907.6 843.4 0.84 6.29 Recovered Depth (mm, Geo Analysis based Approach) 875.7 874.4 (47,83) (291,505) (537,84) (45,42) (296,476) (550,43) 885 1009 1.67 0.90 877.9 978 2.46 2.20 (138,97) (386,531) (642,95) (59,87) (309,521) (563,89) 950.5 983.8 4.95 1.62 984.4 974.1 1.56 2.59 (50,49) (307,494) (566,51) (136,100) (390,544) (651,100) 1111.4 1045.5 1.04 4.95 1078.9 1077.5 1.92 2.05 (63,91) (318,536) (579,92) (99,43) (359,496) (626,44) 1099.6 1199.1 0.04 0.08 1083.6 1188.6 1.49 0.95 (181,84) (440,536) (707,82) (115,82) (374,534) (640,81) 1115.7 1167.9 7.03 2.67 1187.3 1179.7 1.06 1.69 (88,39) (353,500) (624,40) (165,74) (428,534) (700,73) 1321 1237.3 1.62 4.82 1303.8 1298.1 0.29 0.15 (104,74) (368,534) (638,74) (78,45) (345,512) (621,46) 1288.3 1419 0.90 1.36 1289.2 1406.9 0.83 0.49 (150,78) (417,544) (692,77) (92,78) (361,544) (635,77) 1347 1416.7 3.79 1.19 1402.4 1411.6 0.17 0.83 (82,51) (354,523) (632,51) (149,81) (419,553) (698,80) 1529.7 1450.7 1.98 3.29 1525.9 1523 1.73 1.53 (96,81) (366,553) (644,81) 1497.9 0.14 2.44 1509.4 0.63 1.44 Correspondence Pixel Triplet (in the order of left, bottom and right subsections of computer screen) Absolute Error in Percentage (%) Absolute Error in Percentage (%) 2.70 2.84 Table 5.1 Recovered depth by trinocular stereovision, λ = 40mm 93 [...]... Figure 5. 4, line P′F′ and line K′F′are actually line NL and line JS Thus these procedures can be separated into two main paths as illustrated by Figure 5. 5: to find line NL (Flow A in Figure 5. 5, denoted by red lines in Figure 5. 4) and, and to find line JS (Flow B in Figure 5. 5, denoted by blue lines in Figure 5. 4) These two flows can be further separated into more sub-steps as illustrated in Figure 5. 5... Figure 5. 4 Symbolic Illustration of virtual camera modeling using geometrical analysis 5. 1.2.2 Detailed description This section describes the complete idea of modeling the virtual cameras using geometrical analysis method based on the introduction presented in the previous section with emphasis on two problems for the geometrical analysis based 65 approach to determine single- lens trinocular stereovision: ... the origin of coordinate system C: zG = d + h + t (5. 25) As line GH is perpendicular to line MN, the following equation can be obtained: 74 GH • MN = 0 (5. 26) As line MN is known, given any arbitrary value for xG (the x coordinate value of point G), the corresponding value of yG (the y coordinate value of point G) can be determined by using equation (5. 26) Thus line GN (that is, line GH) can be determined... ω ) cos(γ − ω ) 0 0 0 0 1 (5. 29) G L zR (γ- ω) N yR 1 zR1 x R, x R1 yR H M Figure 5. 9 Temporary coordinate system R and R′ used in finding line NL Using equation (5. 28), C TR1 can be obtained, from which the information of zR1 can be extracted Hence line NK″ can also be determined Path B – Solve For Line JS Step B1: Find point O″ Point O″has been determined in equation (5. 11) Step B2: Find line KO″... (90°-φ-η) From Figure 5. 10, the following equation can be obtained: 78 b = tan(90° − φ − η ) (h + t ) K NAO”C 90°-φ φ O″ R η 90-φ-η w J b χ NA’B’C’ S Figure 5. 10 Plane KJS Hence b is b = tan(90° − φ − η ) × (h + t ) (5. 30) After knowing b, the position of point J with respect to the coordinate system C is known: x J = b × cos 30°, y J = −b × sin 30°, z J = d + h + t (5. 31) Step B5: Find line JS 79 Line... organize the triangulation information For example, one method is to find depth information using any two virtual cameras and take the average of the three results obtained from three combinations of the virtual cameras However, organizing all the triangulation information using one linear system (equation (5. 5)), which is more systematic, is preferred here It is well known that camera calibration is... determined Hence γ can be determined through the following equations: γ = arcsin(sin ω × nr ) (5. 27) 75 NA’B’C’ L γ N plane A′B′C′ ω M Figure 5. 8 Plane LNM To find line NL, line MN is rotated by an angle of (γ-ω) about line GH as the rotating axis A temporary coordinate system R (xR, yR, zR) is created (see Figure 5. 9) at point N, let its z axis zR to be along line MN, x axis xR along line GH, and y axis... 30°, 2 h y p = − sin 30°, 2 zp = − f xP = (5. 14) Since the focal point is the origin of the 3D camera coordinate, this point is: x F = 0, y F = 0, z F = 0 (5. 15) Point M is the intersection of line PF and plane AO″C, and hence point M is on the following line: M = P + ( F − P) A × xP + B × y P + C × z P A( x P − x F ) + B( y P − y F ) + C ( z P − z F ) (5. 16) Step A2: Find line PM After obtaining... and M is given by: PM = ( x p − x m ) 2 + ( y p − y m ) 2 + ( z p − z m ) 2 (5. 17) The distance between point P and plane AO″C can be found according to the following equation: P − AO" C = Ax p + By p + Cz p − 1 ( A2 + B 2 + C 2 ) (5. 18) Hence, ρ = arcsin( p − AO" C PM ) (5. 19) An auxiliary line UV (green line in Figure 5. 4) is created, where point U is on line O″A, and point V is on line O″C, UV... ) (5. 37) As line EF is perpendicular to line AB and line CD, the following expression can be obtained: 82 ( PE − PF ) • ( PB − PA ) = 0, ( PE − PF ) • ( PD − PC ) = 0 (5. 38) Replacing PE and PF in Equation (5. 38) using the Equation (5. 37): (( PA + K AB ( PB − PA )) − ( PC + K CD ( PD − PC ))) • ( PB − PA ) = 0, (( PA + K AB ( PB − PA )) − ( PC + K CD ( PD − PC ))) • ( PD − PC ) = 0 Solving the proceeding . 50 CHAPTER 5. SINGLE- LENS TRINOCULAR STEREO- VISION In this chapter, we present a novel design for stereovision - a 3F filter (tri- prism) based single- lens trinocular stereovision. works on trinocular stereovision is [ 45] - [52 ]. Chiou et al. [ 45] discussed the optimal camera geometry of trinocular stereovision with regards to system performance; Agrawal and Davis [52 ] studied. camera synchronization. Developing a single- lens trinocular stereovision system may help to solve these problems, but very few works on single- lens trinocular stereovision systems that can perform