Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 729 Mã bài: 157 Hệ thống tạo ảnh toàn nét và ứng dụng thời gian thực trong các hệ robot cấp độ micro All-In-Focus imaging and real-time microrobotic applications Nguyễn Chánh Nghiệm Trường ĐH Cần Thơ, e-Mail: ncnghiem@ctu.edu.vn Văn Phạm Đan Thủy Trường ĐH Cần Thơ, e-Mail: vpdthuy@ctu.edu.vn Kenichi Ohara and Tatsuo Arai Osaka University Tóm tắt Trong khoa học sự sống, việc quan sát và thao tác các vật thể vi sinh diễn ra rất thường xuyên và mang tính lập lại trong đó việc điều chỉnh lấy nét là một yêu cầu tiên quyết. Nhiều giải thuật lấy nét tự động đã được đề xuất để giúp thao tác viên giảm thiểu thời gian điều chỉnh lấy nét. Những giải thuật này cũng có thể được áp dụng để tự động hóa các khâu vi cảm biến hay thao tác các vi vật thể như đo độ cứng của tế bào, gắp thả, hay giữ cố định các vật thể di động. Bài nghiên cứu này đề xuất ứng dụng giải thuật tạo ảnh toàn nét để giúp tự động hóa thao tác các vi vật thể trong khi có thể quan sát chúng được rõ nét trong thời gian thực. Thí nghiệm gắp thả các vi vật thể với kích thước khác nhau được thực hiện để kiểm tra tính khả dụng của một hệ vi thao tác tự động thời gian thực. Abstract: In life sciences, observing and manipulating various microbiological objects may be performed frequently and repeatedly in which object focusing is the preliminary task of the operator. In order to reduce the manual focusing time, various autofocus algorithms have been proposed. These algorithms can also be implemented to automate microsensing and micromanipulation tasks such as measurement of cell stiffness, pick-and-place of various microobjects, immobilization of moving objects, etc. This paper proposes the All-In-Focus algorithm to automate micromanipulation of microobjects while they can be observed clearly in real-time. Pick-and- place of single microobjects with different sizes is performed to demonstrate the effectiveness of a real-time micromanipulation system. Chữ viết tắt IQM Image Quality Measure AIF All-In-Focus LTPM Line-Type Pattern Matching 1. Introduction Focusing a target microobject is a frequent and preliminary task in observing the microobject and further manipulating it. The difficulty of this manual task depends on the size of the target object. A small microobject requires larger magnification lens with a narrower depth of field. A thick microobject thus requires longer manual focus adjustment. The transparency of most microbiological objects, in addition, contributes more difficulties for precise focusing. In order to reduce the operator time in manual focusing of microobjects, various autofocus algorithms have been proposed. An introduction and comparison of various autofocus algorithms ranging from the well-known to the most recently proposed algorithms can be found in [1]-[3]. Based on the choice of evaluation criteria for the best-focused position, these algorithms are classified into four categories, i.e., derivative-based, statistic-based, histogram-based, and intuitive-based algorithms [2]. Using the Image Quality Measure (IQM) to detect an in-focus area in an image, a Micro VR camera system had been developed to provide real-time all-in-focus image which is a composite image created by merging all in-focus areas from various images of the observed object taken at different 730 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai VCM2012 focal distances [4]. This algorithm can thus be called All-In-Focus (AIF) algorithm and is classified into derivative-based category. The system also provides a depth image in real time so that 3D positions of microobjects can be obtained to facilitate automated micromanipulation, e.g., automated grasping and transporting an 8 μm microsphere [5]. The real-time micro VR camera system estimates the depth from in-focus pixels extracted from a series of images taken along z-direction. It is, therefore, independent on the shape of the object. There are, however, a few problems towards obtaining accurate 3D information from this imaging system. For example, there is a trade-off between the frame rate and the accuracy of the system. In order to achieve real-time detection, fewer images are used to create the AIF image which increases the resolution error. To capture images at different focal position, an actuator is used to move the lens in the optical axis. Vibration from the actuator may also reduce the quality of the AIF image and contribute noise to the system. Thus, the error in depth information of a transparent object in fast motion can be significant. Fig. 1 System overview By integrating a micromanipulation system and utilizing the depth information obtained from the system to find the 3D position of both the end- effector of the micromanipulator and the target object, it is possible to develop an automated micromanipulation system. This paper proposes an automated micromanipulation system that uses a two-fingered microhand as the micromanipulator because it is capable of dexterous micromanipulation such as cell rotation [7], and measurement of mechanical properties of a living cell [8, 9]. To solve the inherent problems of real-time AIF imaging, this paper proposes Line-Type Pattern Matching and Contour-Depth Averaging to measure 3D positions of a micromanipulator's tip and a target micro transparent object, respectively. The effectiveness of the proposed methods is experimentally demonstrated with the pick-and- place of single microobjects with different sizes. The proposed method can be applied to find the 3D positions of transparent end-effector tips of common microtools, as well as glass micropipettes, and other micro biological cells. This helps the All-In-Focus imaging system a versatile 3D imaging system that can be integrated into a micromanipulation system to provides not only real-time extended depth of field with the AIF image but also the 3D positions of transparent microobjects to handle them automatically. Fig. 2 Illustration of All-In-Focus algorithm 2. System overview 2.1 All-In-Focus imaging system The All-In-Focus imaging system is developed based on the Micro VR camera system [4] and consists of a piezo actuator and its controller, a processing unit to create the AIF and HEIGHT image, and a high-speed camera attached to the camera port of the microscope (Fig. 1). The piezo actuator can move the objective lens cyclically up and down over a SWING distance up to 100 µm along the optical z-axis. When the system is running, the high-speed camera (Photron Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 731 Mã bài: 157 Focuscope FV-100C) captures images at different focal planes at the rate of 1000 frames per second. As the lens traverses a cyclic SWING distance, the focal plane changes and a stack of images at consecutive focal planes is collected. These images in the stack all have the same number of pixels. The best focal distance for each pixel location is obtained by evaluating the local frequency of image intensities around that pixel location in all images in the image stack [10]. Thus, the AIF image is created by combining all best-focused pixels from the image stack. Fig. 2 illustrates the AIF imaging algorithm and the AIF image of a protein crystal. The best focal distance at each pixel location is normalized to a pixel value at that pixel location in the HEIGHT image (Fig. 2). Therefore, the AIF image provides good visualization of microobjects (Fig. 3a) while the HEIGHT image provides their positions (Fig. 3b) in the z-axis. (a) (b) Fig. 3 AIF image (a) and HEIGHT image (b) of protein crystal Fig. 4 The world coordinate system The world coordinate system is shown in Fig. 4. The Z-axis of the world coordinates is parallel to the optical axis of the microscope. The ( , ) X Y plane lies on the object plane and its X-axis and Y- axis align with the horizontal x-axis and vertical y- axis of the AIF image, respectively. The relationship between the distance in ( , ) X Y plane and in the number of pixels of the AIF image is obtained by measuring the pixel size of an AIF image of a scalar. Let {20,40,60,80,100} SWING  be the distance over which the piezo actuator moves objective lens. This distance is normalized into a gray scale from 0 to 255 in the HEIGHT image. Therefore, the z- coordinate of a pixel at position ( , ) x y can be estimated from the corresponding pixel value ( , ) H x y in the HEIGHT image as     , * μm 256 H x y Height SWING (1) The distance between two consecutive focal planes which is also the resolution of the AIF imaging system can be calculated as   μm 30* SWING d FRAME   (2) where   1,2,4,6 FRAME  determines the frequency of scanning or the frame rate of the AIF imaging system as   30 _ frames per second frame rate FRAME  (3) Fig. 5 Two-fingered microhand for dexterous micromanipulation applications The highest and lowest frame rate of the AIF imaging system is 30 and 5 frames per second, respectively (Eq. 3). With the lowest frame rate when 6 FRAME  and with 20 SWING  (μm) the best resolution of the system becomes 0.1 d   (μm). It should be noted that the higher the frame rate, the more vibration is introduced to the system since the objective lens moves faster in a cyclic up-and-down motion. 2.2 Two-fingered microhand Glass end-effectors are generally more preferable for biological applications because of its biocompatibility. In this study, a two-fingered microhand [6] that is mounted on the stage of the inverted microscope (Fig. 5) is used as the manipulator of the 732 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai VCM2012 micromanipulation system. The microhand has two microfingers that are fabricated by pulling glass rods or capillary tubes. In addition, it is a potential microtool with dexterous micromanipulability for potential biological applications. One of the two microfingers of this microhand is controlled by a 3-DOF parallel link mechanism. The parallel link mechanism and the other microfinger are mounted on a three-dimensional motorized stage to provide the global motion of the microhand in a large workspace. Dexterous manipulation is realized by the microfinger which is controlled by the parallel link mechanism. This configuration enables manipulation of multisized microobjects in a large workspace. 3. Measuring microobject position in 3D 3.1 Measuring 3D positions of end-effectors Having an elongated shape, a few lines can be detected along the microfinger in its AIF image. The 2D position of the fingertip can be thus obtained from these detected lines. The z-position of the fingertip is estimated from the HEIGHT image using the information of the detected lines. The process is as follows. (a) (b) Fig. 6 (a) Microfingers and 55 μm microsphere. (b) Detected lines superimposed on detected microfingers Fig. 7 Line grouping using middle position of lower endpoints of detected lines in x- direction 3.1.1 Line detection The two microfingers are set in the vertical direction and inclined toward each other (Fig. 6). Due to the shallow depth of field, only part of the microfinger can be in focus. The curvature of the surface of the microfinger functions as the surface of a lens. Therefore, the middle region of this local area will be brighter when it is in focus. This phenomenon was shown in a relevant section and figure in [11]. The AIF imaging system merges all in-focus parts of the object; it thus creates an image of a microfinger with the brighter region inside. As a result, there exist three regions with different intensity levels for each microfinger in the AIF image among which the middle region is the brightest (Fig. 6a). Merging all in-focus regions along the elongated microfinger, four lines are ideally detected in the AIF image for each microfinger by split and merge algorithm [12]. A threshold is set for the length of a detected line to eliminate false lines that may result from the ghost of a microfinger in its AIF image especially when it is moving. The four detected lines for a microfinger characterize a microfinger in the AIF image. Two of these are located at the borders of the microfinger; they are thus termed border lines. The other two lines which are in between the border lines are termed inner lines. 3.1.2 Microfinger classification Since there are two microfingers in the AIF image, it is necessary to classify the detected lines in the A I F . The x-coordinates of the lower endpoints of all detected lines are compared to their average value x_midpoint as shown in Fig. 7. A detected line is classified as left-microfinger group if its lower endpoint’s x-coordinate is smaller than x_midpoint; otherwise, it belongs to the right- m i c r o f i n g e r g r o u p . 3.1.3 Line-type pattern matching for fingertip identification in 2D The AIF imaging system needs at least 30 images to create the AIF image in real-time at 30 frames per second. The system can provide good AIF observation of the microobject even when it is moving. However, line detection for identifying two microfingers of the microhand becomes more difficult if it moves in high-speed. The edges along the microfinger may form broken line segments due to the limited processing speed of the AIF imaging system hardware. Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 733 Mã bài: 157 Because the microhand is set in a vertical direction in the image and three regions with different intensity levels are observed for each microfinger in the AIF image, the image intensity can change either “from bright to dark” or “from dark to bright” when going across a detected line from left to right. This detected line is defined to be type “0” and type “1”, respectively. Let L 1 , L 2 , L 3 , L 4 be the four detected lines for a microfinger in order from left to right. The line-type pattern in case of four lines correctly found from a microfinger is shown in Table 1. This holds true because the microfinger is darker than the image background and the middle region is the brightest among the three image region of the microfinger. Table 2 shows the line-type patterns of three lines inferred from that of the four-line case when a certain line L i cannot be detected. By matching with these patterns, the line-type pattern of three detected lines can also be used to identify a microfinger. Table 1 Ideal line-type pattern of 4 detected lines Line L 1 L 2 L 3 L 4 Line type 0 1 0 1 Table 2 Line-type patterns of 3 detected lines Missed line Line type l 1 l 2 l 3 L 1 1 0 1 L 2 0 0 1 L 3 0 1 1 L 4 0 1 0 (a) (b) Fig. 8 (a) Detected lines from the microfingers. (b) Fingertip positions when microhand was moving at 100 μm/s 115 255   y x H , 0   yx, 255   y x H , 0   tiptip yx ,   tiptip yx ,   yx, fitted line 90 (a) (b) Fig. 9 Pixel values from HEIGHT image along inner line on left microfinger (a) and right microfinger (b) at initial setup. Fitted line is calculated from 80 points It is also possible that a line-type pattern of four detected lines does not match with that in Table 1. This can happen when the microhand is moving in fast motion so that the two broken lines can be found on the finger border (right finger in Fig. 8a). In addition, a line can also be found from the ghost of the microfinger border (left finger in Fig. 8b) due to limitations of the AIF processing speed of the hardware. In these cases, the line-type pattern of a set of three neighboring lines from the four detected lines can give a correct match as shown in Fig. 8. When the actual existence of the microfinger is validated from the detected lines by Line-Type Pattern Matching, the 2D position of the fingertip can be accurately found from these lines. Because the microfinger tip is quite sharp, the y-coordinate of a microfinger tip can be set the same as the y- coordinate of the topmost endpoint of all the lines detected from that microfinger. With the y- coordinate known, the x-coordinate of the tip is computed from the equation of either inner line L 2 or L 3 . 3.1.4 Inclination measurement and depth estimation of the end-effector Depth estimation of the end-effector means finding the position of the microfinger tip in z-axis. The z-position of the microfinger tip found at location ( , ) x y tip tip in the AIF image can be directly estimated from the gray value ( , ) H x y tip tip of the pixel at location ( , ) x y tip tip in the HEIGHT image using Eq. 1. However, the HEIGHT image is very noisy. Therefore, more information is required to obtain accurate z- position of the tip. In this paper, the angle of 734 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai VCM2012 inclination of the microfinger is utilized to obtain accurate depth information of the fingertip. Given the positions of the pixels which lie on a line detected from the microfinger in the AIF image, the pixel values in the HEIGHT image at these positions are collected. A line is fitted from the values of 80 pixels along the tip’s part of the detected line. The angle of inclination of the fitted line estimates the inclination angle of the microfinger to the object plane. Figure 9 shows the values of the HEIGHT image’s pixels along the inner lines of the left microfinger and the right microfinger. Because of the limited SWING range of the AIF imaging system, only the upper part of the detected line in the AIF image (the tip’s part) i s u s e d i n t h i s f i t t i n g p r o c e s s . The z-coordinate of the fingertip is estimated from the fitted line at ( , ) x y tip tip rather than the single pixel value ( , ) H x y tip tip in the HEIGHT image. In Fig. 9, the ordinate of the rightmost point on the fitted line at ( , ) x y tip tip relates with the z- coordinate or z-position of the tip of the microfinger according to Eq. 1. In this sense, the inclination of the microfinger is utilized to eliminate noise in the HEIGHT image to estimate accurate depth information of its tip. The inclination angle of the microfinger can also be useful information when oriented micromanipulation is required although the inclination angle is not controlled in the current microhand system. The inclination angle and depth information can be obtained from either the border lines or the inner lines. However, it is observed that the inner lines are clearer and less broken especially when the microfinger is in fast motion. For this reason, the inner lines of a microfinger are used to estimate its tip’s position in z-axis. If two inner lines can be found for a microfinger after Line-Type Pattern Matching, the z-position of the fingertip is estimated from the fitted line with the smaller regression error. Since microfingers and micropipettes can be fabricated similarly by pulling a glass rod or tube, they may have similar elongated shapes. Thus, the proposed method can also be applied to measure the 3D position of a micropipette. However, a micropipette may have less-invasive rounded shape. Therefore, the method should be modified to identify the position of the tip in the 2D AIF image. Unlike the tip of a sharp microfinger, the x-coordinate of the rounded tip of a micropipette (pointing in y-direction) should be determined as the average of the x-coordinates of the upper endpoints of the detected lines on the micropipette. 3.2 Measuring 3D positions of target objects The AIF imaging system can also be used to find the 3D position of micro transparent objects. Unlike the tip of a microfinger or a sharp end- effector whose position can be characterized by a single point in 3D space, the 3D boundary of a microobject characterizes its 3D position. Under optical microscopes, it is difficult to reconstruct 3D model of a micro- transparent object. Thus, the contour of the object and its centroid in the AIF image provide its 2D position. The z-coordinate of the object can be considered as its centroid position in z-axis. Assuming that the object is round-shaped and suspended on the glass plate, the contour of the object on the plane that passes through the object’s center and is perpendicular to the z-axis can be considered as the outermost contour in the 2D AIF image. Using this assumption, Contour-Depth Averaging is proposed to estimate the z-position of the object as (μ m) 1 ( , ) * ( , ) 256 C H x y Height SWING x y C n    (4) where C is the contour or the boundary of the object in the AIF image and C n is the number of pixel points on the contour C . In this paper, a glass microsphere is used as the target object. The microsphere is transparent and qualifies our assumption. Thus, its 2D contour in the AIF image is detected as a circle using Hough gradient algorithm [13]. 4. Experimental methods The performance of the AIF system depends on the parameter SWING and . FRAME Adjusting parameter FRAME is a trade-off between the resolution (Eq. 2) and the frame rate of AIF imaging (Eq. 3). The resolution of AIF imaging is also determined by changing the scanning range SWING of the AIF imaging system (Eq. 2). Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 735 Mã bài: 157 In the experiment, the values of these parameters are: 80 SWING  μm, 2 FRAME  . These settings are to achieve adequate resolution of AIF imaging 1.3 d   μm for objects with different sizes in the scanning range of 80 μm. However, frame rate of AIF imaging is reduced to 15 frames per second. 0 20 40 60 80 100 120 140 1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 190 199 208 217 226 235 244 253 Frequency Pixel Gray Value Fig. 10 Intensity histogram of pixels on the circle around a microsphere in HEIGHT image The AIF imaging system is integrated into an Olympus IX81 inverted microscope under transmitted light bright-field observation mode. An Olympus LUCPlan-FLN 20X/0.45na Ph1 objective lens is used to achieve comfortable visualization of microobjects which are of different sizes in the desired range from 10 μm to 100 μm. 4.1 Accuracy assessment of depth measurement In order to evaluate the effectiveness of the AIF imaging system, it is necessary to assess the accuracy of depth estimation or measurement of z- positions of both the end-effector tip and the target object. 4.1.1 Depth measurement of the target object Figure 10 shows the histogram of the gray values of the pixels on the circular contour around a 55 μm microsphere in the HEIGHT image. Most of the pixels (88%) have the gray value of 119 and 127. The standard deviation of these pixel values is about 4.0. This corresponds to about 1.24 μm which is about the same as the resolution of the AIF imaging system at the chosen settings. Therefore, the average gray value of all the pixels along the detected circle in the HEIGHT image can be used to find the z-coordinate of the center of that microsphere using Eq. 4. In order to evaluate the linearity against z-position of the object, a microsphere was moved 60 μm in z-direction with a step-distance of 2 μm. The plot of measured z-position of the microsphere versus its displacement is shown in Fig. 11. A high linearity can be observed from the dotted trend line. 4.1.2 Depth measurement of the microhand A linear displacement of 30 μm in z-direction was sent to the microhand and the measured z-position of the moving microhand is shown in Fig. 12. Good linearity of the measured data can also be observed from the trend lines. 15 25 35 45 55 65 75 85 0 10 20 30 40 50 60 70 Measured z-position (micrometer) Displacement in z-direction (micrometer) Fig. 11 Measured z-position of a microsphere 0 10 20 30 40 50 60 70 0 5 10 15 20 25 30 35 Measured z-position (micrometer) Displacement in z-direction (micrometer) f1 f2 Linear (f1) Linear (f2) Fig. 12 Measured z-position of left microfinger f1 and right microfinger f2 4.3 Pick-and-place of different-sized microspheres As an application of the AIF imaging system, pick-and-place task was performed to single microspheres by using a two-fingered microhand [6]. The microspheres are suspended in the water on a glass plate to resemble biological cells in their culture medium. The 3D positions of the two microfingers of the microhand and of a microsphere estimated from the AIF imaging system helped automate the pick-and-place task. Because the microhand was developed to have a multi-scale manipulability, microspheres of 96 μm, 55 μm, and 20 μm in diameter were used. This is 736 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai VCM2012 also the size range of our currently interested objects; for example, lung epithelial cells whose stiffness was measured [8] were about 20 μm in diameter. In this experiment, the microhand is placed over 100 μm from a target microsphere in the 2D object plane. It is manually brought to about the same z- level of the microsphere and coarsely focused so that both the microhand and the target object are within the scanning range of the AIF imaging system. After this initial setup (Fig. 13a), the position of the two fingertips are calculated and the automatic z-alignment is performed by moving the right microfinger to the z-level of the left microfinger (Fig. 13b). A cycle of pick-and-place task is then performed for the target microsphere as follows. Fig. 13 (a) Initial setup. (b) After automatic z- alignment. A cycle of pick-and-place: (c) Approach, (d) pick-up, (e) transport, (f) release target Step 1:The position of the microsphere is calculated and the two fingers are automatically opened wider than its width about 5 μm. The microhand approaches the microsphere so that the microsphere is in between the two microfingers (Fig. 13c). Step 2:The microsphere is grasped by closing the right microfinger so that the distance between the two microfingers is less than the microsphere’s diameter about 5 μm to hold the microsphere firmly. In the case of grasping microbiological objects, they may deform slightly but they should not be damaged by this slight deformation. The microsphere is then picked up a distance z  that is about the object diameter (Fig. 13d). Step 3:The microsphere is transported 100 μ m x   away from its position (Fig. 13e). Step 4:The microsphere is moved down the same distance z  by the microhand and is released (Fig. 13f). 5. Results and discussion 5.1 Real-time tracking of the microhand The microhand was tracked for 500 image frames in this experiment. The success rate was about 93.2%. The average computation time for searching the microhand was about 14.5 ms. The tracking frame rate was about 21 frames per second. Thus, real-time tracking was achieved. During tracking, the performance of LTPM was also recorded. In detecting the two microfingers in 500 successive AIF images for 20 times, the case where 3 lines were found was about 58% and about 93% of these cases have similar line-type patterns shown in Table 2. Although the detection of a high-speed moving micro transparent object is not the scope of this paper, the microhand moved at the highest speed of the system which is limited to 100 μm/s. If the microhand moves faster, the success rate of real- time tracking of the microhand may decrease dramatically due to the hardware limitations of the AIF imaging system. 5.2 Pick-and-place of different-sized microspheres Table 3 shows the success rate of pick-and-place experiment with different-sized microspheres after 20 trials. The success rate decreased for smaller objects. It was observed that smaller objects were more adhesive to the microfinger and they were difficult Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 737 Mã bài: 157 to release. In addition, the AIF imaging system was set up for an appropriate scanning range SWING = 80 μm for different-sized objects. With FRAME = 2, the resolution of the system was about 1.3 μm which may not be suitable for a perfect spherical object such as a 20 μm microsphere. Since the experiment was performed to evaluate the method of obtaining 3D information from the AIF imaging system, no treatment to the microfingers was performed to overcome adhesion problem that might have contributed to the decrease of the success rate. The success rate might also attribute to the vibration generated by the piezo actuator when grasping smaller microspheres. In the case of a microsphere, it can slide out of the two microfingers while being grasped if large vibration occurs. In the case of grasping a biological cell, vibration may not affect much at the grasping step since cells are generally adhesive. However, releasing a cell will be more difficult. Using a fingertip to push a cell which is adhered to the other microfinger may help to successfully release the cell. Table 3 Pick-and-place performance for microspheres of different sizes Microsphere 96 μm 55 μm 20 μm Success rate 90% 80% 74% Although a trade-off between the accuracy and the scanning frequency of AIF imaging was considered when determining parameter , FRAME better piezo actuators with less vibration and higher scanning frequency may improve the accuracy as well as the real-time performance of the system. The success rate of pick-and-place task can also increase with better experimental setup to reduce vibration and by giving the feedback of the object’s size to adaptively change parameter SWING to obtain higher resolution or accuracy of AIF imaging. In this experiment, the size of the smallest microsphere is 20 μm in diameter. The z- resolution of the AIF imaging system might be large compared with the size of the smallest microsphere. To achieve higher success rate of pick-and-place of smaller microobjects such as 20 μm microspheres, the parameter SWING should be adjusted to improve AIF resolution depending on the detected size of the target object before handling it. The resolution of AIF imaging can also be improved by increasing the value of parameter ; FRAME however, this adjustment lowers the frame rate and affects the real-time performance of AIF imaging directly. 6. Conclusion This paper presents the AIF imaging system which is used to extend the depth of focus when observing microobjects. In addition, it also provides 3D information of microobjects being observed. Thus, 3D position measuring techniques have been proposed for both the end-effector and the target object so that handling microobjects can be automated. As a potential tool for micromanipulation, a two- fingered microhand was used in the experiment. Line-Type Pattern Matching was proposed to detect the 3D positions of the tips of the microfingers. Multisized microspheres were used as target objects in the pick-and-place experiment and their z-coordinates could be estimated with Contour- Depth Averaging. As AIF observation of microobjects and their 3D information can be obtained in real-time, an automated micromanipulation system for potential real-time microrobotic applications can be developed by integrating the AIF imaging system to a micromanipulation system such as a dexterous two-fingered microhand. References [1] Groen FC, Young IT, Ligthart G: A Comparison of Different Focus Functions for Use in Autofocus Algorithms, Cytometry, vol. 6, no. 2, pp. 81–91, 1985 [2] Sun Y, Duthaler S, Nelson BJ: Autofocusing in Computer microscopy: Selecting the Optimal Focus Algorithm, Microscopy Research and Technique, vol. 65, no. 3, pp. 139–149, 2004 [3] Mateos-Pérez JM, Redondo R, Nava R, Valdiviezo JC, Cristóbal G, Escalante- Ramírez B, Ruiz-Serrano MJ, Pascau J, and Desco M: Comparative Evaluation of Autofocus Algorithms for a Real-Time System for Automatic Detection of Mycobacterium Tuberculosis, Cytometry, vol. 81A, no. 3, pp. 213–221, 2012 738 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai VCM2012 [4] Ohba K, Ortega C, Tanie K, Rin G, Dangi R, Takei Y, Kaneko T, and Kawahara N: Real- Time Micro Observation Technique for Tele- Micro-Operation, in IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 1, pp. 647–652, 2000 [5] Ohara K, Ohba K, Tanikawa T, Hiraki M, Wakatsuki S, and Mizukawa M: Hands Free Micro Operation for Protein Crystal Analysis, in IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, pp. 1728–1733, 2004 [6] Avci E, Ohara K, Takubo T, Mae Y, Arai T: A new multi-scale micromanipulation system with dexterous motion. In: Int symp micro- nanomechatronics human science, pp. 444– 449, 2009 [7] Inoue K, Tanikawa T, Arai T: Micro- manipulation system with a two-fingered micro-hand and its potential application in bioscience. J Biotechnol, vol. 133, no. 2, pp. 219–224, 2008 [8] Kawakami D, Ohara K, Takubo T, Mae Y, Ichikawa A, Tanikawa T, Arai T: Cell stiffness measurement using two-fingered microhand. ROBIO, pp. 1019–1024, 2010 [9] Inoue K, Nishi D, Takubo T, Arai T: Measurement of mechanical properties of living cells using micro fingers and AFM cantilever. In: Int symp micro- nanomechatronics human science, pp. 1–6, 2006 [10] Ohba K, Ortega JCP, Tanie K, Tsuji M, Yamada S: Microscopic vision system with All-In-Focus and depth images. Mach Vis Appl, vol. 15, no. 2, pp. 55–62, 2003 [11] Boissenin M, Wedekind J, Selvan AN, Amavasai BP, Caparrelli F, Travis JR: Computer vision methods for optical microscopes. Image Vis Comput, vol. 25, no. 7, pp. 1107–1116, 2007 [12] Jain R, Kasturi R, Schunck BG: Machine vision. McGraw-Hill, Inc., New York, 1995 [13] O’Gorman F, Clowes MB: Finding picture edges through collinearity of feature points. In: Proc 3rd int joint conf artif intell, pp 543– 555, 1973 . Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 729 Mã bài: 157 Hệ thống tạo ảnh toàn nét và ứng dụng thời gian thực trong các hệ robot cấp độ micro All-In-Focus imaging and. Bài nghiên cứu này đề xuất ứng dụng giải thuật tạo ảnh toàn nét để giúp tự động hóa thao tác các vi vật thể trong khi có thể quan sát chúng được rõ nét trong thời gian thực. Thí nghiệm gắp thả. nghiệm gắp thả các vi vật thể với kích thước khác nhau được thực hiện để kiểm tra tính khả dụng của một hệ vi thao tác tự động thời gian thực. Abstract: In life sciences, observing and manipulating