17'5 Gdzzly Figure 10. Results from field trip. The diagram in the top left hand corner shows the experimental setup. Note that this setup caused range shadows behind large rocks. 176 Figure 11. The view from above (top), and from the front (bottom) of the grizzly covered in large rocks. 177 Figure 12. Results from the laser scanner looking at a charging face. Note the enlargement of the front view clearly shows a ring of drilled holes and a butt. t78 ideal location of the laser scanner would have been directly above the grizzly, eliminating most shadows, but this was not possible given the limited amount of time on site. Figure 11 shows in more detail the data collected when some large rocks were dumped on the grizzly. The lower portion of the figure shows the scene from the front. Figure 12 shows some different views of the end of a mine tunnel, all derived from one scan. The figure clearly shows details such as the drilled blast holes and a blasting 'butt' (basically a hole in the rock face caused from the previous blasting cycle). The floor appears extremely smooth because it was a pool of water. 3.2. A seml-automated rock breaker Dealing with oversize rocks is a common problem in surface and underground mining. Large rocks may jam a crushing plant or chute, or be too large to travel along a conveyor system. In underground mines grizzly screens are used to filter out oversize rocks. A grizzly screen is typically a very solid steel structure over an ore pass or crusher that provides a mesh size in the range 0.5 to lm. Material may reach the grizzly from an ore pass via a chain feeder or be dumped directly by truck. The grizzly screens must be kept clear from build up of loose material or oversize rocks. A rock-breaker, see Figure 6, is a manually controlled hydraulic arm that carries a hydraulic impact hammer. In structure it is akin to a back-hoe excavator, and kinematically it is similar to a standard industrial robot. In order to automate this process we would need 1. a computer controllable rock breaking boom, 2. a 3-D sensing system, 3. the automation system, and 4. a teleoperation system. The proposed system would use 3-dimensional sensors to monitor the griz- zly, and when necessary control the breaking boom so as to clear the griz- zly. The imaging aspects of rock-breaker automation have been previously studied[14, 15] and trialled in the laboratory using a small scale model [16]. Due to the difficulty, in what is a complex and only partially structured environment, of foreseeing all eventualities 2 we do not believe that at this stage it is feasible to fully automate the process. Complications involved in carrying out this task include dealing with foreign objects such as timber props and ground support bolts. In the event of the system being unable to autonomously clear the grizzly, it would signal a remote operator who would use teleoperation of the breaker to complete the task. Such limited human intervention would be the most cost effective solution for dealing with these situations, and would make it possible for a single operator to supervise several rock breakers located at different sites around the mine. 2Rock shapes and type, and foreign objects. 179 .3-Dsensor Computer controllable rock break~ I ,, I Grizzly \ ii:: ration Automation system system Figure 13. The proposed semi-automated rock breaker. 4. Conclusion The ultimate aim of mine automation will be to remove miners from the haz- ardous areas of the mining environment where the work will instead be per- formed by autonomous and sensate mining machines. Such a vision is many decades from reality and in the interim we can hope to make small steps to- ward this goal. One step is to increase the productivity of existing mining equipment by assisting the operator so that one operator can supervise several machines. The dragline and the rock breaker automation projects described here are example of this. Considerable work within the robotics and computer vision is highly appli- cable to real-world automation needs such as exist within the mining industry. The research community has demonstrated the feasibility of using machine vi- sion to 'close the loop' on the position of robot manipulators. Such technology could be usefully applied to many applications in mining. Superficially these may seem very different problems, but this is largely a matter of scale. Larger machines in fact require reaction times that are considerably longer than those being demonstrated now in robotics laboratories. The biggest, but not insur- mountable, challenge is the complexity of scene analysis in a complex mining environment. Acknowledgements The authors gratefully acknowledge the help of their colleagues Stuart Wolfe, David Hainsworth, Stephen Nothdurft, Zheng-De Li, Jasmine Banks, Hal Gur- genci, Don Flynn, Peter Nicolay, Allan Boughen and Daniel Sweatman. The dragline project is funded by a consortium consisting of the Australian Coal Association Research Programme (as project C5003), Pacific Coal Pty Ltd, BHP Australia Coal Pty Ltd and the Cooperative Research Centre for Mining Technology and Equipment (CMTE), a joint venture between AMIRA, CSIRO 180 and the University of Queensland. Bucyrus Erie Australia and Tritronics Pty. Ltd. have provided valuable in-kind support, and Tarong Coal have gener- ously allowed the automated swing system to be installed on their BE1370. The underground mining robotics work has been supported by the CMTE and AMIRA project P440 which was sponsored by Mount Isa Mines, Normandy Poseidon and Western Mining. References [1] P. I. Corke, Visual Control of Robots: High-Performance visual servoing. Mecha- tronics, Research Studies Press (John Wiley), 1996. [2] S. Hutchinson, G. Hager, and P. Corke, "A tutorial on visual servo control," IEEE Transactions on Robotics and Automation, vol. 12, pp. 651-670, Oct. 1996. [3] D. Hainsworth, G. Winstanley, Y. Li, P. Corke, and H. Gurgenci, "Automatic control of dragline operation using machine vision control of bucket position," in Proc. First CMTE Annual Conference, (Brisbane), pp. 111-114, July 1994. [4] P. I. Corke, G. Winstardey, and J. Roberts, "Dragfine modelling and control," in Proe. IEEE Int. Conf. Robotics and Automation, (Albuquerque, NM), pp. 1657- 1662, 1997. [5] G. Winstanley, P. Corke, and J. Roberts, "Dragfine swing automation," in Proc. IEEE Int. Conf. Robotics and Automation, (Albuquerque, NM), pp. 1827-1832, 1997. [6] Z. Li, P. Corke, and H. Gurgenci, "Modelling and simulation of an electro- hydraulic mining manipulator," in Proc. IEEE Int. Conf. Robotics and Automa- tion, (Albuquerque, NM), pp. 1663-1668, 1997. [7] R. A. Jarvis, "A perspective on range finding techniques for computer vision," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-5, pp. 122-139, Mar. 1983. [8] R. Chatila, S. Fleury, and M. Herrb, "Autonomous navigation in natural envi- ronments," in Experimental Robotics III (T. Yoshikawa and F. Miyazaki, eds.), pp. 425-443, Springer-Verlag, 1994. [9] M. Thompson, ed., Manual of Photogrammetry. Falls Church, VA: American Society of Photogrammetry, 3 ed., 1966. [10] P. Fua~ "A parallel stereo algorithm that produces dense depth maps and pre- serves image features," Machine Vision and Applications, vol. 6~ pp. 35-49~ 1993. [11] R. Zabih and J. WoodfiU, "Non-parametric local transforms for computing visual correspondence," in Proc. 3rd European Conf. Computer Vision, (Stockholm), May 1994. [12] P. Dunn and P. Corke, "Real-time stereopsis using FPGAs," in Proc. Intl. Work- shop on Field Programmable Logic,, (Imperial College, London), Sept. 1997. [13] J. Woodfill and B. V. Herzen, "Real-time stereo vision on the parts reconfig- urable computer," in IEEE Workshop on FPGAs for Custom Computing Ma- chines, pp. 242-250, Apr. 1997. [14] S. Elgazzar, J. Domey, P. Boulanger, and G. Roth, "Three-dimensional imaging for mining automation," in Proe. 5th Canadian Syrup. on Mining Automation, pp. 334-340, 1992. [15] J. Domey and M. Rioux, "3-d vision sensors and their potential applications in mining automation," in 3rd Canadian Syrup. Mining Automation, (Montreal), 181 pp. 187-193, Sept. 1988. [16] C. Cheung, W Ferrie, R. Dimitrakopoulos, and G. Carayanis, "Towards com- puter vision driven rock modelling," in Proc. 2nd Canadian Conf. on Computer Applications in the Mineral Industry, (Vancover~ B.C.), Sept. 1991. HelpMate@, The Trackless Robotic Courier: A Perspective on the Development of a Commercial Autonomous Mobile Robot John M. Evans HelpMate Robotics, Inc. Danbury, CT, USA evans@helpmate, com Bala Krishnamurthy HelpMate Robotics, Inc. Danbury, CT, USA bala@helpmate.com Abstract HelpMate Robotics has developed an autonomous mobile robot courier for material transport in hospitals. These machines are in operation around the world, operating in uncontrolled and unsupervised environments up to 24 hours per day. The history and technology of the HelpMate robot are presented in this paper with the intention of providing a real world perspective on transitionmg technology from the laboratory to the marketplace. 1 Introduction HelpMate Robotics Inc. (HRI), of Danbury, Connecticut, USA, has developed a hospital courier robot that is the benchmark of commercial success in the emerging field of service robots. This paper will explore the decade of development of the HelpMate robot, focusing on the evolution of the technology from laboratory exploration to hardened commercial product. In trying to define market opportunities for robots in service applications and to match enabling technologies against those opportunities, several possibilities originally stood out: floor cleaning, security, hazardous enviromnents and hospital materials transport. All required autonomous mobile robots navigating in indoor structured environments. It was this technology base and these markets on which we focused early attention. 1.1 The Hospital as a Target Market Health care is an obvious market for automation because of the high and rising costs that must be tamed if any attempt at universally available care is to be offered in our society. Cost containment is an overriding theme in health care 183 management, and robotics is one technology to reduce labor costs. Operation of hospitals around the clock on a multi-shift basis makes capital justification easier, so this was a natural target. Development of the HelpMate concept has been described in earlier papers [1,2,3,4,5]. The final machine, shown in Figure 1, is able to navigate in crowded hallways, avoiding people and inanimate obstacles as it encounters them, and using the walls of the hallways as the principal navigation reference. Figure 2 shows the applications for HelpMate. .~ ~!i:~i+:i,i~iii i:i: . !:~?NNiiiiii+~ii{i?iiii!iit ' iiiiiiiiiiiii~i~i~i~)~: ~i~iii~'iiliiii~'iiiii~}~'~i iiiiii~iiiii"!':iiiii"ii;i"'ii+iiii ~l ~.~.:':'.'¢~:::::~ ~::~':s:::-'; N:.:~ :::::::::::::::::::::::: ~i~.~:.+': ':-~:~::::~:.:~:: ::::::::-:.:: .:.~::::: • +~l++i+++++++ ' x +~+::M :5-+~:- ++ +:+:+ + I~~ ++~+::.#++~+."+-'~.+'."."+:++!+ ~ [[~~+|+,+:+::+::::+++++i+ ~ :+++ :+!++:~+++++++++++++ii++++++++:.:i+.~+++ ~~~i'~!+ i~i~ ,-~: :~:~!~iiii!~iii -::.:: ========================== +l ".:~.::::::::::::::~i!! iiii "~i ".:-~!'ii: ~li' " ~+!":'+'~ ~i"+"!~!~i~i':~!~ii":+"i~ ::~ : ::::::::::::::::::::: ~i + "======================================================================== :-: ~.+ :~ ~?:: :.:. ~::::::: <:~.~+-+~.:.:.~,. ~ "~:i ::+:+:+::.:.++ n, ~ , +,+ ?-1 Figure 1: HelpMate Robot Courier 184 Dietary: late and special request trays Supply: equipment and material Lab: speciman and sample transport Pharmacy: medication and supplies Med Records: patient files Administration: mail and reports Radiology: films Mail: mail and packages Figure 2: Applications of HelpMate 2 Navigation: a problem in Sensing Over the years, we have come to believe that the key to autonomous mobile robot navigation is sensory perception, ffyou can obtain a good, dynamic, high resolution picture of the world around the robot, then you can successfully use any of several algorithmic approaches to planning a collision free path through the environment. Much of the early work on navigation algorithms hypothesized known, static worlds, what we would call "blocks worlds" after the highly structured and artificial worlds used in early machine vision research [6]. The heritage is that of trajectory planning for robot arms in free space, with force or compliance control considered as a special case [7]. Work is still presented today on such ideas. Potential field models, clothoids, path planning for non-holonomic vehicles: all presume a totally known and static environment [8,9,10] But the real world is not static. Hospital hallways, in particular, can be filled with moving people: employees, patients and visitors. So the problem becomes one of dynamic sensing, and control strategies must be dynamic and reactive. Further, it turns out that there are many "stealth" objects in the real world for any single sensory modality. Sound waves bounce off any hard, flat, angled surface and are absorbed by soft material such as a blanket. Light bounces off mirrors and chromed surfaces in a specular fashion and passes through glass, and light is absorbed by flat black material such as black wool pants. Hence, a multiplicity of different types of sensors maximizes the probability of sensing objects prior to collision. Contact sensors to detect collision are always required. The HelpMate robot uses sonar, vision, and contact sensors to interact safely with people and obstacles in a hospital world. [3] This combination of sensors is not unique and was in fact already known in the research community before we started working on HelpMate. During the early and [...]... different sensor modalities and control programs is needed to compensate for odometry errors and to avoid obstacles 188 3.10dometry Odometry or dead reckoning (properly ded reckoning, from deduced reckoning, an old Navy term describing the estimation of position of ships from velocity and time and heading measurements) is the basis for almost all mobile robot systems Odometry or dead reckoning is used... reckoning is used to refer to measurement of the robot's position and heading from wheel encoder readings and other internal position and velocity and acceleration sensors such as gyros or accelerometers or doppler velocity sensors A definitive survey is provided by Borenstein, Everett, and Feng [30] Odometry errors accumulate with time and depend upon the environmental conditions A rough or slippery or... of forward and angular velocity every tick time The 'set_velocity' and the 'jog' command interface provided by the drive subsystem facilitates setting of the velocity and the rate of turn in units of degrees per second, This interface is basically that proposed by Crowley, although he adds position and acceleration in each command and expects back an estimation of position uncertainty We deal with position... the obstacle, and the horizontal width of the contour line indicates its size Optical filtering, electronic shuttering, and frame-to-flume differencing are used to improve signal-to-noise and immunity to ambient lighting 2.3 Contact Bumpers For safely, the robot must be able to sense contact There will always be some obstacles in the environment that will defeat any finite array of non-contact proximity... increasing gain in the detection circuit with time, the chance of triggering on multi-path echos is significant Despite these measurement uncertainties, sonar and particularly the Polaroid transducers have been a favorite choice of both researchers and entrepreneurs since 1980 for an obvious reason: price Over two million dollars went into development of the original sensor and drive circuitry and in volume... the hanger bars for acoustic tiles, a standard ceiling design A pair of long range infrared proximity sensors are mounted on the shoulders of the robot, pointed at the ceiling These sensors see the ceiling tape and provide information on both position and orientation at those landmarkso We have used this ceiling tape system for safety as well as for navigation landmarks At any point where there is an... and endowed with enough intelligence to deal with the inexact world model and to rationalize the a priori information with the dynamic sensed model The definition of 'some' and 'enough' has taken over 10 years to refine! Once a world model is available, it is possible to use sensory data to match the position of the robot to that model This can be as simple as using single sensor data for wall following... and coworkers[37], and work at CMU and Grenoble by Crowley [33] and Effes[17] By the time HelpMate was started, the NIST work on hierarchical control had evolved to the concept of interjecting an explicit world model hierarchy between a sensor processing hierarchy and the control hierarchy The idea was to use the sensor data to servo the model and then to use the model data for control calculations... information and the structured geometry of the world in which the robot was to operate Over the years, we have experimented with one and two dimensional images, with views near the floor, normal and oblique views of the ceiling, optic flow of vertical features, and structured light vision systems Two concepts were deployed originally with the first models of the HelpMate, one remains in the current design,... fiberglass shell, was completed during the late spring of 1988 and taken to the Robots 12 exhibition in Cobo Hall in Detroit Time ran short, and the first time the shells were attached to the robot and buttoned up was on the exhibition floor This show was in June, and during set-up the outside doors were open, the air conditioning was turned off, and the temperature approached 40°C on the exhibition floor . completed during the late spring of 1988 and taken to the Robots 12 exhibition in Cobo Hall in Detroit. Time ran short, and the first time the shells were attached to the robot and buttoned. stepped in front of it and then find a way around the person if they stood still. The first field test, at the end of 1987, was from the Dietary department to the elevator bank in the next. some point in any system, however, reference must be made to outside landmarks to re-establish a position estimation that is correct with respect to the external environment. This is the process