The use of robotics in mining will accelerate mine mapping, create virtual models, assist workers and increase safety
Robotics is one of the disruptive technologies of our century and its opportunities for mining are increasingly discussed both in academia (Marshall et al, 2016) and industry (IFR, 2015). In underground mining, mobile robotic systems are found in modern production sites where they are predominantly used as autonomous transport vehicles. In most cases, the areas of operation are prepared in advance for the use of robots. These preparations often include the separation of the operation area for the autonomous vehicles from areas where people are granted access. In the future, robots in underground mining may be used in mixed scenarios, with tasks shared between miner and machine.
Potential benefits arising from the application of mobile robots in underground mining are manifold. An improvement in occupational safety for people and equipment alike can be expected. Furthermore, robots are suitable for exploration, mapping and monitoring operations. Dangerous, potentially collapsing mine sites are better explored by robots as the loss of a machine attains no ethical and moral dimension compared to the loss of a human life. Robots that continuously monitor the environmental conditions (eg mine air measurements) could contribute to improving the safety of mining personnel working underground.
Furthermore, regularly occurring or tedious work routines could be assigned to mobile robots, yielding a reduction of manpower and costs. Similarly, operation times and monitoring cycles may be increased using mobile robots.
The automated, comprehensive, accurate and cost-effective sensor-based mapping of mines by mobile robots creates the potential for a qualitative leap in different areas. Exploration, extraction planning and the optimisation of operations, labour protection and environmental monitoring may benefit from underground robotics.
The challenges in underground mining scenarios include the unavailability of GPS or similar satellite-based positioning systems for navigation, the ruggedness of underground terrain and the rough environment. Thus, the conditions in underground mines are particularly difficult, and adopting well-known robotic concepts and algorithms is not always possible in a straightforward manner.
This article presents a short overview of the Mining-RoX project conducted at the Technical University Bergakademie Freiberg, Germany. The Mining-RoX project strives to evaluate and adapt technologies and algorithms for mobile robots to settings in underground mines to improve mine operation processes of the future. For this, mobile robots are deployed in the university’s research mine, an old ore mine. A variety of robot tasks are researched, including:
- autonomous monitoring of environmental conditions in mines
- robotic co-workers assisting miners
- using robots for 3D mapping
- the creation of site-specific virtual reality (VR) training simulators for mine rescue brigades.
Mapping underground mines with mobile robots
Pioneering research on the use of autonomous robots in abandoned mines was presented in 2003 by Ferguson et al. The ‘Groundhog’ robot explored the Florence mine near Burgettstown and the Mathies mine near New Eagle (both in Pennsylvania, USA). This research project focused on mapping and navigation. A 2D representation of the environment was reconstructed using a laser scanner. The robot mapped 40 m of the abandoned coal mines in teleoperated mode and 308 m in autonomous mode.
Zlot and Bosse (2014) describe the mapping of a large, active underground mine. A pickup truck equipped with multiple laser scanners was used to create a virtual 3D model of a modern copper and gold mine in New South Wales, Australia. A total of 17 km was scanned in 113 minutes. In a post-processing step, a 3D model of the mine was generated that includes minor flaws but is overall a suitable approximation of the mine geometry. The transfer of the mapping technology to an actual robot is mentioned as future work in that article.
Mapping is one of the core problems in robotics, and the problem has certainly not yet been fully solved for mapping of underground mines. In the project ‘Autonomous Rock Surface Modelling and Mapping in Mines’, the team of Martin Adams of the University of Chile investigated the simultaneous localisation and mapping task in underground mines (Hennessey, 2015). By using a 3D Riegl LiDAR scanner, visual sensors and radar to cope with dust, models of an underground mine were acquired. Research at the Mining Systems Laboratory at Queen’s University is looking to develop an underground positioning system (uGPS). Recent work aims to improve the accuracy of mobile mapping methods by estimating the orientations of planar surfaces (Gallant and Marshall, 2016). These approaches still require a human to control the robot on site, but autonomy could become an option with an adequate navigation map.
Alexander – an autonomous mapping robot
The Mining-RoX project at the Technical University Bergakademie Freiberg is concerned with the creation of mobile robot technology to facilitate the autonomous exploration of underground mines. Besides the requirements regarding the environmental conditions, the robot platform faces additional constraints arising from the specific characteristics in mining. The mobile robot Alexander (Figure 1) was designed to meet these demands (Grehl et al, 2015a). The main tasks of the mobile robot are mapping and the monitoring of environmental parameters underground. The robot was equipped with several cameras, laser scanners and sensors for various environmental parameters (eg wind speed and radioactivity). An adjustable light system provides appropriate lighting for the cameras. The robot’s real-time self-localisation and 2D map generation is achieved by a sensor fusion approach based on the robot’s inertial measurement unit, laser scanner and colour camera images.
A 3D reconstruction of the underground mine can be generated online from two RGBD cameras. Besides colour images, the cameras also generate depth images. After a calibration step, the colour value can be combined with the depth information to generate a 3D coloured point cloud. This gives the miner in the control room the opportunity to get an immersive on-site experience. In addition, highly detailed textured 3D models can be generated from the colour images in a post-processing step.
Alexander’s abilities were tested in the local research mine ‘Reiche Zeche’. The robot was deployed in the first level at a depth of 147 m. The robot is able to construct a 2D navigation map in real time, measure temperature, humidity, air velocity and radioactivity and georeference these measurements within the generated map. Two test courses were chosen: a circular path of approximately 80 m next to the shaft on flat ground (Figure 2) and an abandoned extraction area of approximately 600 m on ground that is mostly rugged and muddy and in some parts also features railway tracks, as shown in Figure 1. The robot was able to autonomously explore all areas, except those with railway tracks. Rails and especially railway crossings turned out to be rather difficult for the autonomous navigation approach. In such areas, a member of the research team controlled the robot. Mapping and sensing capabilities were still running and delivering useful results. A total of 5 GB of data were collected for the first test course and 52.5 GB for the second. The data rate is approximately 1 GB per minute, mainly consisting of compressed camera images. These images are used to generate a textured 3D model of the underground mine in post-processing. Alexander is limited to observing and communicating. In order to interact with the environment and assist the miner, the second robot designed in the Mining-RoX project features a versatile manipulation tool.
Julius – a mobile manipulation robot
Julius is equipped with an articulated three-finger hand mounted on a robotic arm. Its four computer units provide the processing power needed for online processing of sensory information and robot control. The research focus lies on exploring co-working scenarios where Julius assists the miner in tedious or unsafe tasks using its robotic arm and the three-finger gripper. Special equipment such as an X-ray fluorescence handheld scanner can be operated by Julius (Figure 3). This limits exposure to radiation, thus enhancing workers’ safety. In addition, the robot can hold its arm in the same position for hours if necessary, guaranteeing exact measurements. Of course, the worker may also take advantage of the other equipment that Julius carries, such as light and power.
The safest mine may be an unmanned one. In this case, the robot can be teleoperated over a wi-fi network from a remote control room. Julius also carries special wi-fi stations to extend the network range in dead areas. When the wi-fi signal becomes weak, Julius uses its gripper to place one of the wi-fi stations on the floor. Carrying up to three stations, this extends the range of the wi-fi network around three corners. Once at site, Julius can use its arm to:
- investigate specific parts with its gripper camera for loose rocks
- push buttons to open or close air doors, handling the mine ventilation
- take water samples with an on-board station in abandoned areas.
A constant video stream from Julius to the operator in the control room is crucial for teleoperation. However, this may be difficult to achieve due to the special conditions in an underground mine. Autonomy is needed as a recovery strategy when a connection breaks. The robot should also be able to pick up previous unloaded wi-fi stations and navigate back to a given fall-back location.
The teleoperation capabilities and wi-fi communication routines of Julius were successfully tested in the research mine ‘Reiche Zeche’. Current research examines tasks where Julius acts as an assistant to the human worker. One such scenario is the extraction of a water sample from the mine. Since the miner does not want to contaminate the sample, they can use the robot with a defined routine and water sampling station to ensure a clean probe. However, as the robot is not able to identify a section on the surface with enough water to fulfil the task, the miner can point out the location and the robot can start its routine. This is just one example of the possibilities of joint human-robot teams in mining, where complex decision-making is done by the human while strenuous, precise or unsafe tasks are delegated to the robot.
From 3D mine models to virtual reality simulators for safety training
A 3D reconstruction of the mine geometry for visualisation, training and simulation purposes can be computed based on the images captured by Alexander in the mine. For this, optical features in consecutive images are matched and combined with the depth information to generate a point cloud model of the mine. This point cloud is combined with known camera positions and images to generate a textured 3D model. The interested reader is pointed to Grehl et al (2015b) for further details of this process. In the test runs mentioned previously, models of 0.6 GB and 1.25 GB were produced. The relatively low data size makes the generated models portable and easy to share.
The 3D mine models can be inspected on a state-of-the art notebook or in larger-scale, VR environments like Cave Automatic Virtual Environments (CAVEs) or head-mounted displays. This gives the opportunity to share experiences of the underground mine with both experts and the general public. The 3D models can also be used in site-specific VR simulations of mining operations. For example, before making a purchasing decision of an expensive machine, its operation could be simulated in a faithful 3D model of a specific mine.
A focus of our work is VR simulators for safety training. Within the Mining-RoX project, researchers from the Technical University Bergakademie Freiberg and the University of Applied Science Mittweida developed a rescue training simulator based on the 3D models acquired by Alexander. The simulator is currently used in academic education to teach students rescue scenarios. During training, students use a virtual environment based on 3D scans of the mine augmented by virtual characters representing rescue personnel and victims (see Figure 4). The instructor can trigger several events in the virtual mine such as fire, smoke or partial collapse. The instructor observes the reactions of the students and can then evaluate their behaviour. The students have to react and interact. Each member of the rescue team has a different perspective and carries different equipment. The simulation of stressful situations in a safe environment allows students to improve their communication skills and processes. The main advantage is the real-world origin of the 3D mine models. During training, mine rescuers acquire spatial knowledge about the mine, which can be of invaluable help in real emergency situations where smoke and dust may severely limit rescuers’ spatial awareness.
Autonomous robots will presumably play a significant role in the future of the mining industry. Within the Mining-RoX research project, we explored several use cases of mobile robots and demonstrated their feasibility in underground mining conditions. Robots can provide a map of the mine and georeference sensor data within the map. In addition, robots equipped with a manipulator arm can serve as assistants to miners, helping with tedious or dangerous tasks. Images captured by robots while traversing the mine can be transformed into realistic virtual mine models. When combined with the capabilities of VR training scenarios, powerful rescue training simulations can be built to improve safety. The development of mobile robots for underground mining has just begun. Mining has been identified as an important field of application within the robotics research community, and the number of research projects is growing worldwide.
Ferguson D, Morris A, Hähnel D, Baker C, Omohundro Z and Reverte C, 2003. An autonomous robotic system for mapping abandoned mines, in Proceedings 16th International Conference on Neural Information Processing Systems, pp 587–594 (MIT Press: Cambridge).
Gallant M J and Marshall A, 2016. Automated three-dimensional axis mapping with a mobile platform, in Proceedings International Conference on Robotics and Automation (ICRA), 6 p (IEEE: New York).
Grehl S, Donner M, Ferber M, Dietze A, Mischo H and Jung B, 2015a. Mining-RoX – mobile robots in underground mining, in Proceedings Third International Future Mining Conference, pp 57–64 (The Australasian Institute of Mining and Metallurgy: Melbourne).
Grehl S, Sastuba M, Donner M, Ferber M, Schreiter F and Mischo H, 2015b. Towards virtualization of underground mines using mobile robots – from 3D scans to virtual mines, in Proceedings 23rd International Symposium on Mine Planing and Equipment Selection, pp 711–722 (The Southern African Institute of Mining and Metallurgy: Johannisburg).
Hennessey M, 2015. Robots explore dangerous mines with novel sensor fusion technology [online], Clearpath Robotics, 27 January. Available from: www.clearpathrobotics.com/2015/01/robots-explore-mines-with-sensor-fusion/
International Federation of Robotics (IFR), 2015. World robotics report: service robots.
Marshall J A, Bonchis A, Nebot E and Scheding S, 2016. Robotics in mining, in Springer Handbook of Robotics, second edition, pp 1549–1576 (Springer: Heidelberg).
Zlot R and Bosse M, 2014. Efficient large-scale three-dimensional mobile mapping for underground mines, Journal of Field Robotics, 31(5):758–779.