April 2017

The development of computer vision technology in mining

  • By Paul Lucey, CEO, Mine Vision Systems

Computer vision technology has the potential to revolutionise mining and improve worker safety

Taking a step into the unknown world of mining innovation is usually only for the brave or foolhardy, particularly for any company that is not a tier-one miner. It is one of the hardest landscapes in which to succeed in mining. Mining is seriously complex, dynamic and conservative in nature and is subject to boom and bust cycles. Despite this, Gold Fields has managed to turn out a very interesting tech company at a very interesting time.

Every project starts with a worthy goal and a catalyst

In 2007, Gold Fields CEO Nick Holland said ‘if we cannot mine safely, we will not mine.’ It was a bold statement for the times, particularly from a South African-based company. Yet people within Gold Fields rose to the challenge, and two projects in particular took the big step into the unknown. They were the robotics program conducted with Carnegie Mellon University (CMU) Pittsburgh and the automation program with Caterpillar and Atlas Copco at Gold Fields’ Agnew mine in Western Australia. These programs were begun independently of one another without either group knowing the other existed, a trait not uncommon in any large company.

I was partly responsible for running the autonomous program in Australia. This project was less audacious than its South African robotics program counterpart.

The goal for both programs was to create safer and more productive mining by removing people from the working face, the most dangerous part of the mine. It turns out that both the robotic and autonomous approaches are very difficult to do; autonomous equipment does work, albeit with some pretty hard limitations and only incremental improvements. However, Gold Fields was able to take the computer vision component required to make the robots function and apply that to mobile mining equipment. The result of combining the two projects is likely to bring a new wave of safety and productivity into mining and far exceed the original goal.

In 2007, the Gold Fields robot ‘Cybot’ was created as a conceptual prototype by the National Robotics and Engineering Centre (NREC), which is part of the CMU Robotics Institute. This project had the support of Gold Fields CEO Nick Holland, something that is very important for a brave and bold project like this.

The long road to progress

By 2008, the automation program at Agnew was in full swing, with the first Caterpillar MineGem autonomous loader being installed. At the time, this was a truly amazing piece of equipment. It was fast and incredibly easy to use. I firmly believed that this was the future of mining.

Back in South Africa, testing of the second prototype had begun, undertaking the dangerous and physical tasks required in a conventional underground mine.

A purpose-built facility was constructed in Pittsburgh at the NREC to replicate the ground conditions in South Africa. The facility featured concrete bunkers that were designed to replicate the tight and sloping conditions experienced in a South African underground mine.

In mid-2008, Gold Fields had started to look at using the new Atlas Copco autonomous drills, and I set up a factory-backed program with Atlas Copco to have these machines up and running at Agnew.

It was also clear by the end of 2009 that the autonomous loader project was developing issues. This was a very complicated piece of equipment requiring specially skilled and experienced people to maintain. It also required a specific mine design for the piece of equipment to work. If the mine development was even slightly off from design, the machine would not be able to navigate the corners. If the machine broke down during a shift without a highly skilled technician on site, production was delayed. Issues such as these were always a constant battle, but a battle that we thought was worth fighting and so we continued. Part of the problem was that ancillary technical tasks were still required to be done by human professionals such as geologists, geotechnical engineers, surveyors and mining engineers. This was always much more challenging when dealing with autonomous equipment and impacted significantly on the efficiency of the program. Gold Fields had just signed an agreement with CSIRO, and I investigated with CSIRO if it was possible for a robot to complete the ancillary tasks.

By this time, a senior executive at Gold Fields had seen the synergies between the two projects and the robotics program was introduced to my program. I was impressed by the sheer audacity of the South African project to challenge the status quo. I approached NREC to solve my ancillary task issue and saw for the first time its computer vision component. This was designed as a tool to control the robots underground; I saw it as a tool to possibly conduct some basic visual tasks on the automation project.

By 2010, the autonomous project was working directly with the robotics group on the computer vision component of the project. I had visited NREC and we had selected the smaller robotic unit as our proposed auxiliary robot.

The breakthrough and first glimpses of what Mine Vision Systems (MVS) would become came in 2010 when working on a computer vision program to set up fiduciary points for survey so the robot could navigate. We discovered that the computer vision was able to navigate in its 3D environment without survey support.

Knowing when to pivot

In 2011, both projects were running out of steam. The autonomous loader was showing massive productivity increases over the current teleremote solutions in isolation, but an overall average drop in productivity of about 10 per cent when stoppage, mine design and ancillary tasks were added in. New products, such as early versions of the RCT guidance

systems, were starting to appear around this time. While these products were not as fast or feature rich as the MineGem system, they were much cheaper and did not require specialist technicians or a special mine design to operate. In this case, the RCT guidance systems eventually became the default systems for Gold Fields. This in itself was a successful outcome of the autonomous project. Gold Fields is still able to gain full productivity from this product because of the early work that occurred from the automation project. At some point, the automated loader ceased being a project and became just another standard mining tool. The Atlas Copco project continued on for another year. This project finally closed when Agnew went to full contract mining and the contractor brought in its own drills and dropped the automation program.

While I would call the automation project a success, it was not the panacea that I had hoped; there were simply too many technical difficulties to overcome. The computer vision component from the robotics program was seen as a possible solution for this.

At the end of 2011, I was the head of innovation and technology at Gold Fields, and a review began of the robotics program in South Africa. By June 2012, the project faced the double blow of severe technical challenges coupled with the spin out of the conventional assets into Sibanye. The robotics program was finally shut down after nearly six years. NREC was notified of Gold Fields’ interest in picking up the vision component sometime in the future. By mid-2013, there were no more projects running.

A year later, I had a new role as the head of engineering, and I took the opportunity to take another look at this project. During this ‘gap’ year, NREC had been busy further developing the vision technology with other industries, with elements of the program finding its way into US military projects, other heavy industries, oil and gas and the Department of Homeland Security. It was the last project that produced the Carnegie Robotics (CRL) SL Sensor that MVS currently uses. CRL was another spin out from CMU and became a future MVS partner.

A CRL SL Sensor was brought out to St Ives in March 2014 and used to map the Athena underground mine. Twenty kilometres of decline was mapped within three hours and then processed in less than four hours. The resulting data showed the potential of what could be done with this technology. This data formed the basis of the proposal to create a start-up company based on this technology. As Gold Fields had taken this technology as far as it could, the decision was finally made in October 2014 to support the proposal and create MVS. Even without taking this step, Gold Fields had learned more than enough to justify the investment to this point. Creating a technology spin out company was another bold and brave move. Rather than covering the technology in a shroud of secrecy, Gold Fields was opening it up to the entire mining community, and the company intended to be at the forefront.

Creating a tech company at the start of the tech boom

In January 2015, MVS was created in Pittsburgh, spinning out from CMU. Less than two weeks later, Uber poached a considerable group of the top engineers from NREC and then CRL. This had a negative impact on Gold Fields’ fledging start-up. Uber had decided to enter the race to build an autonomous car, pitting itself against the likes of Google, Apple and Tesla. A week later, I hopped off a plane after a 34-hour journey to Pittsburgh from Perth to try and salvage our tiny start-up. Doing technology in mining is tough enough without having to deal with tech giants poaching key talent. It turns out that automation and computer vision was about to become one of the hottest techs on the planet at the same time that we thought it would be a good idea to bring that technology into mining.

After the Uber event, MVS had one full-time employee, two part-time employees and not much else. I already had a full-time position at Gold Fields in what was to become a busy year. It was confirmed in 2015 that to do technology in mining for the long term requires a completely tenacious and extremely patient disposition. Despite this, Gold Fields still supported the technology, and we managed to scrape enough time and resources together to keep MVS alive. Despite all this, MVS signed more than 80 non-disclosure agreements with nearly every major mining software company and original equipment manufacturer in mining. We were clearly onto something groundbreaking. Even today, competition is fierce; when we interview people they are also typically being interviewed by Google, Facebook, Apple, Uber and Amazon. Computer vision is used in everything from virtual reality through to autonomous cars and is one of the most sought-after technologies in the world today.

Back at MVS, I had other challenges. Explaining mining to non-mining people was like explaining customs and practices from an alien culture. In 2015, I found myself sitting around a lot with computer vision engineers, roboticists, computer scientists and other very smart people explaining what this technology could solve for mining. This was often met with disbelief and ridicule. For example, I explained how it was still common practice for a geologist to sketch a drive and colour it in later with pencils before painstakingly inputting that into a software program. I had to physically take them to a mine site for them to believe me.

In our defense, Gold Fields created and successfully completed the Ore-X Challenge in 2016, an online crowd-sourcing event that used machine learning algorithms to undertake face mapping. This showed that Gold Fields was still a leader in mining innovation. Again, this was using computer vision and machine learning.

Delivering on the goal

By 2016, MVS had recovered from its near-death Uber experience, and, with some healthy seed funding from Gold Fields and a new team to take the technology forward, the real work could begin. We had a shortlist of tasks to tackle that included geology mapping, reconciliation and geotechnical. We decided on geotechnical as it was the most difficult; there was no real software in the marketplace and I thought that it would be the best showcase of the technology while also benefitting Gold Fields.

MVS has now produced the most accurate and advanced geotechnical computer vision software mining has ever seen. MVS has the ability to create colourised point clouds, statistical data, cross-sectional data, mesh and virtual tape extensometer readings. MVS can export this data into leading software packages, display it conventionally or use it with augmented reality such as Microsoft HoloLens.

The uses for computer vision are endless: geology mapping, fragmentation, short interval control in the open pit and reconciliation – the list goes on. By the end of this decade, I believe that computer vision will allow anyone to view a mining area from the safety of their office in greater detail and with more resources then if they were actually there, and it will be used by every major original equipment manufacturer and software provider. Computer vision will turn previously ‘dumb’ pieces of equipment into intelligent information-gathering and interactive devices. There will be no need to go within 20 m of a working face, be it underground or on the surface. Whether MVS is the company to deliver this or not is yet to be seen, but it will be delivered and you will be able to trace its origins back to Gold Fields and Nick Holland’s commitment to safety and the initiatives he supported back in 2007.

While nothing may happen overnight in mining, change will occur. As Rio Tinto has shown by running the largest fleet of autonomous mining equipment in the world, that change can be profound. All you need is vision, tenacity and good people. I’d like to think by the end of this decade, mining will be the world leader in industrial computer vision.

Share This Article