Controlling Robot Swarms With Augmented Reality

While robots are getting increasingly more capable and autonomous, they may still require our help to carry out tasks in real-world environments. That’s why robotics researchers continue to look for new ways to allow human operators to control robots without extensive training.

At New York University, Jared Alan Frank has turned to augmented reality (AR) to develop a robot control interface that runs on a conventional smartphone or tablet. The system uses the device’s camera to capture details from a scene and overlay virtual objects, as other AR applications do. But in this case, you can simply tap and swipe on the screen to make the robots move or pick up objects.

“You no longer need some of the expensive laboratory-grade equipment that roboticists normally use to do their projects, such as motion capture systems,” says Frank, a Ph.D. candidate in mechanical engineering at NYU’s Tandon School of Engineering.

Using Xcode, Apple’s software development platform, Frank built an app that can detect robots and objects in the environment and create a virtual grid—along with a coordinate system—to keep track of those objects on the screen. The user can then manipulate the objects on the device and watch as the robots carry out the desired actions in the real world. Commands from the app are sent via Wi-Fi to the robots, which in the current version use Raspberry Pi as the main controller.

To help his app make sense of the environment, Frank places visual tags on the robots and on the objects he wants the robots to move. The smartphone or tablet captures the scene using its camera, and the app detects the tags, using that information to keep track of marked objects. These tags, also called fiducial markers, are commonly used in AR apps to integrate physical landmarks and objects into a virtual world.

Ease of operation is one of the key features of this technology, but another advantage that Frank emphasizes is its mobility. He explains that if you’re able to control a swarm of robots using a simple mobile device, it becomes much easier to bring robotic applications outside the confines of the lab environment.

And that’s part of what Frank wants to do next. His plan is to test the technology in places like construction sites and factories. To do that, he might need to improve some of the software and hardware he’s been using, to make sure they can operate safely and reliably under real-world conditions. But above all, his main goal is to keep the system as simple and accessible as possible.

“What we really want,” Frank says, “is for people who’ve never had the ability to do this before to be able to take their device out of their pocket and control a robot.”