IEEE.orgIEEE Xplore Digital LibraryIEEE StandardsMore Sites
      Sign InJoin IEEE
      Controlling Robot Swarms With Augmented Reality
      Share
      FOR THE TECHNOLOGY INSIDER
      Explore by topic
      AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensorsTelecommunicationsTransportation
      IEEE Spectrum
      FOR THE TECHNOLOGY INSIDER

      Topics

      AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensorsTelecommunicationsTransportation

      Sections

      FeaturesNewsOpinionCareersDIYThe Big PictureEngineering Resources

      More

      Special ReportsCollectionsExplainersPodcastsVideosNewslettersTop Programming LanguagesRobots Guide

      For IEEE Members

      Current IssueMagazine ArchiveThe InstituteTI Archive

      For IEEE Members

      Current IssueMagazine ArchiveThe InstituteTI Archive

      IEEE Spectrum

      About UsContact UsReprints & PermissionsAdvertising

      Follow IEEE Spectrum

      Support IEEE Spectrum

      IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, podcasts, and infographics inform our readers about developments in technology, engineering, and science.
      Join IEEE
      Subscribe
      About IEEEContact & SupportAccessibilityNondiscrimination PolicyTermsIEEE Privacy Policy
      © Copyright 2023 IEEE — All rights reserved. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.

      IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy.

      view privacy policy accept & close

      Enjoy more free content and benefits by creating an account

      Saving articles to read later requires an IEEE Spectrum account

      The Institute content is only available for members

      Downloading full PDF issues is exclusive for IEEE Members

      Access to Spectrum's Digital Edition is exclusive for IEEE Members

      Following topics is a feature exclusive for IEEE Members

      Adding your response to an article requires an IEEE Spectrum account

      Create an account to access more content and features on IEEE Spectrum, including the ability to save articles to read later, download Spectrum Collections, and participate in conversations with readers and editors. For more exclusive content and features, consider Joining IEEE.

      Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more →

      CREATE AN ACCOUNTSIGN IN
      JOIN IEEESIGN IN
      Close

      Special offer: Join IEEE now for 2023 and save 50%!

      IEEE Members receive 12 print issues of IEEE Spectrum and enjoy PDF downloads, full access to our archive with thousands of in-depth articles, and other exclusive content and features. Join IEEE today for 2023 and save 50%!

      JOIN IEEE
      RoboticsTopicTypeVideo

      Controlling Robot Swarms With Augmented Reality

      Engineers at NYU have built an AR app to operate robots in the real world

      Alyssa Pagano
      22 Apr 2017
      robot softwareportable devicestype:videomixed realityrobot swarmsroboticsaugmented realitymobile robotsprogramming

      While robots are getting increasingly more capable and autonomous, they may still require our help to carry out tasks in real-world environments. That’s why robotics researchers continue to look for new ways to allow human operators to control robots without extensive training.

      At New York University, Jared Alan Frank has turned to augmented reality (AR) to develop a robot control interface that runs on a conventional smartphone or tablet. The system uses the device’s camera to capture details from a scene and overlay virtual objects, as other AR applications do. But in this case, you can simply tap and swipe on the screen to make the robots move or pick up objects.

      “You no longer need some of the expensive laboratory-grade equipment that roboticists normally use to do their projects, such as motion capture systems,” says Frank, a Ph.D. candidate in mechanical engineering at NYU’s Tandon School of Engineering.

      Using Xcode, Apple’s software development platform, Frank built an app that can detect robots and objects in the environment and create a virtual grid—along with a coordinate system—to keep track of those objects on the screen. The user can then manipulate the objects on the device and watch as the robots carry out the desired actions in the real world. Commands from the app are sent via Wi-Fi to the robots, which in the current version use Raspberry Pi as the main controller.

      To help his app make sense of the environment, Frank places visual tags on the robots and on the objects he wants the robots to move. The smartphone or tablet captures the scene using its camera, and the app detects the tags, using that information to keep track of marked objects. These tags, also called fiducial markers, are commonly used in AR apps to integrate physical landmarks and objects into a virtual world.

      Ease of operation is one of the key features of this technology, but another advantage that Frank emphasizes is its mobility. He explains that if you’re able to control a swarm of robots using a simple mobile device, it becomes much easier to bring robotic applications outside the confines of the lab environment.

      And that’s part of what Frank wants to do next. His plan is to test the technology in places like construction sites and factories. To do that, he might need to improve some of the software and hardware he’s been using, to make sure they can operate safely and reliably under real-world conditions. But above all, his main goal is to keep the system as simple and accessible as possible.

      “What we really want,” Frank says, “is for people who’ve never had the ability to do this before to be able to take their device out of their pocket and control a robot.”

      The Conversation (0)

      Trending Stories

      The most-read stories on IEEE Spectrum right now

      BiomedicalTopicMagazineTypeFeature

      How an Electrical Engineer Solved Australia’s Most Famous Cold Case

      CareersTopicTypeNews

      Eight Graphs That Explain Software Engineering Salaries in 2023

      TransportationTopicTypeAnalysis

      What Is Tesla’s Mystery Magnet?

      RoboticsTopicTypeSpecial ReportsVideo

      360 Video: Zoom Over Zanzibar With Tanzania’s Drone Startups

      Come along for the ride as drones soar over the farms and schools of Tanzania

      Evan Ackerman

      Evan Ackerman is a senior editor at IEEE Spectrum. Since 2007, he has written over 6,000 articles on robotics and technology. He has a degree in Martian geology and is excellent at playing bagpipes.

      Michael Koziol

      Michael Koziol is an associate editor at IEEE Spectrum where he covers everything telecommunications. He graduated from Seattle University with bachelor's degrees in English and physics, and earned his master's degree in science journalism from New York University.

      Eliza Strickland

      Eliza Strickland is a senior editor at IEEE Spectrum, where she covers AI, biomedical engineering, and other topics. She holds a master’s degree in journalism from Columbia University.

      09 May 2019
      6:56
      Photo: IEEE Spectrum
      dronesgadgetstype:videoEast Africa dronesmappingTanzaniaAfricadelivery drones360 video

      With 360-degree video, IEEE Spectrum puts you aboard drones that are flying high above the Tanzanian landscape: You’ll ride along as drones soar above farms, towns, and the blue expanse of Lake Victoria. You’ll also meet the local entrepreneurs who are creating a new industry, finding applications for their drones in land surveying and delivery. And you’ll get a close-up view from a bamboo grove as a drone pilot named Bornlove builds a flying machine from bamboo and other materials.

      You can follow the action in a 360-degree video in three ways: 1) Watch on your computer, using your mouse to click and drag on the video; 2) watch on your phone, moving the phone around to change your view; or 3) watch on a VR headset for the full immersive experience.

      Keep Reading ↓Show less
      RoboticsTopicTypeSpecial ReportsVideo

      360 Video: Go on a Mission With Zipline’s Delivery Drones

      Immerse yourself in the action as Zipline catapults its drones into the Rwandan sky

      Evan Ackerman

      Evan Ackerman is a senior editor at IEEE Spectrum. Since 2007, he has written over 6,000 articles on robotics and technology. He has a degree in Martian geology and is excellent at playing bagpipes.

      Michael Koziol

      Michael Koziol is an associate editor at IEEE Spectrum where he covers everything telecommunications. He graduated from Seattle University with bachelor's degrees in English and physics, and earned his master's degree in science journalism from New York University.

      Eliza Strickland

      Eliza Strickland is a senior editor at IEEE Spectrum, where she covers AI, biomedical engineering, and other topics. She holds a master’s degree in journalism from Columbia University.

      06 May 2019
      IEEE Spectrum
      dronestype:videoEast Africa dronesRwandadelivery dronesZipline360 video

      With 360 video, IEEE Spectrum takes you behind the scenes with one of the world’s first drone-delivery companies. Zipline, based in California, is using drones to deliver blood to hospitals throughout Rwanda. At an operations center in Muhanga, you’ll watch as Zipline technicians assemble the modular drones, fill their cargo holds, and launch them via catapult. You’ll see a package float down from the sky above a rural hospital, and you’ll get a closeup look at Zipline’s ingenious method for capturing returning drones.

      You can follow the action in a 360-degree video in three ways: 1) Watch on your computer, using your mouse to click and drag on the video; 2) watch on your phone, moving the phone around to change your view; or 3) watch on a VR headset for the full immersive experience.

      Keep Reading ↓Show less
      DIYTopicTypeVideo

      A Techie’s Tour of New York City

      Here are some NYC attractions that you won’t find in the guidebooks

      Stephen Cass

      Stephen Cass is the special projects editor at IEEE Spectrum. He currently helms Spectrum's Hands On column, and is also responsible for interactive projects such as the Top Programming Languages app. He has a bachelor's degree in experimental physics from Trinity College Dublin.

      17 Oct 2018
      A Techie's Tour Of NYC
      www.youtube.com
      type:videoNew York CityDIYtourismvideosrocketsNikola Teslahistorytechnologyeventshackerspacenew york citynikola teslanyc tourist videonyc tourist guidetech tour nycvideonyc tech tour

      Do your travel plans include New York City? Are you a techie? If the answer to those questions is yes, let IEEE Spectrum be your guide! We've put together a list of some of our favorite places to visit, including important locations in the history of electrotechnology (New York was once the center of the electrical and electronic world) and places where fun and interesting things are happening today. See where Nikola Tesla lived, check out cutting-edge artists working with technology, or take the kids to see an Atlas and Titan rocket.

      All the locations are accessible via the subway, and many are free to visit. If you do visit, take a selfie and post a link in the comments below.

      Keep Reading ↓Show less
      About IEEEContact & SupportAccessibilityNondiscrimination PolicyTermsIEEE Privacy Policy
      © Copyright 2023 IEEE — All rights reserved. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.