As far as I know, the universe does not have a desperate need for robot puppeteers, and considering the difficulty of making even a halfway decent robot puppeteer, you’d think that any sensible roboticist would keep well clear of the problem. But some folks over at ETH Zurich decided that they’d have a crack at it anyway, and they started by describing why they’d likely be better off if they hadn’t:
Marionettes are underactuated, high-dimensional, highly non-linear coupled pendulum systems. They are driven by gravity, the tension forces generated by a small number of cables, and the internal forces arising from mechanical articulation constraints. As such, the map between the actions of a puppeteer and the motions performed by the marionette is notoriously unintuitive, and mastering this unique art form takes unfaltering dedication and a great deal of practice. Our goal is to enable autonomous robots to animate marionettes with a level of skill that approaches that of human puppeteers.
I’m not much of a puppeteer myself, but this looks not bad at all, considering that the ABB YuMi robot is missing quite a few degrees of freedom in its hands. For context, here’s someone who has mastered this unique artform through unfaltering dedication and a great deal of practice, master puppeteer Scott Land:
The ETH Zurich project can’t yet animate a complex marionette, but it’s a respectable showing with the dragon, I think. As input, all the robot needs to know is the design of the puppet at the target motion you want the puppet to make. While moving the puppet in real life, the robot is continuously simulating its motions over the next second while iteratively optimizing to try to get the puppet to move the way it’s supposed to.
The usefulness of this research, thankfully, is not constrained to puppets:
Our long term goal is to enable robots to manipulate various types of complex physical systems – clothing, soft parcels in warehouses or stores, flexible sheets and cables in hospitals or on construction sites, plush toys or bedding in our homes, etc – as skillfully as humans do. We believe the technical framework we have set up for robotic puppeteering will also prove useful in beginning to address this very important grand-challenge.
“PuppetMaster: Robotic Animation of Marionettes,” by Simon Zimmermann, Roi Poranne, James M. Bern, and Stelian Coros from ETH Zurich, was presented at Siggraph 2019.
[ Paper ]