In recent years, researchers have developed more and more computing technologies to enable robots to have human-like capabilities. However, most of the technologies developed so far only focus on artificially reproducing vision and touch, while ignoring other sensations such as auditory perception.

A research team at Carnegie Mellon University (CMU) recently conducted a study to explore the possibility of using sound to develop robots with more advanced sensing capabilities. Their paper was published in "Robotics: Science and Systems" and introduced the largest sound action visual data set compiled so far. This data set is used as a robotic platform called Tilt-Bot (the main body is Sawyer 7 DOF collaborative robotic arm) collected and interacted with a variety of objects.

"In robot learning, we usually only use visual input for perception, but humans have more than just vision." said Lerrel Pinto, one of the researchers who conducted this study. "Sound is a key part of learning and understanding our physical environment. Therefore, we asked a question: Can robots learn from us in sound judgments? To answer this question, we created Tilt-Bot, which can interact with objects. And collect objects. Interactive large-scale audiovisual data sets."

Essentially, Tilt-Bot is a robotic platform composed of Sawyer+ automatic pallets, which can tilt objects until they hit a wall of the pallet. Pinto and his colleagues placed contact microphones on the wall of the robot tray to record the sound of objects hitting the wall, and used overhead cameras to intuitively capture the movement of each object.

Researchers collected visual and audio data of more than 15,000 Tilt-Bot interactions with 60 different subjects. This allows them to compile new image and audio data sets, which can help train robots to establish associations between actions, images and sounds.

Pinto and his colleagues used this data set in their paper to explore the relationship between sound and motion in robotic applications, and collected many interesting findings. First, they found that analyzing the sound recordings of objects moving and hitting surfaces can allow the machine to distinguish between different objects, such as metal screwdrivers and metal wrenches.

Pinto explained: "An exciting preliminary result of our research is that you can identify close to 80% of object types based on sound alone." "We also showed that machines can learn audio-based object representations, thereby Help solve robot tasks in the future. For example, when recognizing the sound of an empty wine glass, the robot can understand that operating it will require actions different from the operation. It will play a role when the wine glass is filled."

Interestingly, recordings can sometimes provide more valuable information than visual representations to solve robotic tasks, because they can also be used to effectively predict the future motion of objects. In a series of experiments using objects that the robot did not encounter during training, they found that the audio embeddings collected when the robot interacts with these objects can predict the forward model (how to best manipulate the object in the future) by 24 % Instead of passive visual embedding.

The data set compiled by this group of researchers can eventually help develop robots that can choose their actions and object manipulation strategies based on audio recordings and images collected around them. Pinto and his colleagues are now planning further research to explore the potential of sound analysis in creating robots with more advanced features.

Pinto said: "This work is only the first step in the overall integration of sound into robotics." "In future work, we will study more practical applications of sound and motion."

References: Gandhi D, Gupta A, Pinto L. Swoosh! Rattle! Thump!--Actions that Sound[J]. arXiv preprint arXiv:2007.01851, 2020.