Problem Statement (Challenge)

Our challenge is to use AI to facilitate effective human-robot collaboration. Our work will employ gesture and speech recognition and potentially other communication modes, such as contact interactions, to determine a human’s actions and intent and use these to determine appropriate robot actions in response to the human.

Effective human-robot collaboration can have far-reaching impacts. It will enable improved productivity in manufacturing and material handling while empowering rather than replacing human workers. It will enable robots to provide the means to care for aging populations around the world. Robots will be able to provide general assistance in the home, at the store, and in emergency situations.

The problem of human-robot collaboration is timely because the basic building blocks for effective human-robot collaboration have become widely available in the last five years. We can purchase robot arms that are human-safe and capable of force interaction for less than an economy car. We have cheap sensors for 3D vision and audio, and the cost for contact and force sensing is also coming down. We have robust speech to text transcription, and gesture tracking can be accomplished with low cost sensors. We also have AI technologies, including machine and deep learning and the computing power needed to make use of the large data sets that can now be generated. We are currently seeing the broad adoption of personal AI assistance, such as Siri and Google Now. These provide information assistance through natural communication. We are proposing to achieve the same for communication in a physical, spatial context to provide physical assistance. All the building blocks are in place to develop an AI system that can facilitate natural human-robot collaboration in the physical domain.

Our work has far reaching applications outside of robotics. Any application where monitoring human behavior is important would likely benefit from our work. Examples include security applications and education.

Competition Plan Summary

Effective human-robot collaboration can have far-reaching impacts in applications such as manufacturing, material handling, home care, service industries, and  entertainment. By keeping humans in the loop through collaboration, we can realize the benefits of machine precision and endurance in addition to human creativity and flexibility.  Our strategy for developing an AI mediated system for facilitating human-robot collaborations is fundamentally data-driven. We will create software and engineer a system that will enable us to collect the large data sets we need for our AI system to learn to mediate human-robot communication in a way that is natural for humans and does not require any training by the human user.

We considered several domains for this project, including tasks around the home and manufacturing assembly tasks. We decided on the use of alphabet wooden blocks as it provides a sufficiently realistic environment but simplifies the robotic aspects of the project, such as motion planning, grasping, and manipulation. This allows us to focus our efforts on the AI challenges involved in human-robot collaboration. Use of alphabet wooden blocks also makes our test suite easy and inexpensive for others to replicate.

The aim of our project is to make it possible for novice users of a robot to complete a series of alphabet block assembly tasks under controlled conditions. The tasks in our test suite require varying degrees and types of human-robot interactions. Since our focus in developing this test suite is AI rather than robotics, we have attempted to make the tasks and performance metrics independent of the performance characteristics of the particular robot used.  If successful:

  • We will have a compelling demonstration of effective human-robot collaboration involving a physical task.
  • We will have AI software that is easily transferable to new equipment and new tasks in different domains.
  • We will have a canonical test suite that can serve as a tool to advance the field of human-robot collaboration.