The TWIST system: Motion capture technology (MoCap) revolutionizes the control of humanoid robots – Image: Xpert.Digital
Teleoperated Whole-Body Imitation System: Real-time human-robot interaction will change robotics
Human movements for robots: The potential of the TWIST system
Scientists have achieved a significant breakthrough in the development of teleoperation systems for humanoid robots. By using motion capture technology, humanoid robots can now perform human-like movements in real time. This innovation enables precise and intuitive control of robots, representing an important step towards the development of robots with whole-body dexterity. Particularly noteworthy is the new system TWIST (Teleoperated Whole-Body Imitation System), which transmits a person's complete body movements to a robot, thus ushering in a new era of human-robot interaction.
Related to this:
The fundamentals of motion-capture-based teleoperation
Teleoperation refers to the remote control of machines and is of particular importance in the field of robotics. Telerobotic systems are used when the workspace is too far away, too small, too large, or too dangerous for humans. The spatial decoupling between human (operator) and robot (teleoperator) enables applications in various fields such as minimally invasive surgery, bomb disposal, and space applications.
Motion capture technology (MoCap) forms the basis for modern teleoperation systems. This technology enables detailed recordings and simulations of human movement, allowing for the digitization of individuals or entire groups of people. The captured movements are intelligently processed and can be used to animate bodies and their movements.
How motion capture technology works
Motion capture technology precisely tracks and records the body movements of real people using a special suit equipped with markers and optical systems. This process collects movement data from all body parts – not just arms, hands, legs, and feet, but also the torso, hips, and head. This comprehensive data is then transformed into commands that humanoid robots can execute using artificial intelligence (AI).
The TWIST system: A breakthrough in robot teleoperation
The TWIST system, developed at Stanford University and Simon Fraser University, represents a significant advance in humanoid robot teleoperation. It combines motion capture technology with methods of reinforcement learning and imitation learning.
“We want humanoids to possess the same level of whole-body dexterity as humans,” explains Yanjie Ze, lead author of the TWIST study. “Imagine a messy kitchen. Humans can hold things with both hands and use their feet to move obstacles, such as a basket on the floor. Humans can also open the door with the sides of their body or their elbows. We want humanoids to be able to do the same by directly imitating humans.”
Technical implementation of TWIST
The TWIST system comprises three essential components:
- Data acquisition and retargeting: Through offline and online retargeting, human movements are adapted for the robot. This is achieved through optimized transmission of 3D joint positions and orientations, with body orientation and foot placement also being adjusted in real time.
- Controller training in simulation: TWIST uses a two-stage approach with a “teacher-student” methodology:
- The “Teacher” controller has privileged access to future reference movements, which allows it to plan smoother movements.
- The “student” controller is trained through a combination of Reinforcement Learning (RL) and Behavior Cloning (BC) and can only access current movement information.
- Full-body controller: The trained controller enables the robot to utilize all degrees of freedom while maintaining balance. This results in more natural and human-like movements.
In tests with Unitree's humanoid G1 robot, researchers found that it was sufficient to capture whole-body movements and transfer them precisely to the robot's joints, ensuring that the movements of the different limbs were coordinated.
Related to this:
Challenges in humanoid teleoperation
The development of teleoperation systems for humanoid robots presents researchers with several complex challenges:
Bridging the embodiment gap
A key challenge is bridging the “embodiment gap”—the anatomical differences between humans and robots. Because robots have different proportions, joint configurations, and physical properties than humans, a direct transfer of human movements is not readily possible.
Balance and whole-body coordination
Humanoid whole-body tracking requires not only the precise control of individual joints, but also the dynamic maintenance of balance during complex movements. Conventional teleoperation systems often focus only on isolated movements such as locomotion or manipulation, whereas TWIST enables coordinated whole-body movements.
Latency and Sensory Feedback
Teleoperation systems must overcome problems such as latency (time delay) and limitations in sensory feedback. These factors can impair the synchronization of human actions with robotic responses.
Diverse applications of motion-capture teleoperation
Motion-capture-based teleoperation of humanoid robots opens up numerous application possibilities:
Dangerous situations and rescue operations
In hazardous environments, teleoperated robots can be used instead of humans, for example in explosive ordnance disposal (EOD). Between 2015 and 2020, there were approximately 2,000 EOD operations annually in the United Kingdom alone, highlighting the need for safe alternatives.
Complex manipulation tasks
Humanoid robots can perform complex manipulation tasks via teleoperation, for example in unstructured environments such as kitchens or workshops. Their ability to coordinate the use of their entire body, including arms, hands, legs, and feet, offers crucial advantages in this context.
Social robotics and expressiveness
For humanoid social robots, the ability to perform expressive movements is essential. The OCRA system (Optimization-based Customizable Retargeting Algorithm), developed at the MPI, enables real-time motion mapping between different kinematic chains, resulting in intuitive and human-like movements.
Alternative approaches and comparison of different systems
Besides TWIST, there are various other approaches for motion-capture-based teleoperation:
IMU-based systems
Some researchers are using IMU-based (Inertial Measurement Unit) motion-capture systems, which are portable and less expensive than optical systems. This technology is used, for example, for the teleoperation of loco-manipulation tasks, which combine locomotion and manipulation.
Neural network-based approaches
An alternative approach uses neural networks to learn a mapping between the sensor data from the motion-capture suit and the angular positions of the robot actuators. This method does not require a prior analytical or mathematical model of the robot and can therefore be applied to various human-robot pairings.
Systems for specific body parts
In addition to whole-body teleoperation systems, there are also specialized systems that focus on specific body parts, such as dual-hand motion capture. These systems play an important role in the precise control of bionic bimanual robots for delicate manipulation tasks.
Related to this:
Recent progress and future prospects
The development of teleoperation systems for humanoid robots is progressing rapidly. In addition to TWIST, researchers have recently presented other innovative systems:
H2O: Human to Humanoid
The H2O system enables real-time teleoperation of a fully humanoid robot using only an RGB camera. It utilizes an RL-based frame and a “sim-to-data” process to filter and select suitable movements for humanoid robots.
AR-supported teleoperation
Researchers are also investigating how augmented reality (AR) can support motion capture-based teleoperation. By visualizing a virtual reference of the human arm alongside the robot arm, users can better understand the motion mapping.
AI and Motion Capture: The Future of Human-Robot Interaction
Motion-capture-based teleoperation of humanoid robots has advanced considerably in recent years. Systems like TWIST represent a significant step forward by enabling robots to perform human-like, full-body movements in real time.
The combination of motion-capture technology and advanced AI methods such as reinforcement learning and behavior cloning opens up new possibilities for human-robot interaction. Humanoid robots can now perform not only isolated movements, but also coordinated whole-body actions, enabling greater dexterity and expressiveness.
In the future, these technologies could significantly expand the use of humanoid robots in hazardous environments, for complex manipulation tasks, and in social contexts. The continuous improvement in the precision, robustness, and user-friendliness of teleoperation systems will help to further reduce the gap between human capabilities and robotic execution.
Related to this:
Your global marketing and business development partner
☑️ Our business language is English or German
☑️ NEW: Correspondence in your native language!
Konrad Wolfenstein
I and my team are happy to be available to you as your personal advisor.
You can contact me by filling out the contact form here simply call me at +49 7348 4088 965. My email address is wolfenstein@xpert.digital:or
I'm looking forward to our joint project.
