This project builds upon and enhances the dexterous hand teleoperation capabilities of AnyTeleop. By integrating advanced pose estimation and retargeting optimizations, it offers a accessible and stable solution for robotic hand control.
- 🚀 Robust to Occlusion: Handles partial hand occlusions effectively during operation.
- 🦾 6-DoF Wrist Pose: Supports accurate calculation and mapping of wrist pose.
- 💻 Minimal Hardware: Requires only a single RGB camera and a standard laptop.
Create a clean conda environment with Python 3.9:
conda create -n retarget python=3.9
conda activate retargetInstall dex-retargeting which provides the optimization solvers for retargeting hand keypoints to robot joint positions:
pip install dex_retargeting loguruClone this repository and install the required dependencies:
git clone --recursive https://github.com/Ralph-cong/AnyTeleop.git
cd AnyTeleop
pip install -e ".[example]"
pip install tyro pyyaml sapien==3.0.0b0 pyrealsense2 numpy-quaternionThis project uses modified versions of WiLoR-mini (for hand detection/3D estimation) and Ultralytics.
Install WiLoR-mini:
# Clone into thirdparty directory
git clone https://github.com/warmshao/WiLoR-mini.git thirdparty/WiLoR-mini
# Install in editable mode
pip install -e thirdparty/WiLoR-miniIf you encounter library linking errors during execution, you may need to export the library path:
export LD_LIBRARY_PATH=$HOME/anaconda3/envs/retarget/lib/python3.9/site-packages/cmeel.prefix/lib:$LD_LIBRARY_PATHRetarget human hand motion to a floating robot hand. 👉 View Tutorial
Retarget human hand motion to control a robot arm + hand system to complete manipulation tasks in ManiSkill. 👉 View Tutorial
This project is developed based on: