Lecture

Reinforcement Learning for Robotics

11 Aug 2024 • Richard Kuo

Introduction to Reinforcement Learning for Robot / Drone.

AI-Robots

Figure AI

Sanctuary AI

ADAM

Paper: ADAM: a robotic companion for enhanced quality of life in aging populations

End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation

Embodied AI

Blog: Overview of Embodied Artificial Intelligence
Embodied AI is the field for solving AI problems for virtual robots that can move, see, speak, and interact in the virtual world and with other virtual robots — these simulated robot solutions are then transferred to real world robots
The simulated environments for Embodied AI training: SUNCG, Matterport3D, iGibson, Replica, Habitat, and DART

送餐路徑生成：從文件或上層發佈訊息獲取預先定義的路徑。
編輯航線路徑點：清除路徑中不合適的航線路徑點。
MFAC無模型自適應控制之航段管制：自動調整送餐路徑之導航點之間的航段長度，依序共分成路徑跟隨之依據以及MFAC無模型自適應控制之應用。
DWA之區域路徑傳遞：依照MFAC調整之結果，產出相關生成路徑，並以DWA進行區域設定。
自走車基於DDPG的室內路徑規劃

Paper: Long-Range Indoor Navigation with PRM-RL

Gym-Gazebo

Code: erlerobot/gym-gazebo

DART (Dynamic Animation and Robotics Toolkit)

Dartsim/dart

Python bindings: dartpy, pydart2 (deprecated)
OpenAI Gym with DART support: gym-dart (dartpy based), DartEnv (pydart2 based, deprecated)

PyBullet-Gym

PyBullet Quickstart Guide

code: rkuo2000/pybullet-gym

installation

pip install gym
pip install stable-baselines3
git clone https://github.com/rkuo2000/pybullet-gym
export PYTHONPATH=$PATH:/home/yourname/pybullet-gym

Train
python train.py Ant 10000000

Enjoy with trained-model
python enjoy.py Ant

Enjoy with pretrained weights
python enjoy_Ant.py
python enjoy_HumanoidFlagrunHarder.py (a copy from pybulletgym/examples/roboschool-weights/enjoy_TF_*.py)

PyBullet-Robots

env_name = “AtlasPyBulletEnv-v0”
atlas_v4_with_multisense.urdf

OP3 Soccer

Paper: Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

RoboCar Gym

Pybullet-RoboCar

Blog:
Creating OpenAI Gym Environments with PyBullet (Part 1)
Creating OpenAI Gym Environments with PyBullet (Part 2)

Quadruped Gym

Motion Imitation

Code: TF 1.15

Code: PyTorch
For Training:
python motion_imitation/run_torch.py --mode train --motion_file 'dog_pace.txt|dog_spin.txt' \ --int_save_freq 10000000 --visualize --num_envs 50 --type_name 'dog_pace'
For Testing:
python motion_imitation/run_torch.py --mode test --motion_file 'dog_pace.txt' --model_file 'file_path' \ --encoder_file 'file_path' --visualize

Rex: an open-source quadruped robot

Code: nicrusso7/rex-gym

Drones Gym

PyBullet-Gym for Drones

Installation

sudo apt install ffmpeg
pip install numpy pillow matplotlib cycler
pip install gym pybullet stable_baselines3 ray[rllib]
git clone https://github.com/rkuo2000/gym-pybullet-drones.git
cd gym-pybullet-drones

Train & Enjoy
python train.py # modify train.py for different env, algorithm and timesteps
python enjoy.py # modify enjoy.py for different env
Fly using DSLPIDControl.py:（PID飛行）
python examples/fly.py --num_drones 1
To learn take-off:（起飛）
python examples/learn.py
compare.py which replays and compare to a trace saved in files/example_trace.pkl

Experiments
cd experiments/learning

env : hover, takeoff, flythrugate, tune（旋停, 起飛, 穿越, 調整）
algo: a2c, ppo, sac, td3, ddpg

To learn hover:（旋停）
python singleagent.py --env hover --algo a2c

To visualize the best trained agent:
python test_singleagent.py --exp ./results/save-hover-a2c

For multi-agent RL, using rllib:
python multiagent.py --num_drones 3 --env hover --algo a2c --num_workers 2

Flightmare

Flightmare is a flexible modular quadrotor simulator.

Introduction
Prerequisites
Install Python Packages
Install ROS

running ROS

roslaunch flightros rotors_gazebo.launch

flighRL

cd /path/to/flightmare/flightrl
pip install .
cd examples
python3 run_drone_control.py --train 1

AirSim

Assistive Gym

Paper: Assistive Gym: A Physics Simulation Framework for Assistive Robotics

Four collaborative robots (PR2, Jaco, Baxter, Sawyer)
Support for the Stretch and PANDA robots

Code: Healthcare-Robotics/assistive-gym

Assistive VR Gym

Paper: Assistive VR Gym: Interactions with Real People to Improve Virtual Assistive Robots
Code: Healthcare-Robotics/assistive-vr-gym

Learning Dexity

Dexterous Gym

DexPilot

Paper: DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System

TriFinger

Paper: TriFinger: An Open-Source Robot for Learning Dexterity

Code: TriFinger Robot Simulation

Multi-Task Reset-Free (MTRF) Learning

Paper: Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention

Dexterous Anthropomorphic Robotic Hand

Blog: Robotic hand can crush beer cans and hold eggs without breaking them

Paper: Integrated linkage-driven dexterous anthropomorphic robotic hand

Robotic Telekinesis

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Fixed-Finger Gripper

Paper: F1 Hand: A Versatile Fixed-Finger Gripper for Delicate Teleoperation and Autonomous Grasping

Learning Diverse Dexterous Grasps

Paper: Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimization

ViLa

Blog: https://bangqu.com/9Fa2ra.html

Mobile ALOHA

Paper: Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation
Github: https://github.com/MarkFzp/mobile-aloha
Code: https://github.com/MarkFzp/act-plus-plus

ALOHA 2

Paper: ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation

ALOHA unleashed

Paper: ALOHA Unleashed: A Simple Recipe for Robot Dexterity
![](<iframe width="942" height="530" src="https://www.youtube.com/embed/1EAsn71O9yA" title="ALOHA Unleashed: Autonomous Policies" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>)

DemoStart

Paper: DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots
Blog: Our latest advances in robot dexterity

ReKep

Paper: ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Code: https://github.com/huangwl18/ReKep

This site was last updated November 15, 2024.