robotics-diffusion-transformer commited on
Commit
ec7b73a
·
verified ·
1 Parent(s): 146c9bc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -41,12 +41,12 @@ tags:
41
 
42
  RDT takes language instruction, RGB images (of up to three views), control frequency (if any), and proprioception as input and predicts the next 64 robot actions.
43
  RDT supports control of almost all robot manipulators with the help of the unified action space, which
44
- includes all the main physical quantities of the robot manipulator (e.g., the end-effector and joint, position and velocity, and base movement).
45
  To deploy on your robot platform, you need to fill the relevant quantities of the raw action vector into the unified space vector. See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for more information.
46
 
47
  **Out-of-Scope**: Due to the embodiment gap, RDT cannot yet generalize to new robot platforms (not seen in the pre-training datasets).
48
  In this case, we recommend collecting a small dataset of the target robot and then using it to fine-tune RDT.
49
- See our repository for a tutorial.
50
 
51
  Here's an example of how to use the RDT-1B model for inference on a robot:
52
  ```python
@@ -78,9 +78,9 @@ model = create_model(
78
  # Load the pre-computed language embeddings
79
  lang_embeddings_path = 'your/language/embedding/path'
80
  text_embedding = torch.load(lang_embeddings_path)['embeddings']
81
- images: List(PIL.Image) = ... # The images from last 2 frame
82
  proprio = ... # The current robot state
83
- # Perform inference to predict the next chunk_size actions
84
  actions = policy.step(
85
  proprio=proprio,
86
  images=images,
 
41
 
42
  RDT takes language instruction, RGB images (of up to three views), control frequency (if any), and proprioception as input and predicts the next 64 robot actions.
43
  RDT supports control of almost all robot manipulators with the help of the unified action space, which
44
+ includes all the main physical quantities of the robot manipulator (e.g., the end-effector and joint, position and velocity, and the wheeled locomotion).
45
  To deploy on your robot platform, you need to fill the relevant quantities of the raw action vector into the unified space vector. See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for more information.
46
 
47
  **Out-of-Scope**: Due to the embodiment gap, RDT cannot yet generalize to new robot platforms (not seen in the pre-training datasets).
48
  In this case, we recommend collecting a small dataset of the target robot and then using it to fine-tune RDT.
49
+ See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for a tutorial.
50
 
51
  Here's an example of how to use the RDT-1B model for inference on a robot:
52
  ```python
 
78
  # Load the pre-computed language embeddings
79
  lang_embeddings_path = 'your/language/embedding/path'
80
  text_embedding = torch.load(lang_embeddings_path)['embeddings']
81
+ images: List(PIL.Image) = ... # The images from last 2 frames
82
  proprio = ... # The current robot state
83
+ # Perform inference to predict the next `chunk_size` actions
84
  actions = policy.step(
85
  proprio=proprio,
86
  images=images,