robotics-diffusion-transformer
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -41,12 +41,12 @@ tags:
|
|
41 |
|
42 |
RDT takes language instruction, RGB images (of up to three views), control frequency (if any), and proprioception as input and predicts the next 64 robot actions.
|
43 |
RDT supports control of almost all robot manipulators with the help of the unified action space, which
|
44 |
-
includes all the main physical quantities of the robot manipulator (e.g., the end-effector and joint, position and velocity, and
|
45 |
To deploy on your robot platform, you need to fill the relevant quantities of the raw action vector into the unified space vector. See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for more information.
|
46 |
|
47 |
**Out-of-Scope**: Due to the embodiment gap, RDT cannot yet generalize to new robot platforms (not seen in the pre-training datasets).
|
48 |
In this case, we recommend collecting a small dataset of the target robot and then using it to fine-tune RDT.
|
49 |
-
See our repository for a tutorial.
|
50 |
|
51 |
Here's an example of how to use the RDT-1B model for inference on a robot:
|
52 |
```python
|
@@ -78,9 +78,9 @@ model = create_model(
|
|
78 |
# Load the pre-computed language embeddings
|
79 |
lang_embeddings_path = 'your/language/embedding/path'
|
80 |
text_embedding = torch.load(lang_embeddings_path)['embeddings']
|
81 |
-
images: List(PIL.Image) = ... # The images from last 2
|
82 |
proprio = ... # The current robot state
|
83 |
-
# Perform inference to predict the next chunk_size actions
|
84 |
actions = policy.step(
|
85 |
proprio=proprio,
|
86 |
images=images,
|
|
|
41 |
|
42 |
RDT takes language instruction, RGB images (of up to three views), control frequency (if any), and proprioception as input and predicts the next 64 robot actions.
|
43 |
RDT supports control of almost all robot manipulators with the help of the unified action space, which
|
44 |
+
includes all the main physical quantities of the robot manipulator (e.g., the end-effector and joint, position and velocity, and the wheeled locomotion).
|
45 |
To deploy on your robot platform, you need to fill the relevant quantities of the raw action vector into the unified space vector. See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for more information.
|
46 |
|
47 |
**Out-of-Scope**: Due to the embodiment gap, RDT cannot yet generalize to new robot platforms (not seen in the pre-training datasets).
|
48 |
In this case, we recommend collecting a small dataset of the target robot and then using it to fine-tune RDT.
|
49 |
+
See [our repository](https://github.com/thu-ml/RoboticsDiffusionTransformer) for a tutorial.
|
50 |
|
51 |
Here's an example of how to use the RDT-1B model for inference on a robot:
|
52 |
```python
|
|
|
78 |
# Load the pre-computed language embeddings
|
79 |
lang_embeddings_path = 'your/language/embedding/path'
|
80 |
text_embedding = torch.load(lang_embeddings_path)['embeddings']
|
81 |
+
images: List(PIL.Image) = ... # The images from last 2 frames
|
82 |
proprio = ... # The current robot state
|
83 |
+
# Perform inference to predict the next `chunk_size` actions
|
84 |
actions = policy.step(
|
85 |
proprio=proprio,
|
86 |
images=images,
|