PokeFlex: A Real-World Dataset of Deformable Objects for Robotics
Deformable objects;Robotics;3D mesh reconstruction.
Yes
N/A.
No ethics review needed.
- Q1: How do you plan to address the limitations in reconstructing fine-grained details for smaller objects in future work?
- Q2: Are there plans to extend the dataset to include thin deformable objects like cloth or ropes, which are significant in many robotic applications?
- Q3: Do you intend to incorporate more complex or varied manipulation protocols beyond poking and dropping to simulate a wider range of real-world interactions?
- Q4: Is there any intention to provide additional annotations (e.g., segmentation masks, keypoints) to facilitate other types of learning tasks?
- Rich, Real-World Dataset: PokeFlex offers a comprehensive multimodal dataset with 3D meshes, point clouds, RGB images, depth maps, and force/torque measurements, captured with a high-quality volumetric system. This real-world data bridges the sim-to-real gap, enhancing applicability in deformable object manipulation.
- Reproducibility and Accessibility: Open-source 3D-printed objects and affordable sensors (Azure Kinect, Intel RealSense) make the dataset accessible and reproducible, while baseline models and evaluation metrics support benchmarking and further robotics research.
The paper introduces PokeFlex, a new dataset focused on real-world deformable objects for robotics applications. The dataset includes:

- Multimodal Data: 3D textured meshes, point clouds, RGB images, and depth maps.
- Objects: 18 deformable objects with varying stiffness and shapes, including everyday items and 3D-printed objects.
- Interaction Protocols: Deformations induced by poking with a robot arm (with force/torque data) and dropping onto a flat surface.
- Purpose: For downstream tasks like online 3D mesh reconstruction and enable applications such as real-world deployment of control methods based on mesh simulations.
- Data Acquisition: A volumetric capture system for 360 deg. reconstruction and integrates lower-cost RGB-D sensors for reproducibility.
- Baseline Models: Shows the use of PokeFlex in online 3D mesh reconstruction using different data modalities and provides evaluation criteria.
- Limited Object Diversity: While the dataset includes 18 objects, the diversity may still be limited compared to the vast range of deformable objects encountered in real-world applications. Notably, the dataset focuses on volumetric objects and excludes thin deformable items like cloth or cables, which are important in areas like garment handling.
- Controlled Interaction Protocols: The poking and dropping protocols are specific and may not capture the full spectrum of possible deformations in less controlled or more complex environments. Additional manipulation actions could enrich the dataset.
- Potential Bias in Data Collection: The use of a transparent acrylic stick for poking and specific dropping heights might introduce biases that limit the generalizability of the dataset to other types of interactions and objects.
- Related Work Coverage: The paper could benefit from a more thorough comparison with existing datasets and approaches in the literature, such as DOFS [1], CEPB [2], and other relevant works on deformable object datasets and manipulation. Discussing how PokeFlex complements or improves upon these datasets would strengthen the contribution.

References:
[1] Zhang, Zhen, et al. "DOFS: A Real-world 3D Deformable Object Dataset with Full Spatial Information for Dynamics Model Learning." arXiv preprint arXiv:2410.21758 (2024).
[2] Tripicchio, Paolo, Salvatore D’Avella, and Carlo Alberto Avizzano. "CEPB dataset: a photorealistic dataset to foster the research on bin picking in cluttered environments." Frontiers in Robotics and AI 11 (2024).
Yes
No ethics review needed.
1. What are the densesyntheticPoint Clouds(5kpoints) and SparsesyntheticPoint Clouds(100points)? Why are the error still large when using synthetic point clouds? 

2. What is NVP? IT should be explained briefly to make the paper self-contained.
The main advantage of this dataset is that it contains paired 3D meshes and contact forces. They further design a deformation mesh reconstruction method which takes the template mesh and force or image of point cloud sequece as input. By using the GT deformation meth for training, the trained model performs good on different trajectoris of the same objects.
In this paper, the authors collected a real-world multimodal dataset of 18 deformable objects. The dataset includes 3D textured meshes, point clouds, RGB images and depth maps.The authors use a volumetric capture system to build the 360degree reconstruction. It includes two types of deformation, poking and dropping. 

The author further use the dataset to train neural network models for deformed mesh reconstruction based on template meshes and different modalites included in the dataset. The neural network model architecture is reasonable and the perforamnce is good on the different trajectories of the same objects.
1. According to Table2, the academic value and the difference between the submsion and PLUSH(Chenetal.,2022) should be further discussed, since it is not difficult to reconstruct the meshes from point clouds. Why do we need the mesh? What other application can it enable on top of deformation reconstruction?

2. The generalizability of the network on unseen objects is not evaluated, which may be the most valuable part of the model. If the model can only work on trained objects, the usefulness of this method is very limited.

3. The technical novelty of the network model is limited and not validated through ablation studies.
Yes
It is important to specify whether the objects used in the dataset are under open licenses. For instance, as noted earlier, the pizza model appears to have a license that restricts redistribution. Clarifying the licensing status of each object in the dataset would help ensure compliance with legal guidelines and provide transparency to other researchers regarding any limitations on dataset sharing and reuse.
Yes, Legal compliance (e.g., GDPR, copyright, terms of use)
1) In Equation 3, the value of epsilon is not specified. What value does epsilon take to define that the point p_i​ is close enough to the contact point t?
2) In Equation 3, how was the value of 0.2 chosen, and what is the reasoning behind this choice?
3) Why is the ROI loss weighted by 0.5 in the total loss calculation?
4) In Section 4.2, it is mentioned that the train-validation split was generated by randomly selecting one sequence per object. Are the sequences consistent across all objects? For the multi-object training split (Section "Learning from different data modalities"), how was the split handled? Did you consider a sequence in the validation set for one object that fell into the training set for another?
5) For the single-object training described in the section "Learning from RGB images of different cameras", could you provide results for additional objects? Results from only one object do not yield a statistically significant sample for generalizing findings.
6) For Figure 11, could you include data for additional objects (potentially in the supplementary materials)? In this case, what is the training setup—is the dataset composed of 20k samples, or just the 8.4k active paired poking frames? If the dataset used was the full 20k samples, how was it balanced to avoid an overrepresentation of frames without poking compared to those with poking?
7) Still regarding Figure 11, the x-axis represents distance values—how many samples are averaged at each value? Are there more samples at 0.1 than at 0.4? If so, how is this imbalance managed during training and validation?
8) Were environmental conditions, such as lighting, explicitly considered during dataset acquisition? Was any analysis performed to evaluate how variations in lighting or surrounding conditions might influence the networks used?
The PokeFlex dataset stands out as an original contribution to the field. It features a diverse set of objects, including 3D-printed and everyday objects. This variety enriches the dataset and makes it applicable to a wide range of research scenarios. The inclusion of different types of input data—such as 3D textured meshes, point clouds, RGB images, and depth maps—enhances its utility for various downstream tasks.
The paper clearly conveys the significant effort involved in acquiring this dataset, highlighting the meticulous process behind its compilation. Additionally, the introduction of an innovative deep learning model to address 3D mesh reconstruction challenges is particularly noteworthy, showcasing the authors' commitment to advancing the field. Furthermore, the paper is well-structured and easily readable, making it accessible to a broad audience.
The paper presents PokeFlex, a dataset that features real-world paired and annotated multimodal data, including 3D textured meshes, point clouds, RGB images, and depth maps. This dataset aims to enhance various downstream tasks and to demonstrate its effectiveness in online 3D mesh reconstruction.
PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping these objects onto a surface or by interacting with them using a robot arm, with corresponding interaction forces and torques recorded. Additionally, the authors introduce a novel deep learning model to tackle 3D mesh reconstruction using diverse input data types.
The paper presents results for only a limited subset of the 18 objects in the PokeFlex dataset—showing performance for just one object in some instances, five in others, and graphs for three. To better assess the dataset's overall effectiveness, it would be beneficial to provide results for all objects. This comprehensive analysis would establish a solid baseline for future comparisons.
Moreover, the choice of a transparent acrylic stick used to poke was not adequately justified or discussed. The implications of this choice on RGB and depth data need more attention, particularly regarding how reflection and refraction may distort images and how time-of-flight depth readings may be inaccurate around the stick.
To further evaluate the dataset's effectiveness, it would be helpful to have a benchmark model trained on the dataset itself to determine its robustness and construction quality. While the paper mentions the "dropping" method, the experiments reported focus solely on "poking." Including results from the dropping method would provide additional insights into the dataset's applications.
Additionally, the model that utilizes multiple data modalities only explored combinations of images and robot data. It would be interesting to see how other combinations of data types perform. 
Moreover, in Equation 3, the threshold epsilon is not defined, nor is there clarity on how values like 0.2 or 0.5 are determined. Addressing these points would enhance the clarity and rigor of the research.
Additionally, several citations in the references appear to be incorrect or improperly formatted, specifically the third, thirteenth, fourteenth, eighteenth, and twenty-ninth citations.
Another point concerns object licensing: one of the objects (Stylized pizza slice) in the dataset is reportedly subject to a non-distributable license, yet this restriction is not noted. This oversight may lead to potential misuse or distribution issues.
Finally, neither the full dataset nor the trained deep learning model is currently available, which limits reproducibility. The sample dataset provided in the supplementary materials includes only one object, a dice, rather than a representative subset or the entire collection. Making the full dataset and model accessible would not only improve transparency but also enable a more comprehensive evaluation by the research community.
Yes
No ethnic review needed.
No ethics review needed.
+ Groundtruth mesh: If my understanding is correct, the ground truth mesh is obtained using an MVS (Multi-View Stereo) system. The paper's discussion of this point is somewhat unclear.
+ Multimodal: In typical robotic tasks, the position and action of the end-effector are also crucial. Here, the authors use a transparent stick for poking, but the dataset should incorporate the point cloud or mesh of the stick into the trajectory. This can be simply achieved by combining the robot's position with the mesh of the stick. Without this integration, it is difficult to effectively utilize the robot action information provided in the dataset.
+ Error of UR force and torque: The force and torque errors of the UR robot are significantly higher compared to other robots, such as the Franka Emika robot, as noted in the official product sheets.[1] The authors should discuss the impact of these errors on the experiments and results presented in the paper. 

[1] https://assets.ctfassets.net/oxcgtdo88e20/4zQbvCXiDH2RzaaiiuwWci/ca242c85669825e8c25ff3467f8774da/UR5e_Product_Factsheet_EN_WEB.pdf
+ RealWorld dataset: The most prominent feature of this paper is that it presents a real-world dataset featuring real-world meshes. Benchmarks for real-world meshes in robotics are relatively scarce. Additionally, the paper demonstrates good reproducibility, providing detailed 3D printing information in the appendix.
+ Multimodal and Rich: Furthermore, this dataset is a multimodal dataset that, in addition to using high-cost acquisition systems, incorporates the commonly used RealSense camera as an eye-in-hand sensor and the Kinect as an eye-on-base sensor. The settings align well with robotic tasks and visual tasks within the field.
+ Comprehensive evaluation on reconstruction task: Finally, the paper conducts detailed experiments on reconstruction tasks, including the use of RGB data and multimodal data.
The paper mainly introduce a deformable objects realworld datasets featuring real-world paired and annotated multimodal data. The datasets can be mainly used for online 3D mesh reconstruction. The dataset has different data modalities and is primarily collected using a professional capture system, with the collection process also incorporating cameras such as Kinect and RealSense, and simultaneously synchronizing with the robotics system. The paper further shows the application of this datasets on mesh reconstruction and has comprehensive evaluation of learning-base reconstruction.
+ The purpose of the dataset: One major issue with this paper is that it seems there are few other application tasks beyond online mesh reconstruction for which the dataset can be utilized.  Although online mesh reconstruction is an interesting computer vision task in itself, it appears that there are few practical applications for online mesh reconstruction. Therefore, this work seems to fall short of expectations in various fields. I will elaborate on this point from three different areas.
     + robotics: In robotics, especially in the field of deformable object manipulation, meshes are rarely used; point clouds are more commonly employed[2][3][4]. The main challenges in this area are often related to occlusion and pose estimation issues. For example, Garment Tracking [1] addresses the problem of garment point cloud completion, but this work does not seem to tackle occlusion problems, thus reducing the significance of online mesh reconstruction in this context.
     + CV: This task has limited difficulty in the computer vision field, primarily because the occlusion problem is less significant and the deformation of plush toys is minimal compared to garments. For instance, in dropping tasks, there appear to be only translational and rotational changes. Real-time 6D pose detection has already been extensively studied, so this part of mesh reconstruction can be easily addressed using mesh + pose estimation techniques.
+ Number of object: The scale of 18 objects is small, and each object providing only two trajectories (poking and dropping) results in a limited data size. This makes it difficult to serve as a large-scale evaluation and can only be used for real-world fine-tuning. Papers in similar categories  such as garmentlab[2] have scanned nearly 100 objects, including at least 20 complex plush toys.
+ Object Selection: The deformable objects chosen for this paper are somewhat too simple. There are many more complex plush toys, such as teddy bears, which could also be used for these tasks and would better reflect real-world scenarios. These more complex objects often involve occlusion issues, making the work more meaningful. Most of the objects selected in this paper are simple shapes like cubes or cylinders, which limits the scope and applicability of the task.
+ Action Selection: The actions chosen, poking and dropping, are overly simplistic and lack practical value, especially in the robotics domain. The paper could benefit from including more complex actions, such as pick-and-place, which would add significant practical value and better reflect real-world robotic tasks.
+ Simulation Evaluation: The authors mention in the paper that existing simulation methods require real-world fine-tuning. However, simulations actually perform very well in tasks such as dropping and poking, for example, using the FEM algorithm in IsaacSim. The authors need to include validation within simulators to verify whether the meshes scanned in the paper can be used for simulation and to conduct a thorough analysis of the differences between current simulation algorithms and real-world conditions. This would strengthen the setting of the paper.
+ Action Protocol: The action protocol in this paper is vague. I recommend that the authors follow established protocols from datasets like YCB or GarmentLab[2] to refine and standardize their action protocols. This would enhance the clarity and reproducibility of the experiments.

[1] GarmentTracking: https://github.com/xiaoxiaoxh/GarmentTracking

[2] GarmentLab: https://garmentlab.github.io/

[3] UniGarmentManip: https://warshallrho.github.io/unigarmentmanip/
Novel dataset for deformable objects.
@inproceedings{
anonymous2024pokeflex,
title={PokeFlex: A Real-World Dataset of Deformable Objects for Robotics},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=XwibrZ9MHG},
note={under review}
}
Data-driven methods have shown great potential in solving challenging manipulation tasks, however, their application in the domain of deformable objects has been constrained, in part, by the lack of data. 
To address this, we propose PokeFlex, a dataset featuring real-world paired and annotated multimodal data that includes 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks such as online 3D mesh reconstruction, and it can potentially enable underexplored applications such as the real-world deployment of traditional control methods based on mesh simulations.
To deal with the challenges posed by real-world 3D mesh reconstruction, 
we leverage a professional volumetric capture system that allows complete 360° reconstruction. PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping objects onto a flat surface or by poking the objects with a robot arm. Interaction forces and torques are also reported for the latter case. 
Using different data modalities, we demonstrated a use case for the PokeFlex dataset in online 3D mesh reconstruction. We refer the reader to our  [website](https://anonymized-pokeflex-dataset.github.io/) or the  [password protected supplementary material](https://drive.google.com/drive/folders/1d8iNoJZ0dUVlzP6XxP7xwGPhdVtwQ7du) for demos and examples of our dataset (password in pdf).
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Deformable objects Robotics 3D mesh reconstruction.
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1c0ea7a5bdd6b370bb91e8035cc40b648a21a002.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
PokeFlex: A Real-World Dataset of Deformable Objects for Robotics
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission