Mouse Lockbox Dataset: Behavior Recognition of Mice Solving Mechanical Puzzles
dataset;video;action classification;behavior;mice;machine learning;computer vision;neuroscience;computational neuroscience
Yes
No ethics review needed.
A few things in the introduction that would be good to clarify in the text.

L55 -- you say that it is a disadvantage that poses are low dimensional. But there are many scenarios where this is advantage.
L58 -- I don't understand what you mean when you write, "It impedes mapping of keypoint locations from different perspectives into a common coordinate system."
L59-60 -- there is actually an ideal set of keypoint locations -- skeletal joints 
L107 -- "the more recent works" are actually older than some of the ones you cite in the previous sentence

Relatedly, you spend a lot of the intro saying how bad it is to use keypoints for behavioral analysis, but then go on to use keypoints for your benchmark action recognition results. What am I missing?

Please provide a clear breakdown of how much of your dataset video time is the animals engaged with the puzzle. What is your definition of "playtime"? I would prefer if these data were in the table and not in a ring/pie chart that is harder to read. 

How precise is your keypoint tracking (3D error in mm)? How many keypoints were tracked exactly? What tracking method did you use? What method/model did you use to go from keypoints to frame labels in your benchmarking? There is a lot of missing detail that is required of a datasets / benchmarks paper that the authors must provide for the work to be useful to others.

Similarly, as a dataset/benchmark paper, it is standard to show results from several different methods, or at least some internal comparisons over different model configurations/hyperparamters.
The author present an interesting and novel new dataset of mice solving puzzles. It is not 100% clear to me the extent of the research questions one could use such a dataset for, and I encourage the reviewers to expound on this in their revision.
The authors present a new, large dataset of lab mice solving mechanical lockbox puzzles. Unlike existing animal behavior datasets, their lockboxes capture more cognitive / learning / problem solving abilities of the animals. In the future, this dataset, or the use of the lockboxes morie generally, could be used to better understand motor and cognitive learning and complex neural computations and processes. At the moment, the dataset's benchmark includes recognizing a set of actions or action.puzzle states.
As a dataset paper, the authors should be providing additional technical details, data quality metrics, and an expanded set of methods for benchmarking. I've left specific comments on these points in the questions section.
Yes
No ethics review needed.
1. Will the key-point tracking results be released with the _Mouse Lockbox Dataset_? In the conclusive section, the authors claim, "(...) we are convinced that approaches beyond keypoint (pose) tracking, e.g., representations learnt without any or under self-supervision, are crucial to future advancements in neuroscience.", however when dealing with metric tasks, e.g., estimating the distance of the mouse from the lockbox, keypoint data represent an important piece of information. Also, to qualitatively assess the baseline performance, adding an image of the key points extracted by the baseline would be interesting.
1. The paper is clear and well-written, reporting several important details on the dataset generation process. 
2. The dataset represents an interesting source for computational ethology to study the mice's behavior in controlled environments when dealing with mechanical problem-solving.
3. The ~13% of the dataset is annotated with several features that can be leveraged to effectively delve into the mice's behavior.
4. The related work section clearly reports and compares the proposed dataset with the most important datasets in rodent computational ethology literature.
The authors propose a novel dataset, the _Mouse Lockbox Dataset_, which includes videos of individual mice dealing with mechanical puzzles named lockboxes. While existing datasets mainly target mice's social or common behaviors, the aim of the proposed dataset is to provide a useful resource for studying single-agent intelligent behaviors.
The paper describes with great detail both the setup used by the authors to record it and the procedure they established to annotate the (13% of the) dataset. The human-annotated labels are also compared with a benchmark automatic method, which relies on the poses of the mice as extracted with DeepLabCut. In particular, the authors compare the human-human agreement on the labels with the one obtained by comparing the human annotator and the automatic method performance.
1. While the proposed dataset represents a novel source for studying mice behavior, the authors lack a proper experiment to showcase the relevance of their contribution. Benchmarking an automatic method against the human annotators reinforces the idea of a high-quality dataset but does not deliver to a non-ethologist reader the idea of how to use the  _Mouse Lockbox Dataset_. 
2. To complete section 2, the authors could report a table comparing their proposed dataset with the available literature described therein.
3. The method used for benchmarking is not reported in the paper, although it is cited in the introduction. To help the reader understand the importance of this baseline, the authors should introduce it in its dedicated section, reporting at least some high-level details about the structure of this baseline.
4. The introduction is fragmented (with several paragraphs) and does not focus enough on the paper's contributions. In particular, the section describes recent advancements and methods in computational ethology well, but only the last two paragraphs describe the content of the proposed paper.
5. In general, the paper would greatly benefit from introducing and evaluating some methods that can be used on the _Mouse Lockbox Dataset_ to provide a reference for future users and encourage research with it.
Yes
No ethics review needed.
1. Could you clarify whether alternative pose-tracking methods were considered, particularly to improve the accuracy of detecting fine-grained actions like biting?
2. Is there a plan to expand the dataset with additional annotated videos, or are any tools being developed to help automate or verify annotation accuracy for challenging actions?
3. Given the temporal desynchronization in the multi-perspective recordings, have you considered any post-processing or alignment correction methods to reduce this effect?
1. This dataset addresses a crucial gap in animal behavior research by focusing on complex, single-agent interactions in mice—particularly rare for capturing intricate, non-social behaviors.
2. Its scale and detail are impressive, with over 110 hours of footage from multiple angles. The multi-perspective setup and extensive annotations make it a highly valuable resource for training and evaluating behavior recognition models.
3. The inclusion of a keypoint-based action classification benchmark is a well-considered addition. It establishes a standard for future comparisons and highlights the strengths of current methods (e.g., proximity detection) as well as their limitations (e.g., recognizing fine-grained actions).
4. The authors provide a detailed methodology for creating the dataset, which supports reproducibility and helps ensure the reliability of the process.
The paper introduces a novel video dataset of individual mice interacting with complex mechanical lockbox puzzles, labeled as the "Mouse Lockbox Dataset." It provides a significant contribution to computational neuroethology, presenting over 110 hours of video data capturing intricate, non-social behaviors. The dataset uniquely includes multi-perspective recordings, human-annotated action labels for a subset of videos, and benchmark results utilizing a keypoint-based action classification method. This work aims to support machine learning advancements in behavioral action classification, particularly in fine-grained tasks such as object manipulation.
1. Although this dataset is a valuable resource for behavior recognition, the paper lacks clear examples of specific use cases and potential research problems it could address. The absence of concrete applications or suggested research directions may limit its utility as an inspiration for future work. Providing examples of problems or experimental scenarios where this dataset could be useful would strengthen its research impact and relevance.
2. The pseudo-synchronization of the multi-camera setup presents a minor challenge for users needing precise 3D reconstructions. While the misalignment is slight, it may still affect analyses requiring high precision.
3. Certain actions, such as biting, have lower annotator agreement, indicating challenges in reliably annotating these behaviors. This suggests that alternative or more standardized annotation methods may be necessary to improve reliability.
Yes
No ethics review needed.
•	It would be great if the authors provided a baseline method on a task with their dataset.
•	A data dictionary of the labels as well as a getting started notebook was not provided. These would greatly increase the ease of use of the dataset.
•	The dataset has over 117 hours of videos of mice behaviour doing complex tasks. This multi-perspective video data is novel and crucial to enable the development of advance methods in the field of neuroscience. 
•	Action based labelling is very detailed, providing annotations at sub-second intervals.
•	The authors are very transparent in how they gathered the data and in compliance of local laws regarding to animal research. 
•	Human annotation is provided by the authors with special care taken to ensure that there is no leakage between the labelled and unlabelled set by keeping the mouses separate for labelled set. 
•	The human annotators followed a specific set of ethograms which makes the process objective in nature and lowers the possibility of disagreement.
•	The authors provide detailed statistics of their dataset
The paper introduces a new dataset in the field of computational ethology. It focuses on creating various scenarios where mouse behaviour and actions are recorded while it tries to solve complex mechanical lockbox opening problems in return for treats. The novelty of the paper lies with the fact that existing datasets in this topic focuses only on problems which are not sufficiently complex or are concerned with studying social behaviour of animals.
•	Labelled data provided is very limited and represents only 13% of the total available data. More annotated data would be immensely helpful to researchers.
•	The dataset focuses exclusively on female mice and the reasoning behind this is not explained and the authors do not cite any paper which disproves this. This choice could limit generalizability. 
•	There is an uneven distribution between playtime action labels. If this cannot be made closer to each other, the authors should mention why it is so.
•	The dataset is provided in grayscale. While this is not a major weakness, RGB images could contain more information that is lost in the grayscale conversion.
•	The authors do not provide camera homography information of the cameras involved.
Yes
No ethics review needed.
1. What pose keypoint definitions were used in this work? Will different key point definitions affect recognition?
2. Are all the mice in the dataset presented in this paper under normal light conditions?
3. Does the hunger state of mice affect their ability to solve mechanical puzzles?
1. This paper collected a large-scale video dataset of individual mice solving complex mechanical puzzles. 
2. This paper is well-organized and easy to read.
3. This paper provides a comprehensive review of relevant work.
This paper collected a multi-perspective video dataset of individual mice solving complex mechanical puzzles. This paper also provided the human-annotated labels, i.e., the proximity between a mouse and a mechanism. The authors give detailed statistics of playtime shares per lockbox.
1. This paper only collected a new dataset and ignored benchmark methods for recognizing mouse actions.
2. The comparison between the dataset collected in this paper and previous datasets is unclear. The author can provide a summary table to illustrate the differences from previous work.
3. There is no baseline or proposed method regarding the behavior recognition of mice.
@inproceedings{
anonymous2024mouse,
title={Mouse Lockbox Dataset: Behavior Recognition of Mice Solving Mechanical Puzzles},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kffZUFZVHT},
note={under review}
}
Machine learning and computer vision have a major impact on the study of natural animal behavior, as they enable automated action classification of large bodies of videos. Mice are the standard mammalian model system in many fields of research, but the open datasets that are currently available to refine machine learning methods mostly focus on either simple or social behaviors. In this work, we present a large video dataset of individual mice solving complex mechanical puzzles, so-called lockboxes. The dataset consists of a total of well over 110 hours of animal behavior, recorded with three cameras from different perspectives. As a benchmark for frame-level action classification methods, we provide human-annotated labels for all videos of two different mice, that equal 13% of our dataset. The used keypoint (pose) tracking-based action classification framework illustrates the challenges of automated labeling of fine-grained behaviors, such as the manipulation of objects. We hope that our work will help accelerate the advancement of automated action and behavior classification in the computational neuroscience community. An anonymized preview of our dataset is available for the reviewers of this manuscript at https://www.dropbox.com/scl/fo/h7nkai8574h23qfq9m1b2/AP4gNZOpDJJ7z0yGtbWQiOc?rlkey=w36jzxqjkghg0j0xva5zsxy2v&st=5r9msqjw&dl=0
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
dataset video action classification behavior mice machine learning computer vision neuroscience computational neuroscience
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/99345b8ca77a98d0242306a99c73c81505f127fe.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Mouse Lockbox Dataset: Behavior Recognition of Mice Solving Mechanical Puzzles
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission