# 🚰 The Sound of Water: Inferring Physical Properties from Pouring Liquids
In this folder, we provide the following trained model checkpoints:
*Key insight*: As water is poured, the fundamental frequency that we hear changes predictably over time as a function of physical properties (e.g., container dimensions).
**TL;DR**: We present a method to infer physical properties of liquids from *just* the sound of pouring. We show in theory how *pitch* can be used to derive various physical properties such as container height, flow rate, etc. Then, we train a pitch detection network (`wav2vec2`) using simulated and real data. The resulting model can predict the physical properties of pouring liquids with high accuracy. The latent representations learned also encode information about liquid mass and container shape.
## Models
We provide audio models trained to detect pitch in the sound of pouring water.
We train these models in two stages:
1. **Pre-training on synthetic data**: We simulate sounds of pouring water using [DDSP](https://arxiv.org/abs/2001.04643) using only 80 samples. This is used to generate lots of simulated sounds of pouring water. Then, we train `wav2vec2` on this data.
2. **Fine-tuning on real data**: We fine-tune the model on real data. Since real data does not come with ground truth, we use visual co-supervision from the video stream to fine-tune the audio model.
Here, we provide checkpoints for both the stages.
Use this snippet to download the models:
```python
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="bpiyush/sound-of-water-models",
local_dir="/path/to/download/",
)
```
## 📜 Citation
If you find this repository useful, please consider giving a star ⭐ and citation
```bibtex
@article{sound_of_water_bagad,
title={The Sound of Water: Inferring Physical Properties from Pouring Liquids},
author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
journal={arXiv},
year={2024}
}
```
## 🙏 Acknowledgements
* We thank Ashish Thandavan for support with infrastructure and Sindhu
Hegde, Ragav Sachdeva, Jaesung Huh, Vladimir Iashin, Prajwal KR, and Aditya Singh for useful
discussions.
* This research is funded by EPSRC Programme Grant VisualAI EP/T028572/1, and a Royal Society Research Professorship RP / R1 / 191132.
We also want to highlight closely related work that could be of interest:
* [Analyzing Liquid Pouring Sequences via Audio-Visual Neural Networks](https://gamma.cs.unc.edu/PSNN/). IROS (2019).
* [Human sensitivity to acoustic information from vessel filling](https://psycnet.apa.org/record/2000-13210-019). Journal of Experimental Psychology (2020).
* [See the Glass Half Full: Reasoning About Liquid Containers, Their Volume and Content](https://arxiv.org/abs/1701.02718). ICCV (2017).
* [CREPE: A Convolutional Representation for Pitch Estimation](https://arxiv.org/abs/1802.06182). ICASSP (2018).
## 🙅🏻 Potential Biases
Our model is based on `wav2vec2` which is trained on a large-scale speech recognition data. While this data is not as large as usual datasets in AI, it may still have undesirable biases that are present in the training data.
