Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,85 @@
+# 🚰 The Sound of Water: Inferring Physical Properties from Pouring Liquids
+
+In this folder, we provide the following trained model checkpoints:
+
+<p align="center">
+  <img src="./assets/pitch_on_spectrogram-compressed.gif" alt="Teaser" width="100%">
+</p>
+
+*Key insight*: As water is poured, the fundamental frequency that we hear changes predictably over time as a function of physical properties (e.g., container dimensions).
+
+**TL;DR**: We present a method to infer physical properties of liquids from *just* the sound of pouring. We show in theory how *pitch* can be used to derive various physical properties such as container height, flow rate, etc. Then, we train a pitch detection network (`wav2vec2`) using simulated and real data. The resulting model can predict the physical properties of pouring liquids with high accuracy. The latent representations learned also encode information about liquid mass and container shape.
+
+
+## Models
+
+We provide audio models trained to detect pitch in the sound of pouring water.
+We train these models in two stages:
+
+1. **Pre-training on synthetic data**: We simulate sounds of pouring water using [DDSP](https://arxiv.org/abs/2001.04643) using only 80 samples. This is used to generate lots of simulated sounds of pouring water. Then, we train `wav2vec2` on this data.
+2. **Fine-tuning on real data**: We fine-tune the model on real data. Since real data does not come with ground truth, we use visual co-supervision from the video stream to fine-tune the audio model.
+
+Here, we provide checkpoints for both the stages.
+
+<table style="font-size: 12px;" class="center">
+  <tr>
+    <th><b> File name </b></th>
+    <th><b> Description </b></th>
+    <th><b> Size </b></th>
+  </tr>
+  <tr>
+    <td>dsr9mf13_ep100_step12423_synthetic_pretrained.pth</td>
+    <td>Pre-trained on synthetic data</td>
+    <td>361M</td>
+  </tr>
+  <tr>
+    <td>dsr9mf13_ep100_step12423_real_finetuned_with_cosupervision.pth</td>
+    <td>Trained with visual co-supervision</td>
+    <td>361M</td>
+  </tr>
+</table>
+
+
+Use this snippet to download the models:
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    repo_id="bpiyush/sound-of-water-models",
+    local_dir="/path/to/download/",
+)
+```
+
+<!-- Add a citation -->
+## 📜 Citation
+
+If you find this repository useful, please consider giving a star ⭐ and citation
+
+```bibtex
+@article{sound_of_water_bagad,
+  title={The Sound of Water: Inferring Physical Properties from Pouring Liquids},
+  author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
+  journal={arXiv},
+  year={2024}
+}
+```
+
+<!-- Add acknowledgements, license, etc. here. -->
+## 🙏 Acknowledgements
+
+* We thank Ashish Thandavan for support with infrastructure and Sindhu
+Hegde, Ragav Sachdeva, Jaesung Huh, Vladimir Iashin, Prajwal KR, and Aditya Singh for useful
+discussions.
+* This research is funded by EPSRC Programme Grant VisualAI EP/T028572/1, and a Royal Society Research Professorship RP / R1 / 191132.
+
+We also want to highlight closely related work that could be of interest:
+
+* [Analyzing Liquid Pouring Sequences via Audio-Visual Neural Networks](https://gamma.cs.unc.edu/PSNN/). IROS (2019).
+* [Human sensitivity to acoustic information from vessel filling](https://psycnet.apa.org/record/2000-13210-019). Journal of Experimental Psychology (2020).
+* [See the Glass Half Full: Reasoning About Liquid Containers, Their Volume and Content](https://arxiv.org/abs/1701.02718). ICCV (2017).
+* [CREPE: A Convolutional Representation for Pitch Estimation](https://arxiv.org/abs/1802.06182). ICASSP (2018).
+
+## 🙅🏻 Potential Biases
+
+Our model is based on `wav2vec2` which is trained on a large-scale speech recognition data. While this data is not as large as usual datasets in AI, it may still have undesirable biases that are present in the training data.