esalguero commited on
Commit
fe2f0df
·
verified ·
1 Parent(s): 488ad1c

Delete README_hf.md

Browse files
Files changed (1) hide show
  1. README_hf.md +0 -167
README_hf.md DELETED
@@ -1,167 +0,0 @@
1
- ---
2
- language:
3
- - en
4
- tags:
5
- - computer-vision
6
- - segmentation
7
- - few-shot-learning
8
- - zero-shot-learning
9
- - sam2
10
- - clip
11
- - pytorch
12
- license: apache-2.0
13
- datasets:
14
- - custom
15
- metrics:
16
- - iou
17
- - dice
18
- - precision
19
- - recall
20
- library_name: pytorch
21
- pipeline_tag: image-segmentation
22
- ---
23
-
24
- # SAM 2 Few-Shot/Zero-Shot Segmentation
25
-
26
- This repository contains a comprehensive research framework for combining Segment Anything Model 2 (SAM 2) with few-shot and zero-shot learning techniques for domain-specific segmentation tasks.
27
-
28
- ## 🎯 Overview
29
-
30
- This project investigates how minimal supervision can adapt SAM 2 to new object categories across three distinct domains:
31
- - **Satellite Imagery**: Buildings, roads, vegetation, water
32
- - **Fashion**: Shirts, pants, dresses, shoes
33
- - **Robotics**: Robots, tools, safety equipment
34
-
35
- ## 🏗️ Architecture
36
-
37
- ### Few-Shot Learning Framework
38
- - **Memory Bank**: Stores CLIP-encoded examples for each class
39
- - **Similarity-Based Prompting**: Uses visual similarity to generate SAM 2 prompts
40
- - **Episodic Training**: Standard few-shot learning protocol
41
-
42
- ### Zero-Shot Learning Framework
43
- - **Advanced Prompt Engineering**: 4 strategies (basic, descriptive, contextual, detailed)
44
- - **Attention-Based Localization**: Uses CLIP's cross-attention for prompt generation
45
- - **Multi-Strategy Prompting**: Combines different prompt types
46
-
47
- ## 📊 Performance
48
-
49
- ### Few-Shot Learning (5 shots)
50
- | Domain | Mean IoU | Mean Dice | Best Class | Worst Class |
51
- |--------|----------|-----------|------------|-------------|
52
- | Satellite | 65% | 71% | Building (78%) | Water (52%) |
53
- | Fashion | 62% | 68% | Shirt (75%) | Shoes (48%) |
54
- | Robotics | 59% | 65% | Robot (72%) | Safety (45%) |
55
-
56
- ### Zero-Shot Learning (Best Strategy)
57
- | Domain | Mean IoU | Mean Dice | Best Class | Worst Class |
58
- |--------|----------|-----------|------------|-------------|
59
- | Satellite | 42% | 48% | Building (62%) | Water (28%) |
60
- | Fashion | 38% | 45% | Shirt (58%) | Shoes (25%) |
61
- | Robotics | 35% | 42% | Robot (55%) | Safety (22%) |
62
-
63
- ## 🚀 Quick Start
64
-
65
- ### Installation
66
- ```bash
67
- pip install -r requirements.txt
68
- python scripts/download_sam2.py
69
- ```
70
-
71
- ### Few-Shot Experiment
72
- ```python
73
- from models.sam2_fewshot import SAM2FewShot
74
-
75
- # Initialize model
76
- model = SAM2FewShot(
77
- sam2_checkpoint="sam2_checkpoint",
78
- device="cuda"
79
- )
80
-
81
- # Add support examples
82
- model.add_few_shot_example("satellite", "building", image, mask)
83
-
84
- # Perform segmentation
85
- predictions = model.segment(
86
- query_image,
87
- "satellite",
88
- ["building"],
89
- use_few_shot=True
90
- )
91
- ```
92
-
93
- ### Zero-Shot Experiment
94
- ```python
95
- from models.sam2_zeroshot import SAM2ZeroShot
96
-
97
- # Initialize model
98
- model = SAM2ZeroShot(
99
- sam2_checkpoint="sam2_checkpoint",
100
- device="cuda"
101
- )
102
-
103
- # Perform zero-shot segmentation
104
- predictions = model.segment(
105
- image,
106
- "fashion",
107
- ["shirt", "pants", "dress", "shoes"]
108
- )
109
- ```
110
-
111
- ## 📁 Project Structure
112
-
113
- ```
114
- ├── models/
115
- │ ├── sam2_fewshot.py # Few-shot learning model
116
- │ └── sam2_zeroshot.py # Zero-shot learning model
117
- ├── experiments/
118
- │ ├── few_shot_satellite.py # Satellite experiments
119
- │ └── zero_shot_fashion.py # Fashion experiments
120
- ├── utils/
121
- │ ├── data_loader.py # Domain-specific data loaders
122
- │ ├── metrics.py # Comprehensive evaluation metrics
123
- │ └── visualization.py # Visualization tools
124
- ├── scripts/
125
- │ └── download_sam2.py # Setup script
126
- └── notebooks/
127
- └── analysis.ipynb # Interactive analysis
128
- ```
129
-
130
- ## 🔬 Research Contributions
131
-
132
- 1. **Novel Architecture**: Combines SAM 2 + CLIP for few-shot/zero-shot segmentation
133
- 2. **Domain-Specific Prompting**: Advanced prompt engineering for different domains
134
- 3. **Attention-Based Prompt Generation**: Leverages CLIP attention for localization
135
- 4. **Comprehensive Evaluation**: Extensive experiments across multiple domains
136
- 5. **Open-Source Implementation**: Complete codebase for reproducibility
137
-
138
- ## 📚 Citation
139
-
140
- If you use this work in your research, please cite:
141
-
142
- ```bibtex
143
- @misc{sam2_fewshot_zeroshot_2024,
144
- title={SAM 2 Few-Shot/Zero-Shot Segmentation: Domain Adaptation with Minimal Supervision},
145
- author={Your Name},
146
- year={2024},
147
- url={https://huggingface.co/esalguero/Segmentation}
148
- }
149
- ```
150
-
151
- ## 🤝 Contributing
152
-
153
- We welcome contributions! Please feel free to submit issues, pull requests, or suggestions for improvements.
154
-
155
- ## 📄 License
156
-
157
- This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
158
-
159
- ## 🔗 Links
160
-
161
- - **GitHub Repository**: [https://github.com/ParallelLLC/Segmentation](https://github.com/ParallelLLC/Segmentation)
162
- - **Research Paper**: See `research_paper.md` for complete methodology
163
- - **Interactive Analysis**: Use `notebooks/analysis.ipynb` for exploration
164
-
165
- ---
166
-
167
- **Keywords**: Few-shot learning, Zero-shot learning, Semantic segmentation, SAM 2, CLIP, Domain adaptation, Computer vision