Update README.md

README.md

@@ -1,141 +1,134 @@
----
-library_name: sentence-transformers
-pipeline_tag: sentence-similarity
-tags:
-- sentence-transformers
-- sentence-similarity
-- feature-extraction
----
-
-# SentenceTransformer
-
-This is a [sentence-transformers](https://www.SBERT.net) model trained. It maps sentences & paragraphs to a 3584-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
-
-## Model Details
-
-### Model Description
-- **Model Type:** Sentence Transformer
-<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
-- **Maximum Sequence Length:** 32768 tokens
-- **Output Dimensionality:** 3584 dimensions
-- **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
-
-### Model Sources
-
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
-
-### Full Model Architecture
-
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False}) with Transformer model: Qwen2Model 
-  (1): Pooling({'word_embedding_dimension': 3584, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
-  (2): Normalize()
-)
-```
-
-## Usage
-
-### Direct Usage (Sentence Transformers)
-
-First install the Sentence Transformers library:
+---
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+license: apache-2.0
+datasets:
+- nomic-ai/cornstack-python-v1
+- nomic-ai/cornstack-javascript-v1
+- nomic-ai/cornstack-java-v1
+- nomic-ai/cornstack-go-v1
+- nomic-ai/cornstack-php-v1
+- nomic-ai/cornstack-ruby-v1
+base_model:
+- Qwen/Qwen2.5-Coder-7B-Instruct
+---
+
+# Nomic Embed Code: A State-of-the-Art Code Retriever
+
+
+`nomic-embed-code` is a state-of-the-art code embedding model that excels at code retrieval tasks:
+
+- **High Performance**: Outperforms Voyage Code 3 and OpenAI Embed 3 Large on CodeSearchNet
+- **Multilingual Code Support**: Trained for multiple programming languages (Python, Java, Ruby, PHP, JavaScript, Go)
+- **Advanced Architecture**: 7B parameter code embedding model
+- **Fully Open-Source**: Model weights, training data, and [evaluation code](https://github.com/gangiswag/cornstack/) released
+
+| Model | Python | Java | Ruby | PHP | JavaScript | Go |
+|-------|--------|------|------|-----|------------|-----|
+| **Nomic Embed Code** | **81.7** | **80.5** | 81.8 | **72.3** | 77.1 | **93.8** |
+| Voyage Code 3 | 80.8 | **80.5** | **84.6** | 71.7 | **79.2** | 93.2 |
+| OpenAI Embed 3 Large | 70.8 | 72.9 | 75.3 | 59.6 | 68.1 | 87.6 |
+| Nomic CodeRankEmbed-137M | 78.4 | 76.9 | 79.3 | 68.8 | 71.4 | 92.7 |
+| CodeSage Large v2 (1B) | 74.2 | 72.3 | 76.7 | 65.2 | 72.5 | 84.6 |
+| CodeSage Large (1B) | 70.8 | 70.2 | 71.9 | 61.3 | 69.5 | 83.7 |
+| Qodo Embed 1 7B | 59.9 | 61.6 | 68.4 | 48.5 | 57.0 | 81.4 |
+
+## Model Architecture
+
+- **Total Parameters**: 7B
+- **Training Approach**: Trained on the CoRNStack dataset with dual-consistency filtering and progressive hard negative mining
+- **Supported Languages**: Python, Java, Ruby, PHP, JavaScript, and Go
+
+## Usage Guide
+
+### Installation
+
+You can install the necessary dependencies with:
 
 ```bash
-pip install -U sentence-transformers
+pip install transformers sentence-transformers torch
 ```
 
-Then you can load this model and run inference.
-```python
-from sentence_transformers import SentenceTransformer
+### Transformers
 
-# Download from the 🤗 Hub
-model = SentenceTransformer("nomic-ai/qwen7b-coder-instruct-5M-last-token-causal-8e6-eos-token")
-# Run inference
-sentences = [
-    'The weather is lovely today.',
-    "It's so sunny outside!",
-    'He drove to the stadium.',
-]
-embeddings = model.encode(sentences)
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("nomic-ai/nomic-embed-code")
+model = AutoModel.from_pretrained("nomic-ai/nomic-embed-code")
+
+def last_token_pooling(model_output, attention_mask):
+    sequence_lengths = attention_mask.sum(-1) - 1
+    return hidden_states[torch.arange(hidden_states.shape[0]), sequence_lengths]
+
+queries = ['Represent this query for searching relevant code: Calculate the n-th factorial']
+codes = ['def fact(n):\n if n < 0:\n  raise ValueError\n return 1 if n == 0 else n * fact(n - 1)']
+code_snippets = queries + codes
+
+encoded_input = tokenizer(code_snippets, padding=True, truncation=True, return_tensors='pt')
+model.eval()
+with torch.no_grad():
+    model_output = model(**encoded_input)
+embeddings = last_token_pooling(model_output, encoded_input['attention_mask'])
+embeddings = F.normalize(embeddings, p=2, dim=1)
 print(embeddings.shape)
-# [3, 3584]
 
-# Get the similarity scores for the embeddings
-similarities = model.similarity(embeddings, embeddings)
-print(similarities.shape)
-# [3, 3]
+similarity = F.cosine_similarity(embeddings[0], embeddings[1], dim=0)
+print(similarity)
 ```
 
-<!--
-### Direct Usage (Transformers)
-
-<details><summary>Click to see the direct usage in Transformers</summary>
-
-</details>
--->
-
-<!--
-### Downstream Usage (Sentence Transformers)
+### SentenceTransformers
 
-You can finetune this model on your own dataset.
-
-<details><summary>Click to expand</summary>
-
-</details>
--->
-
-<!--
-### Out-of-Scope Use
+```python
+from sentence_transformers import SentenceTransformer
 
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
+queries = ['Represent this query for searching relevant code: Calculate the n-th factorial']
+codes = ['def fact(n):\n if n < 0:\n  raise ValueError\n return 1 if n == 0 else n * fact(n - 1)']
 
-<!--
-## Bias, Risks and Limitations
+model = SentenceTransformer("nomic-ai/nomic-embed-code")
+query_emb = model.encode(queries, prompt_name="query")
+code_emb = model.encode(code_snippets)
 
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
+similarity = model.similarity(query_emb, code_emb)
+print(similarity)
+```
 
-<!--
-### Recommendations
 
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
+### CoRNStack Dataset Curation
 
-## Training Details
+Starting with the deduplicated Stackv2, we create text-code pairs from function docstrings and respective code. We filtered out low-quality pairs where the docstring wasn't English, too short, or that contained URLs, HTML tags, or invalid characters. We additionally kept docstrings with text lengths of 256 tokens or longer to help the model learn long-range dependencies.
 
-### Framework Versions
-- Python: 3.10.12
-- Sentence Transformers: 3.3.0
-- Transformers: 4.45.2
-- PyTorch: 2.4.1+cu121
-- Accelerate: 1.4.0
-- Datasets: 2.21.0
-- Tokenizers: 0.20.0
 
-## Citation
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/607997c83a565c15675055b3/rb-J54KLgg21f59Ba1jWp.png)
 
-### BibTeX
+After the initial filtering, we used dual-consistency filtering to remove potentially noisy examples. We embed each docstring and code pair and compute the similarity between each docstring and every code example. We remove pairs from the dataset if the corresponding code example is not found in the top-2 most similar examples for a given docstring.
 
-<!--
-## Glossary
+During training, we employ a novel curriculum-based hard negative mining strategy to ensure the model learns from challenging examples. We use a softmax-based sampling strategy to progressively sample hard negatives with increasing difficulty over time.
 
-*Clearly define terms in order to be accessible across audiences.*
--->
+## Join the Nomic Community
 
-<!--
-## Model Card Authors
+- Website: [https://nomic.ai](https://nomic.ai)
+- Twitter: [https://twitter.com/nomic_ai](https://twitter.com/nomic_ai)
+- Discord: [https://discord.gg/myY5YDR8z8](https://discord.gg/myY5YDR8z8)
 
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
+# Citation
 
-<!--
-## Model Card Contact
+If you find the model, dataset, or training code useful, please cite our work:
 
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->
+```bibtex
+@misc{suresh2025cornstackhighqualitycontrastivedata,
+      title={CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking}, 
+      author={Tarun Suresh and Revanth Gangi Reddy and Yifei Xu and Zach Nussbaum and Andriy Mulyar and Brandon Duderstadt and Heng Ji},
+      year={2025},
+      eprint={2412.01007},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2412.01007}, 
+}
+```