Update BM25S model

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+corpus.jsonl filter=lfs diff=lfs merge=lfs -text
+corpus.mmindex.json filter=lfs diff=lfs merge=lfs -text
+vocab.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,91 @@
+---
+language: en
+tags:
+- bm25
+- bm25s
+- retrieval
+- search
+- lexical
+---
+
+# BM25S Index
+
+This is a BM25S index created with the [`bm25s` library](https://github.com/xhluca/bm25s) (version `0.0.1dev0`), an ultra-fast implementation of BM25. It can be used for lexical retrieval tasks.
+
+[BM25S GitHub Repository](https://github.com/xhluca/bm25s)
+
+## Installation
+
+You can install the `bm25s` library with `pip`:
+
+```bash
+pip install "bm25s==0.0.1dev0"
+
+# Include extra dependencies like stemmer
+pip install "bm25s[full]==0.0.1dev0"
+
+# For huggingface hub usage
+pip install huggingface_hub
+```
+
+## Loading a `bm25s` index
+
+You can use this index for information retrieval tasks. Here is an example:
+
+```python
+import bm25s
+from bm25s.hf import BM25HF
+
+# Load the index
+retriever = BM25HF.load_from_hub("xhluca/bm25s-nq-index", revision="main")
+
+# You can retrieve now
+query = "a cat is a feline"
+results = retriever.retrieve(query, k=3)
+```
+
+## Saving a `bm25s` index
+
+You can save a `bm25s` index to the Hugging Face Hub. Here is an example:
+
+```python
+import bm25s
+from bm25s.hf import BM25HF
+
+# Create a BM25 index and add documents
+retriever = BM25HF()
+corpus = [
+    "a cat is a feline and likes to purr",
+    "a dog is the human's best friend and loves to play",
+    "a bird is a beautiful animal that can fly",
+    "a fish is a creature that lives in water and swims",
+]
+corpus_tokens = bm25s.tokenize(corpus)
+retriever.index(corpus_tokens)
+
+token = None  # You can get a token from the Hugging Face website
+retriever.save_to_hub("xhluca/bm25s-nq-index", token=token)
+```
+
+
+## Stats
+
+This dataset was created using the following data:
+
+| Statistic | Value |
+| --- | --- |
+| Number of documents | 2681468 |
+| Number of tokens | 116237970 |
+| Average tokens per document | 43.3486321671562 |
+
+## Parameters
+
+The index was created with the following parameters:
+
+| Parameter | Value |
+| --- | --- |
+| k1 | `1.5` |
+| b | `0.75` |
+| delta | `0.5` |
+| method | `lucene` |
+| idf method | `lucene` |

corpus.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44b59b73c4d65a2ca25aed039c6e80f74f80af1e8345b394af11cf74e78b94a6
+size 1444980459

corpus.mmindex.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:935ad3f79be3e0476835047c3788af0857acc7d794646285cf739134590b3db1
+size 27471651

data.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb6e57b4f6cba920cbd90fb455adb9119689f397adeade8e4737929e11e41781
+size 464952008

indices.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bf64d392fca3d08b7a0655a88372ee0815bf82d0f10f9337c384aba9cb46199
+size 464952008

indptr.csc.index.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5179cf8c792ab866f48bfb934ff7b09c25833b6e4e3b5f7501e2dd3c5558e712
+size 3572064

params.index.json

@@ -0,0 +1,11 @@
+{
+    "k1": 1.5,
+    "b": 0.75,
+    "delta": 0.5,
+    "method": "lucene",
+    "idf_method": "lucene",
+    "dtype": "float32",
+    "int_dtype": "int32",
+    "num_docs": 2681468,
+    "version": "0.0.1dev0"
+}

vocab.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:138b5bc83f8c203e5cb316f9360caa425da73891f30f12d67d49d4c7e9cb1e9a
+size 17082686