facebook/natural_reasoning
Viewer
•
Updated
•
1.15M
•
10.7k
•
398
Lucas Schönhold
lucasschoenhold
·
AI & ML interests
None yet
Recent Activity
liked
a dataset
7 days ago
facebook/natural_reasoning
liked
a Space
7 days ago
open-llm-leaderboard/open_llm_leaderboard
liked
a dataset
14 days ago
SakanaAI/AI-CUDA-Engineer-Archive
Organizations
None yet
lucasschoenhold's activity
liked
a
dataset
7 days ago
liked
a
Space
7 days ago
liked
a
dataset
14 days ago
reacted to
singhsidhukuldeep's
post with 👀
14 days ago
Post
1655
I just came across a groundbreaking paper titled "Hypencoder: Hypernetworks for Information Retrieval" by researchers from the University of Massachusetts Amherst that introduces a fundamentally new paradigm for search technology.
Most current retrieval models rely on simple inner product calculations between query and document vectors, which severely limits their expressiveness. The authors prove theoretically that inner product similarity functions fundamentally constrain what types of relevance relationships can be captured.
Hypencoder takes a radically different approach: instead of encoding a query as a vector, it generates a small neural network (called a "q-net") that acts as a learned relevance function. This neural network takes document representations as input and produces relevance scores.
Under the hood, Hypencoder uses:
- Attention-based hypernetwork layers (hyperhead layers) that transform contextualized query embeddings into weights and biases for the q-net
- A document encoder that produces vector representations similar to existing models
- A graph-based greedy search algorithm for efficient retrieval that can search 8.8M documents in under 60ms
The results are impressive - Hypencoder significantly outperforms strong dense retrieval models on standard benchmarks like MS MARCO and TREC Deep Learning Track. The performance gap widens even further on complex retrieval tasks like tip-of-the-tongue queries and instruction-following retrieval.
What makes this approach particularly powerful is that neural networks are universal approximators, allowing Hypencoder to express far more complex relevance relationships than inner product similarity functions. The framework is also flexible enough to replicate any existing neural retrieval method while adding the ability to learn query-dependent weights.
Most current retrieval models rely on simple inner product calculations between query and document vectors, which severely limits their expressiveness. The authors prove theoretically that inner product similarity functions fundamentally constrain what types of relevance relationships can be captured.
Hypencoder takes a radically different approach: instead of encoding a query as a vector, it generates a small neural network (called a "q-net") that acts as a learned relevance function. This neural network takes document representations as input and produces relevance scores.
Under the hood, Hypencoder uses:
- Attention-based hypernetwork layers (hyperhead layers) that transform contextualized query embeddings into weights and biases for the q-net
- A document encoder that produces vector representations similar to existing models
- A graph-based greedy search algorithm for efficient retrieval that can search 8.8M documents in under 60ms
The results are impressive - Hypencoder significantly outperforms strong dense retrieval models on standard benchmarks like MS MARCO and TREC Deep Learning Track. The performance gap widens even further on complex retrieval tasks like tip-of-the-tongue queries and instruction-following retrieval.
What makes this approach particularly powerful is that neural networks are universal approximators, allowing Hypencoder to express far more complex relevance relationships than inner product similarity functions. The framework is also flexible enough to replicate any existing neural retrieval method while adding the ability to learn query-dependent weights.
liked
a
Space
2 months ago
liked
a
dataset
3 months ago
liked
a
dataset
6 months ago
liked
a
model
9 months ago
upvoted
an
article
9 months ago
Article
A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using transformers, accelerate and bitsandbytes
•
80
liked
5
models
10 months ago
deepseek-ai/DeepSeek-V2
Text Generation
•
Updated
•
12.5k
•
310
deepseek-ai/deepseek-math-7b-instruct
Text Generation
•
Updated
•
14.6k
•
117
openbmb/MiniCPM-Llama3-V-2_5
Image-Text-to-Text
•
Updated
•
25k
•
1.39k
nvidia/Llama3-ChatQA-1.5-70B
Text Generation
•
Updated
•
326
•
332
openai/whisper-large-v3
Automatic Speech Recognition
•
Updated
•
4.16M
•
•
4.16k
liked
a
dataset
10 months ago
liked
3
models
10 months ago
liked
a
model
11 months ago
upvoted
a
collection
11 months ago