Random beats complex: Simple example selection works better for long-in-context learning (ICL).

This paper reveals that sophisticated example selection methods for In-Context Learning don't significantly outperform random selection when using long-context models.

-----

🤔 Original Problem:

→ Traditional In-Context Learning (ICL) was limited by short context windows, making example selection crucial for performance

→ With new Long Context Language Models (LCLMs) supporting millions of tokens, we need to understand if previous sample selection strategies still matter

-----

🔍 Solution in this Paper:

→ The researchers conducted extensive experiments across 18 datasets spanning translation, summarization, reasoning, and classification tasks

→ They tested three types of selection methods: relevance-based, diversity-based, and difficulty-based approaches

→ The study compared these methods against simple random selection using models like Gemini 1.5 Pro and Flash

→ For scenarios with limited examples, they proposed a data augmentation approach that generates and filters synthetic examples

-----

💡 Key Insights:

→ Sophisticated selection methods show no significant improvement over random selection in many-shot scenarios

→ The challenge has shifted from optimizing example selection to maximizing context window utilization

→ Performance plateaus and declines as context length approaches the limit

→ LCLMs become vulnerable to noise in complex scenarios, especially in low-resource tasks

-----

📊 Results:

→ Data augmentation improved ICL performance by 5% in low-resource tasks

→ Statistical significance in fewer than 15% of instances for sophisticated selection methods

→ Performance decline begins when more than 25% of available context capacity is utilized