Unsupervised Meta-Learning via In-Context Learning
meta-learning;unsupervised learning;in-context learning
Yes
No ethics review needed.
One more question:
As the class of instances in FSL are pseudo-labels randomly picked in range $0-N$, how to explain the top linear lay for classification that project the output embedding of the query into the final category distribution? An instance could be of class 0 in task $T_i$ but of class 1 in task $T_{i+1}$, how can the in-context classifier (i.e. the top linear lay) predict its class without finetuning upon the support set?
1) The work proposes a novel pseudo-task creation mechanism to generate FSL tasks out of unsupervised dataset, providing a new (maybe efficient) way to learn from unlabeled data. 
2) Experiments with CAML show that CAMeLU outperforms other unsupervised methods to achieve promising results.
This work aims to tackle few-shot learning (FSL) by learning from unsupervised dataset. It shares a similar idea with CAML[1] in model design by taking FSL as a sequence modeling problem. The instance embedding $f_{\psi}$ and the class encoding $g_{\phi}$ of  
 a support image are contacted to form a "token" in the context. of the sequence, and a transformer encoder learns from the context to generated output feature for the query. The output feature finally is projected by a top linear layer predict the class of the query. 

To learn from the given unsupervised dataset, this work proposes a novel task creation mechanism to generate pseudo-tasks. In each run of the task creation mechanism, $N$ samples are randomly selected from the unsupervised dataset and data augmentation strategies are applied to each sample to generate $K$ instances (which are utilized as the support examples), simulating the $N$-way $K$-shot setting. Query samples in the pseudo-task are generated by mixup the selected sample (and its augmented version) with randomly selected samples in the "background" dataset. 

The pseudo-task creation provides a way to perform meta-learning in the pre-training stage when no labeled dataset is available, which is interesting and instructive.


[1] Fifty, C., Duan, D., Junkins, R. G., Amid, E., Leskovec, J., Re, C., & Thrun, S. Context-Aware Meta-Learning. In The Twelfth International Conference on Learning Representations.
However, I have the following concerns:
1) The model in this work is the same as CAML, which shows limited novelty. 
2) Though the task formulation is interesting, only CAML and its successor (e.g. CAMeLU) are fairly trained in the setting. Other unsupervised method cannot leverage the supervised pseudo-tasks as supervised methods do. Essentially, CAMeLU is a supervised method. The pseudo-task generation provides a way to bridge the unsupervised scenario and the supervised scenario.
3) Instead of providing a specific instance (e.g. CAML) to illustrate how the proposed task generation facilitates FSL from unsupervised dataset. I prefer to see more evidences that showcase the pseudo-task creation is a valid and efficient mechanism that makes supervised method like ProtoNet, MAML, etc, feasible in unsupervised scenario. This, I think, is the core contribution of this work.
4) In experiment, building a new subset of Imagenet-1k (i.e. ImageNet-964) is not necessary since the method is so-called unsupervised. Note that, CLIP and its variants would cause information leakage if not carefully considered, because the texts may explicitly work as anchors to categorize instances belong to the same class together.
5) I donot think it is proper to put experiments on CIFAR-FS into the cross-domain setting, because CIFAR-FS contains natural scene color images as in the miniImagenet. Conventionally, cross-domain setting refers to either transferring from coarse-grained setting to fine-grained setting as in FSL, or transferring from natural scene color image setting to painting image setting where instances from two domains show quite different appearances.
6) Typos: e.g. In A.1 Line 793, the iamge embedding's output size decreases to 768, resulting ....
Yes
No ethics review needed.
1. I was wondering if mixup is used during model training, would we still get a reasonable performance in a meta learning setting? 


2. What is the specific justification for the design that was used for the transformer model that is used for experiments? What is the computational overhead that it adds?

3. Metalearning is overcrowded field with so many existing works. What was the reasoning to select methods that were used for comparison? It looks some of these methods are not the most recent methods which likely will have better performance. It is necessary to include 8-10 methods for comparison with majority of them to be most recent works that achieve the highest performances to demonstrate that the proposed method is competitive.
1. The paper is relatively straightforward to read and follow.

2. The performance boost compared to the baselines is significant on all benchmarks.

3. The code is provided and although I did not run it but it looks well documented and easy to run which will make reproducing the results straightforward.
The paper develops CAMeLU for unsupervised meta-learning setting, where the goal is to leverage the in-context learning capabilities of transformers to create transferable feature representations from unlabeled data for new tasks with small numbers of labeled data. By modeling meta-learning as a sequence prediction problem, each observed task is modeled as a sequence of support instances and an unknown query image. The transformer encoder subnetwork then identifies task-specific patterns from support images for that task which enhances accuracy of predictions on query images by making the model specialized for the input task. Data augmentation and task mixing are also used to generate diverse training tasks that challenge the model to improve generalization on new tasks. Experimental results on a benchmark formed from five datasets are offered to demonstrate that CAMeLU is effective for unsupervised meta-learning and leads to a performance close to supervised and self-supervised meta-earning methods.
1. The model may become very large due to the added transformer model which computationally may become expensive and challenging to handle compared to methods are using only CNN architectures.

2. Comparison is limited to only a few methods, despite the fact meta learning have a rich literature.

3. Analytic experiments are limited and do provide much insight about the proposed method. For example, what are the circumstance and criteria under which the proposed method is more effective? What are the limitations? What is the reason behind the improved performance compared to the baselines?
Yes
No ethics review needed.
1.	During the construction of the query set using mixup, does the correlation between the sampled image and the combined augmented image affect the model's performance and stability? For instance, do cases where they belong to the same class or are completely unrelated impact the outcome?

2.	Under time constraints, does the computational cost of this construction method lead to a significant performance improvement in the model?

3.	How sensitive is the model's performance to the parameters used in the query set construction method?
1.	Overcomes the limitation in the UML field of relying solely on simple data augmentation for constructing training tasks, by proposing a novel sequence modeling-based task construction approach.

2.	Effectively leverages the advantages of in-context learning in LLM.

3.	Provides extensive experiments and a thorough hyperparameter tuning process, which comprehensively demonstrate the advantages of the proposed algorithm.
This paper proposes CAMeLU, a novel approach to unsupervised meta-learning (UML) that utilizes transformer architectures' in-context learning capabilities to extract contextual information from support samples and perform predictions on query data as a sequence modeling task.
1.	Lacks an explanation of the benefits of combining transformer architecture with the proposed data construction method, and does not clarify whether other architectures could also be adapted. In short, there is insufficient discussion of the model's generalizability.

2.	Lacks detailed descriptions of the fixed feature extractor f and the learned class encoder g.
Yes
No ethics review needed.
* What are the training costs for CAMeLU? When constructing tasks from unlabeled datasets, especially when using a mixup strategy, do the hyperparameters need to be carefully tuned?

* The comparison with CAML in Figure 3. Why doesn't CAML show an improvement in relative validation accuracy during training? What could be the reason for this?
* The authors propose a novel task creation mechanism that generates diverse few-shot tasks from unlabeled datasets through a combination of data augmentations and a mixing strategy. This seems reasonable.

* They utilize the transformer’s in-context learning capability, eliminating the need for fine-tuning to specific test domains, which is commendable.

* The experiments are thorough, and the exploration shows that the model achieved generalization.
This paper reframes meta-learning as a sequence modeling problem, allowing the transformer encoder to learn task context from support images and use it to predict query images. The core is a mechanism for generating diverse tasks using a combination of data augmentations and a mixing strategy. Experimental results demonstrate the superiority over existing unsupervised meta-learning baselines, highlighting the efficacy of the model in leveraging generalization over memorization.
* The comparisons in the experiments may not be entirely fair. For instance, differences in model architecture between methods, and whether additional feature extractors or class encoders were used, should be noted in a table to enable clearer comparisons. To my knowledge, this seems somewhat unfair to the in-domain baselines. It would be appreciated if a table showing the parameter count and computational overhead for each method were included.

* In comparison to CAML, another transformer-based in-context learning method, or SSL methods, CAMeLU doesn’t perform as well.
We introduce a novel unsupervised meta-learning approach that reframes meta-learning as a sequence modeling problem and leverages the generalization abilities of in-context learners.
@inproceedings{
anonymous2024unsupervised,
title={Unsupervised Meta-Learning via In-Context Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Jprs1v2wPA},
note={under review}
}
Unsupervised meta-learning aims to learn feature representations from unsupervised datasets that can transfer to downstream tasks with limited labeled data.
In this paper, we propose a novel approach to unsupervised meta-learning that leverages the generalization abilities of in-context learning observed in transformer architectures. Our method reframes meta-learning as a sequence modeling problem, enabling the transformer encoder to learn task context from support images and utilize it to predict query images. 
At the core of our approach lies the creation of diverse tasks generated using a combination of data augmentations and a mixing strategy that challenges the model during training while fostering generalization to unseen tasks at test time. 
Experimental results on benchmark datasets showcase the superiority of our approach over existing unsupervised meta-learning baselines, establishing it as the new state-of-the-art in the field. Remarkably, our method achieves competitive results with supervised and self-supervised approaches, underscoring the efficacy of the model in leveraging generalization over memorization.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
meta-learning unsupervised learning in-context learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9ec192262b18481079fa767e28ee6401f3393a6f.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b59f33a965103bd5be4082e5ae16d82e741abf24.zip
Unsupervised Meta-Learning via In-Context Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission