Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
- Version
- Download 1005
- File Size 895.35 KB
- File Count 1
- Create Date 2023-11-27
- Last Updated 2023-11-27
Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Titel: Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Autor: Pan, Kaihang ; Li, Juncheng ; Song, Hongye ; Lin, Jun ; Liu, Xiaozhong ; Tang, Siliang
Abstract:
Prompt tuning is a parameter-efficient method, which learns soft prompts and conditions frozen language models to perform specific downstream tasks. Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts. On the other hand, it can easily overfit to few-shot training samples, thereby undermining generalizability. Existing works leverage pre-training or supervised meta-learning to initialize soft prompts but they fail to data-efficiently generalize to unseen downstream tasks. To address the above problems, this paper proposes a novel Self-sUpervised meta-Prompt learning framework with MEta-gradient Regularization for few-shot generalization (SUPMER). SUPMER leverages self-supervised meta-learning with a diverse set of well-designed meta-training tasks to learn a universal prompt initialization for efficient adaptation using only unlabeled data. Additionally, it jointly meta-learns a gradient regularization function to transform raw gradients into a domain-generalizable direction, thus alleviating the problem of overfitting. Extensive experiments show that SUPMER achieves better performance for different few-shot downstream tasks, and also exhibits a stronger domain generalization ability. The code for SUPMER will be available at https://github.com/beepkh/SUPMER.
Attached Files
File | Action |
---|---|
2303.12314.pdf | Download |