huggingface 套件使用备忘

last modify

Keywords: huggingface

• 组件

  • transformers

  • PEFT

• 案例

组件

Hugging Face - Documentation

transformers

模型库

• 文档: Transformers Doc
• 仓库: huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

PEFT

Parameter-Efficient Fine-Tuning (PEFT)

训练模型时只微调部分参数, 常见方法: LoRA, P-Tuning, Prefix Tuning 等;

• 文档: PEFT Doc
• 仓库: huggingface/peft: 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

LoRA: LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS Prefix Tuning: Prefix-Tuning: Optimizing Continuous Prompts for Generation, P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks P-Tuning: GPT Understands, Too Prompt Tuning: The Power of Scale for Parameter-Efficient Prompt Tuning AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

案例

• tloen/alpaca-lora: Instruct-tune LLaMA on consumer hardware

  使用 lora 微调 alpaca;