Fine-tuning Alpaca 30b 4-bit on consumer hardware - background

I want to write about fine-tuning Alpaca 30b 4-bit on consumer hardware, but before I can, I'll need to give a little background. My basic goal was to figure out "what's the most powerful AI I can customize and run on my shiny new 4090."

The answer right now is LLaMA 30b. Normally, fine-tuning this model is impossible on consumer hardware due to the low VRAM (clever nVidia) but there are clever new methods called LoRA and PEFT whereby the …

alpaca basic consumer figure fine-tuning hardware llama shiny