[D] Fine-tune Phi-3 model for domain specific data - seeking advice and insights

Hi,

I am currently working on fine-tuning the Phi-3 model for financial data. While the loss is decreasing during training, suggesting that the model is learning quite well, the results on a custom benchmark are surprisingly poor. In fact, the accuracy has decreased compared to the base model.

Results I've observed:

* Phi-3-mini-4k-instruct (base model): Average domain accuracy of 40%
* Qlora - Phi-3-mini-4k-instruct (fine-tuned model): Average domain accuracy of 35%

I have tried various approaches, including QLora, Lora, and …

accuracy advice benchmark data domain financial fine-tuning insights loss machinelearning phi phi-3 results training while