Sentence-transformer Bert model performs worse after fine-tuning

I'm using symanto/sn-xlm-roberta-base-snli-mnli-anli-xnli from HuggingFace. After multiple tries with different batch sizes, epochs, learning rates and even different unsupervised learning models methods such as this, I couldn't get my sentence transformer to perform better than raw model straight from HuggingFace. I'm not sure what I'm doing wrong. I'm sure there are no bugs in my code since I followed the sentence transformer models almost verbatim.

background on my task: my datasets consists of a list of sentences(legal articles— around …

bert fine-tuning languagetechnology transformer