[D] Can you extract the encoder part of an llm for feature extraction ?

I am still pretty new to this so this might be a dumb question. If you have an opensource model like the latest mixtral one, could you extract the layers that do the encoding and use that for feature extraction ? If so could it be worth it to try that over using BERT or ROBERTA ?

encoder encoding extract extraction feature feature extraction llm machinelearning mixtral opensource part question