Is perplexity a good measure for evaluating language models?

I fine-tuned a language model on a dataset for sentence generation and decided to use perplexity to evaluate that model. However, I noticed that visually, the many of the sentences that had lower perplexities made less sense than the ones with high perplexities. Perhaps, i'm misunderstanding this metric but aren't sentences that make more sense supposed to have lower perplexities? I'm a beginner in this area, so if anyone could enlighten me, I'd appreciate it.

good language language models languagetechnology