How to test statistical significance on text data?

Web: https://www.reddit.com/r/LanguageTechnology/comments/sbhz36/how_to_test_statistical_significance_on_text_data/

So, I was in an interview and I was asked so many questions about statistical details on text data. For example 1. How would you sample million sentences from billions of sentences? What strategies will you use for sampling?

1. Having sampled, how would determine that the sampled data follows actual data distribution? (In nutshell how would you determine whether two text data distributions are similar or not)

Follow up for these questions were, When will you decide to re-train your …

data languagetechnology statistical test text