Correlation Dimension of Natural Language in a Statistical Manifold

arXiv:2405.06321v1 Announce Type: new
Abstract: The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barab\'asi-Albert process. Long memory is …

abstract algorithm arxiv cond-mat.stat-mech correlation cs.ai cs.cl fisher language language model manifold natural natural language scale space statistical type via