Reducing the dimensionality and granularity in hierarchical categorical variables

arXiv:2403.03613v1 Announce Type: cross
Abstract: Hierarchical categorical variables often exhibit many levels (high granularity) and many classes within each level (high dimensionality). This may cause overfitting and estimation issues when including such covariates in a predictive model. In current literature, a hierarchical covariate is often incorporated via nested random effects. However, this does not facilitate the assumption of classes having the same effect on the response variable. In this paper, we propose a methodology to obtain a reduced representation of …

abstract arxiv categorical current dimensionality effects hierarchical however literature overfitting predictive random stat.me stat.ml type variables via