TED: Accelerate Model Training by Internal Generalization

arXiv:2405.03228v1 Announce Type: new
Abstract: Large language models have demonstrated strong performance in recent years, but the high cost of training drives the need for efficient methods to compress dataset sizes. We propose TED pruning, a method that addresses the challenge of overfitting under high pruning ratios by quantifying the model's ability to improve performance on pruned data while fitting retained data, known as Internal Generalization (IG). TED uses an optimization objective based on Internal Generalization Distance (IGD), measuring changes …

abstract arxiv challenge cost cs.lg dataset language language models large language large language models overfitting performance pruning ted training type