The Deep Connection Between Boosting and LASSO

Understanding a powerful algorithm through a radical perspective shift

Gradient boosting is both ubiquitously used and ubiquitously underappreciated.

When discussing its effectiveness, people usually default to two perspectives:

1. Gradient boosting starts with low variance and high bias, and looks to decrease bias by adding more weak learners
2. Gradient boosting approximates “gradient descent” in a function space, and looks to minimize the loss in this space

Both of these perspectives are interesting and highly …

boosting data science decision-tree gradient-boosting lasso linear regression machine learning