Web: http://arxiv.org/abs/2206.05794

Sept. 29, 2022, 1:13 a.m. | Tomer Galanti, Zachary S. Siegel, Aparna Gupte, Tomaso Poggio

stat.ML updates on arXiv.org arxiv.org

We analyze deep ReLU neural networks trained with mini-batch Stochastic
Gradient Descent (SGD) and weight decay. We show, both theoretically and
empirically, that when training a neural network using SGD with weight decay
and small batch size, the resulting weight matrices tend to be of small rank.
Our analysis relies on a minimal set of assumptions; the neural networks may be
arbitrarily wide or deep and may include residual connections, as well as
convolutional layers. The same analysis implies the …

arxiv bias low networks neural networks

More from arxiv.org / stat.ML updates on arXiv.org


@ METRO/MAKRO | Nanterre, France

Data Analyst

@ Netcentric | Barcelona, Spain

Power BI Developer

@ Lendi Group | Sydney, Australia

Staff Data Scientist - Merchant Services (Remote, North America)

@ Shopify | Dallas, TX, United States

Machine Learning / Data Engineer

@ WATI | Vietnam - Remote

F/H Data Manager

@ Bosch Group | Saint-Ouen-sur-Seine, France

[Fixed-term contract until July 2023] Data Quality Controller - Space Industry Luxembourg (m/f/o)

@ LuxSpace Sarl | Betzdorf, Luxembourg

Senior Data Engineer (Azure DataBricks/datalake)

@ SpectraMedix | East Windsor, NJ, United States

Abschlussarbeit im Bereich Data Analytics (w/m/div.)

@ Bosch Group | Rülzheim, Germany

Data Engineer - Marketing

@ Publicis Groupe | London, United Kingdom

Data Engineer (Consulting division)

@ Starschema | Budapest, Hungary

Team Leader, Master Data Management - Support CN, HK & TW

@ Publicis Groupe | Kuala Lumpur, Malaysia