Authors
David Sussillo, LF Abbott
Publication date
2014/12/19
Journal
arXiv preprint arXiv:1412.6558
Description
Abstract: Training very deep networks is an important open problem in machine learning.
One of many difficulties is that the norm of the back-propagated error gradient can grow or
decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is
not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent
network, application to an FFN amounts to multiplying the error gradient by a different
random matrix at each layer. We show that the successive application of correctly scaled ...
One of many difficulties is that the norm of the back-propagated error gradient can grow or
decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is
not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent
network, application to an FFN amounts to multiplying the error gradient by a different
random matrix at each layer. We show that the successive application of correctly scaled ...
Total citations
20153
Scholar articles
D Sussillo, LF Abbott - arXiv preprint arXiv:1412.6558, 2014
Dates and citation counts are estimated and are determined automatically by a computer program.