Yoshua Bengio: Deep Learning

Asked which difference between biological and artificial neural networks he finds most profound, Yoshua Bengio says there is so much unknown about biological networks that they may hold keys to improving artificial ones. One thing he has studied recently is credit assignment through very long time spans, which brains seem to manage but which artificial nets can do only inconveniently and not in a biologically plausible way. Studying this mismatch could help explain how brains work and suggest ideas to bring into artificial networks.

He clarifies that he means storing episodic memories that can later be accessed to infer the causes of present observations and to assign credit to decisions or interpretations made when those memories were stored, then changing how one would have reacted. Current networks such as LSTMs do fairly well over dozens or hundreds of time steps, but it gets harder over longer durations, whereas humans can revise a view from last year because of new evidence.

Fridman suggests efficient forgetting is a big part of it, and Bengio agrees there is a selection of what we remember, linking it to higher-level cognition: deciding what comes to consciousness and what gets stored in memory, which are not trivial either.

Asked about the weakest aspect of how deep networks represent the world, Bengio says state-of-the-art nets trained on images or text have only a very basic, low-level understanding of what explains their data, far less robust and abstract than ours. He suggests training that focuses on causal explanation, which current training does not do, and learning jointly about language and the world it refers to, rather than separately from images and from text, so that each side helps the other.