What problems in artificial intelligence cannot be addressed using artificial neural networks?

Answer by Yoshua Bengio:

It depends what you mean by 'neural network'. If you include in that category the learning algorithms yet to be discovered that explain the learning abilities of human brains, than obviously and by definition there are no AI problems that neural networks will not be able to solve. 

However, some of the main current limitations of *current* neural networks are the following, in my humble opinion:

– we probably do not yet have learning algorithms powerful enough to properly train them when the network is 'complex' (e.g. deep and able to represent high-level abstractions), i.e., current algorithms probably fall in sub-optimal apparent local minima (where much better solutions probably exist, and we know this sometimes happen, e.g. see my JMLR 2010 paper with Dumitru Erhan, http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf)

– we might also need progress on the front of reinforcement algorithms for neural networks (but this is not my area)

– we probably need progress on the front of fast inference algorithms (in the case of probabilistic neural networks such as Deep Boltzmann Machines, which may be necessary to really capture interesting top-down effects along with handling of uncertainty at all levels and not just at the output); the brain seems to do inference in just a few cycles whereas current approximate inference algorithms are 10x slower at least and still unsatisfactory in many cases (but see some good ideas e.g. in the Larochelle & Salakhutdinov paper http://www.dmi.usherb.ca/~larocheh/publications/aistats_2010_dbm_recnet.pdf)

Other researchers probably have other ideas of things to add in their 'wish lists'…

Also: it is generally believed that the ability to manipulate symbols and recursive structures is not in the realm of neural networks but this is not taking into account the fact that one could use recurrent or recursive neural networks, than can indeed represent symbolic and recursive structures. However, what is true is that the optimization problem for such networks is even harder (they are really very deep because each time step counts as extra depth) and their adequate training/optimization remains a difficult open problem.

Another possible misconception that is related is that neural networks are thought not to be able handle variable-size data-structures. This is easily achieved at least in principle by allowing sparse representations. There has been much work in using sparsity in deep architectures and neural networks in recent years. When you have a very long state vector (the brain has billions of neurons) that is sparse, it can easily encode 'variable-size' data structures (think of the size of the data-structure as the number of non-zeros). The boundedness of the brain is no more a limitation to this than is the boundedness of a typical computer memory.

What problems in artificial intelligence cannot be addressed using artificial neural networks?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s