On Chomsky and the Two Cultures of Statistical Learning
Chomsky's remarks were in response to Steven Pinker's question about the success of probabilistic models trained with statistical methods. This essay is a response to Pinker and Chomsky, and will address these questions: For example, a decade before Chomsky, Claude Shannon proposed probabilistic models of communication based on Markov chains of words. If you have a vocabulary of 100,000 words and a second-order Markov model in which the probability of a word depends on the previous two words, then you need a quadrillion (1015) probability values to specify the model. The only feasible way to learn these 1015 values is to gather statistics from data and introduce some smoothing method for the many cases where there is no data. Therefore, most (but not all) probabilistic models are trained. Also, many (but not all) trained models are probabilistic. As another example, consider the Newtonian model of gravitational attraction, which says that the force between two objects of mass m1 and m2