Probabilistic Models of Human Sentence Processing
Probabilistic models of human sentence processing (Jurafsky, 1996; Corley & Crocker, 1996) offer a principled way of combining grammatical knowledge with knowledge about the distributional properties of linguistic structures. A number of researchers have recently extended this approach by incorporating information-theoretic notions into models of sentence processing (Hale, 2001; Genzel & Charniak, 2002). These information-theoretic models rely on the hypothesis that the amount of information conveyed by a word or sentence predicts the processing effort associated this word or sentence. In this talk, we present a series of studies that test this hypothesis using eye-tracking data. Such data provide a moment-by-moment record of sentence processing: words that are more difficult to process cause longer eye-fixations, or induce reverse eye-movements. We use these data to investigate the predictions of Genzel & Charniak's (2002) entropy rate principle. This principle states that speakers produce text such that the entropy rate remains constant from sentence to sentence. This makes a number of predictions: (a) entropy rate and processing effort are correlated, (b) in connected text, processing effort is independent of sentence position, and (c) for isolated sentences, entropy rate (and hence processing effort) increases with sentence position. Using a corpus of eye-tracking data, we show that predictions (a) and (b) are borne out. We also present the results of a reading experiment which confirms prediction (c). Corley, S., & Crocker, M. W. (1996). Evidence for a tagging model of human lexical category disambiguation. In Proceedings of the 18th Annual Conference of the Cognitive Science Society, Mahwah, NJ. Lawrence Erlbaum Associates. Genzel, D., & Charniak, E. (2002). Entropy rate constancy in text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, (pp. 199-206), Philadelphia. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the 2nd Conference of the North American Chapter of the Association for Computational Linguistics, Pittsburgh, PA. Jurafsky, D. (1996). A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science, 20, 137-194.