Natural language processing (source code)

= Natural language processing

<Natural language processing> studies computational methods for analyzing, transforming and generating human language. Tasks include translation, extracting information from text, and predicting a missing or subsequent word. A system must resolve ambiguity from context: “bank” in a sentence about a river and in a sentence about a loan has different meanings.

A <neural language model> tackles one precise part of the problem by assigning <probabilities> to word sequences. For example, after “the cat sat on the”, its distribution can rank “mat” above “democracy”, without thereby proving that it understands the scene. Evaluating a specific task and its errors is more informative than treating fluent text as a universal test of language competence.<Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig>[Chapters 23–24, natural language processing and deep learning for language]{ref}