The process by which a stream of characters is grouped into a series of lexical items, or tokens, and all available dictionary data is associated with the lexical items. Lexical analysis comprises three separate steps: segmentation, normalization and annotation.