lexical analyzer

views updated

lexical analyzer (scanner) The part of a compiler that breaks up the input into meaningful units, e.g. names, constants, reserved words, operators. The lexical analyzer will also remove redundant characters, e.g. spaces, and may deal with character-set mappings, e.g. replacing upper-case letters by the equivalent lower-case letters. The units recognized by the lexical analyzer are called tokens, and are output in some conveniently coded form for subsequent processing by the compiler.