Lexical Analysis

Lexical Analysis
3.1 The Role of the Lexical Analyzer

  1. To read the input characters of the source program

  2. Group them into lexemes

  3. Produce as output a sequence of tokens for each lexeme in the source program

Scanning and Lexical analysis

3.1.1 Lexical Analysis Versus Parsing

Why do we separate lexer and parser?

  1. Simplicity of design is the most important consideration

  2. Compiler efficiency is improved

  3. Compiler portability is enhanced
    (only lexer has to deal with device-input)

3.1.2 Tokens, Patterns, and Lexemes

ex printf(“Hello World!");

printf and score are lexemes matching the pattern for token id.