A method for designing a lexical analyzer

Description

The scanner is a subroutine which is frequently called by an application program like a compiler. The primary function of a scanner is to combine characters from the input stream into recognizable units called tokens. A method has been presented in this paper for designing such a scanner, also frequently referred to as a lexical analyzer in the current literature. The major steps involved in this design process are: identification of tokens, construction of a state diagram, building driver tables and finally writing a scanning routine. The rules for generating the driver tables are described and an algorithm for the scanner, utilizing these driver tables, is included. The method has been successfully used to build the system scanner for a user oriented plotting language. It is concluded that the method is well defined, gives rise to a modular design and as such easily lends itself to language extensions.