How to define Tokens for C Language using LEX

LEX is a compiler to test the rules for the tokens for any given programming language. LEX is written in C. Token generation is the first phase of the Compiler called as Lexical Analysis. The input for this phase is the program statements written in High Level Language like C, C++ or JAVA etc and the output is the Tokens means meaningful constructs of any programming language.

For example, consider the following statement written in C language.

int a=2;

then the tokens are keyword, variable, operator, digit and the delimiter. int, a, = and 2 are called as the lexemes for the above tokens.

Lexeme                          Token
int                                   keyword
a                                     variable
=                                    operator
 2                                   digit
;                                     delimiter

In the LEX program these tokens are represented by the set of rules that is with the Regular Expression. A regular expression is, in theoretical computer science and formal language theory, a sequence of characters that define a search pattern.

LEX program to find the tokens for the following statement

a=2;

A LEX file consists of three section. The Definition section, the Rule section and the Routine section. In the Rule section the rules has to be written for each of the tokens. The tokens are generated according to that rule.

/*The Definition Section*/
%{
               #include<stdio.h>
               #include<string.h>
%}

/*The Rule Section*/
%%
[a-zA-Z][a-zA-Z0-9]*   {printf("Variable \t");}
[0-9]*                             {printf("Digit \t");}
=                                     {printf("Operator \t");}
;                                      {printf("Delimiter \t");}
%%

/*The Routine Section*/
int main()
{
                   yylex();
                   return 0;
}
 yywrap()
{
                   return 1;
}

[a-zA-Z][a-zA-Z0-9]*   {printf("Variable \t");}  Here the first part is the Regular Expression and second part is the Action part. The Rule Section is always enclosed in %% sign.

How to run?

Let the file name is lex.l then run the following commands.

After compilation with LEX compiler a c file is generated lex.yy.c. Compile this c file with c compiler and run it.