lexer

Responsible for the lexical analysis stage of the Catspeak compiler. This stage converts UTF8 encoded text from individual characters into discrete clusters of characters called tokens.