LEX
ER
Lexical Analyzer · Spring 2024
Emmanuel Montoya
· Rouni Assaf · Tony Dinh
Removes whitespace & comments, then tokenizes source code.
Keyword
Identifier
Literal
Operator
Separator
Comment (stripped)
// input source
load sample
clear
▶ tokenize
// token output
⬡
Paste code on the left and hit
▶ tokenize