CrabPascal's Lexical Engine: Tokenizing Delphi Code with Precision

The lexer forms the first stage of CrabPascal's compiler pipeline, handling Delphi's complex syntax—nested comments, string escapes, and hex notation—with exactitude. Though tokenization appears straightforward, it provides the precise position tracking and syntactic classification that downstream parsers and type checkers depend upon. Implemented in Rust, CrabPascal's lexer achieves both memory safety and performance, enabling millisecond-scale analysis of large codebases.