Building a SQL Tokenizer and Formatter From Scratch — Supporting 6 Dialects

The article describes building a SQL tokenizer and formatter from scratch that supports six SQL dialects (PostgreSQL, MySQL, SQL Server, Oracle, SQLite, BigQuery). It uses a two-stage architecture—a single-pass character-by-character tokenizer followed by a state machine formatter—without requiring an AST. Key challenges include handling four different quoting styles, disambiguating wildcards from multiplication operators using context, and detecting compound keywords like ORDER BY through lookahead.