Two different jobs
A lexer groups characters into tokens. A parser groups tokens into syntactic structures according to a grammar. Keeping this boundary clear makes diagnostics and testing easier.
A tiny expression
Source: total + 42
Tokens: IDENT(total), PLUS, INTEGER(42), EOF
Tree: Add(Name(total), Integer(42))Preserve source spans
Keep start and end positions on tokens and syntax nodes. A later type error should still point to the original source.
What comes next
Semantic analysis resolves names and checks types. It should not have to reconstruct source positions discarded by the parser.
Explore the language and compiler projects.