First: What is a Lexeme?

A lexeme is simply the actual text written by the programmer. (krypticmouse)

Example:

int age = 25;

The lexemes are:

int
age
=
25
;

Think of lexeme as:

"The raw word or symbol found in the source code."


Then what is a Token?

A token is the category of a lexeme. (DEV Community)

Example:

int age = 25;
Lexeme Token
int KEYWORD
age IDENTIFIER
= ASSIGN
25 INTEGER_LITERAL
; SEMICOLON

So:

Lexeme = actual text
Token = meaning/category of that text

Why do we need Tokens?

Imagine parser gets this: