Skip to content
AI Engineer Guide.

Tokenizer

Type text and see it split into tokens, with the characters-per-token ratio.

Approximation: split on words, numbers and punctuation. A real tokenizer merges frequent subwords, so its count will differ — use the model's tokenizer for billing.

Retrievalaugmentedgenerationgroundsanswersinevidence.
Tokens8
Characters59
Chars / token7.4

Related: Tokenization