What are AI tokens? De­f­i­n­i­tion, func­tion­al­i­ty and cal­cu­la­tion.

AI tokens represent the smallest lin­guis­tic unit that AI models need to process and interpret text. With the help of AI to­k­eniza­tion, language is broken down into these building blocks, which form the basis for the analysis and gen­er­a­tion of texts. With tools such as the OpenAI Tokenizer, the text tokens can be de­ter­mined quickly and easily.

What are AI tokens?

AI tokens (ar­ti­fi­cial in­tel­li­gence tokens) are the smallest data units of AI models such as ChatGPT, LLama2 and Copilot. They are the most important building block for the pro­cess­ing, in­ter­pre­ta­tion and gen­er­a­tion of text, because only by breaking down a text into tokens can ar­ti­fi­cial in­tel­li­gence un­der­stand language and provide suitable answers to users’ queries.

How many AI tokens a text is made up of depends on various factors. In addition to the text length, the language used, and the AI model are also important. If you use an API access such as the ChatGPT API, the number of tokens also de­ter­mines which costs are incurred. In most cases, AI ap­pli­ca­tions charge for the AI tokens used in­di­vid­u­al­ly.

How does AI to­k­eniza­tion work?

The process by which an AI model converts text into tokens is called AI to­k­eniza­tion. This step is necessary because large language models require natural language in a machine-an­a­lyz­able form. To­k­eniza­tion therefore forms the basis for text in­ter­pre­ta­tion, pattern recog­ni­tion and response gen­er­a­tion. Without this con­ver­sion process, ar­ti­fi­cial in­tel­li­gence wouldn’t be able to grasp meaning and re­la­tion­ships. The con­ver­sion of text into tokens consists of several steps and works as follows:

1. Nor­mal­iza­tion: In the first step, the AI model converts the text into a stan­dard­ized form, which reduces com­plex­i­ty and variance. In the course of nor­mal­iza­tion, the entire text is converted into lower-case letters. The model also removes special char­ac­ters and sometimes restricts words to basic forms.
2. Text de­com­po­si­tion into tokens: Next, the AI breaks down the text into tokens, i.e., smaller lin­guis­tic units. How the text modules are broken down depends on the com­plex­i­ty and training method of the model. The sentence “AI is rev­o­lu­tion­iz­ing market research.” consisted of eleven tokens in GPT-3, nine tokens in GPT-3.5 and GPT-4 and only eight tokens in GPT-4o.
3. As­sign­ment of numerical values: Sub­se­quent­ly, the AI model assigns each AI token a numerical value called a token ID. The IDs are, in a sense, the vo­cab­u­lary of ar­ti­fi­cial in­tel­li­gence, which contains all the tokens known to the model.
4. Pro­cess­ing of the AI tokens: The language model analyzes the re­la­tion­ship between the tokens in order to recognize patterns and create pre­dic­tions or answers. These are generated on the basis of prob­a­bil­i­ties. The AI model looks at con­tex­tu­al in­for­ma­tion and always de­ter­mines sub­se­quent AI tokens based on the previous ones.

How are the tokens of a text cal­cu­lat­ed?

How tokens are cal­cu­lat­ed by the AI can be un­der­stood with the help of to­k­eniz­ers, which break down texts into the smallest pro­cess­ing units. They work according to specific al­go­rithms that are based on the training data and the ar­chi­tec­ture of the AI model. In addition to dis­play­ing the number of tokens, such tools can also provide detailed in­for­ma­tion on each in­di­vid­ual token, such as the as­so­ci­at­ed numeric token ID. This not only makes it easier to calculate costs, but also to optimize the ef­fi­cien­cy of texts when com­mu­ni­cat­ing with AI models.

An example of a freely ac­ces­si­ble tokenizer is the OpenAI Tokenizer, which is designed for current ChatGPT models. After you’ve copied or typed the desired text into the input field, the ap­pli­ca­tion presents the in­di­vid­ual AI tokens to you by high­light­ing the units in color.

What are some practical examples of AI tokens and to­k­eniza­tion?

To get a better idea of AI to­k­eniza­tion, we’ve written a short sample text to il­lus­trate it:

The GPT-4o model breaks down this text con­sist­ing of 269 char­ac­ters into 52 tokens, which looks as follows:

Close