Understanding ChatGPT API prices

AI can now produce a remarkably convincing amount of prose in a few seconds. But computational eloquence still comes with a bill. So if you’re planning to use its API, the price depends on a combination of tokens, models, and context. This article unpacks how those pieces fit together and what they mean in actual dollars.

Breaking into Tokens

The most basic unit required to reckon the cost of the API calls is a token. Each Large Language Model (LLM) has a different way of breaking text into tokens, it can be a single character or an entire word, usually spaces are attached to the beginning of the token that follows them.

OpenAI offers an online tool called Tokenizer, which counts the tokens that you type on a text box and displays them broken down with colors. At the bottom of the page it gives a rule of thumb:

A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).

That rule only applies to common English. Token counts can vary significantly across languages, so I used Google Translate to get a glimpse of how the same text may be tokenized:

Tokenizer results for different languages and symbols

Common English words and word pieces are often represented efficiently by the tokenizer, which lines up with the general rule that one token is approximately equal to 4 characters of common English.

Other Latin-script languages, such as Portuguese and German, can be split into smaller pieces and therefore use more tokens for comparable text.

For non-Latin scripts and emojis, token boundaries can be even less intuitive. Tiktoken uses byte-pair encoding over UTF-8 bytes, so a token does not necessarily correspond to a Unicode character, glyph, syllable, or word.

To programmatically count the tokens you can use Tiktoken, the official Python library. OpenAI’s Cookbook has instructions and alternatives for other languages like JavaScript, Go, Java and C#.

API Pricing

API calls are billed for the total of tokens on the input request and the output response. The text you send is broken down into tokens, which are added to the amount of its prompted response. For example, if you send 10 tokens in your query and receive 40 tokens as a response, a total of 50 tokens are billed.

OpenAI offers a range of prices across its models. GPT-4 and custom-trained models offer greater capabilities at a higher price, while GPT-3.5 Turbo is a lower-cost option suited to many chatbot applications.

New API accounts receive $5 in free trial credits, usable within the first three months. ChatGPT Plus and API billing are separate; if those credits expire or are spent, you’ll need to add billing information to continue using the API.

Using an online token calculator we can estimate how to spend those credits across API requests. For the cheapest GPT-3.5 Turbo model with a 4K context at $0.0015 per 1K tokens for input and $0.002 per 1K tokens for output as an example: if we sent 1000 tokens and got the same amount back, totalling 2000 tokens (or ~1500 words in English) at $0.0035 per execution, we could run about 1428 executions with that $5 credit. If each request used 2000 input tokens and generated 2000 output tokens, filling the 4K context, the same $5 would cover about 714 executions.

In this context, “context” refers to the total number of tokens that the model can hold to generate each response. Relevant context can improve a response, but adding more context also increases cost and does not automatically make the response more accurate.

Balancing context and cost

If you’re used to ChatGPT on the web, API responses can feel bare and disconnected because they are stateless and lack the context which the web version carries from the previous prompts. To connect responses you must explicitly pass the context, which can be done by saving the previous response and appending it to the next, increasing the amount of tokens on the requests and thus the costs.

Responses can be tweaked with several options, such as temperature and top_p to control the randomness of responses, presence_penalty and frequency_penalty to avoid repetition, and stop to stop the response at a certain token.

Keeping up a large context across requests or generating several long replies may reach the limit for the chosen model’s context options, the cheapest being a 4K context for the GPT-3.5 Turbo model, which also has a more costly version at 16K, and 8K or 32K for the GPT-4.

The costs for those higher models and context limits rise significantly, and the API will return an error if the limit is reached, so you have to find a balance between paying more for a larger context or spending less by splitting the requests and managing context in a more refined way across prompts.

For a simplified example, imagine using the API to review the contents of a book. The 16K context of the GPT-3.5 model would be enough to send 8K tokens (~6K words in English) and receive the same amount back, which roughly estimates to 13 pages of single-spaced Arial 12-point text. So each request sends 8K tokens for $0.003/1K tokens and receives 8K tokens for $0.004/1K tokens, totalling $0.056 per request. For a 1300-page book, that would be 100 requests and $5.6 in total.

Be mindful that using this will incur API costs: You can try out the API with the Playground (paid) to get a feel of how it works. The API reference has more details on the options and parameters accepted.

Limit spending

After defining a model and context that minimally fit your usage, the two factors that ramp up costs are the total amount of tokens and number of API calls in your application. You can specify the maximum number of tokens to be returned on each request, which can be used to limit the response size and cost.

This becomes relevant when integrating with a high volume of data, such as contents for books, articles, social media posts, e-commerce products, or heavy interaction with a large user base.

OpenAI has a Production best practices guide that recommends a few strategies to limit spending and manage costs. It states the limits for billing:

Once you’ve entered your billing information, you will have an approved usage limit of $120 per month, which is set by OpenAI. To increase your quota beyond the $120 monthly billing limit, please submit a quota increase request.

You can set limits at the Usage limits panel and there’s also a dashboard to track usage. And the API itself has rate limits for the number of requests per minute, tokens per minute (and per day on some models) which you’re allowed to make.

Moderating user interaction

Allowing users of your application to interact with the API requires some restraint to control costs and avoid abuse. OpenAI has a list of safety best practices and the free Moderation API can be used to filter out offensive content.

You might want to set your own validation for length and other criteria to make it more cost-efficient. And if there’s heavy user interaction, you probably want to define a cooldown period between requests on endpoints that call the API to avoid spamming it.