You can budget an AI prompt before you send it: count the tokens in your input, add the output you expect, and multiply each by your provider’s price per million tokens. Cleaning the text first, by removing duplicates, stray line breaks and formatting whitespace, lowers the count and the bill. This guide shows the formula, two worked examples, and the cleanup steps in order.
What a token is
Language models do not read letters or whole words. They read tokens: chunks of text that can be a word, part of a word, a space plus a word, a number or a punctuation mark. Common English words are often a single token. Rare words, names, code and numbers split into several.
OpenAI’s help center gives two rules of thumb for English: one token is roughly four characters, or about three quarters of a word. You can read the details in OpenAI’s explanation of tokens. Each model family uses its own tokenizer, so the same sentence can produce different counts on different models. Anthropic documents a token counting endpoint for exact numbers on Claude models, which is the right tool when a few percent matter.
Why the count matters
- Cost. API providers bill separately for input tokens (what you send) and output tokens (what the model writes). Output is usually priced higher per token.
- Context window. Every model has a maximum number of tokens it can handle in one request, including your prompt, any history, and the reply. Go over it and the request fails or older text is cut off.
- Speed. Longer prompts take longer to process.
The cost formula
cost per call = (input tokens ÷ 1,000,000 × input price)
+ (output tokens ÷ 1,000,000 × output price)Get the two prices from your provider’s pricing page on the day you budget, because they change. In the examples below we use made-up prices of $2 per million input tokens and $8 per million output tokens. They are not any provider’s real prices.
Estimating tokens from text
- Paste the text into the AI token counter. It divides your character count by four, the English rule of thumb, and shows estimated tokens, words and characters. Enter your own per-million prices to see an estimated cost.
- Check the word counter as a sanity check: tokens should be close to words ÷ 0.75.
- Add your expected output length. If you ask for a 200-word summary, that is about 270 tokens.
Treat the result as an estimate. English prose comes out close. Code, JSON, tables and non-English text usually use more tokens than the four-characters rule predicts, so add a margin of 20 to 30 percent for those, or use the provider’s own counter.
Worked example 1: summarising 1,000 articles
Each article is 12,000 characters, which is about 3,000 tokens, plus a 100-token instruction. You expect a 300-token summary.
- Input per call: 3,100 tokens. Across 1,000 calls: 3.1 million tokens.
- Output per call: 300 tokens. Across 1,000 calls: 0.3 million tokens.
- Input cost: 3.1 × $2 = $6.20
- Output cost: 0.3 × $8 = $2.40
- Total: $8.60 at the example prices
Now trim each article by 25 percent by removing navigation text, footers and repeated boilerplate. Input falls to about 2,350 tokens per call, 2.35 million overall, and input cost drops to $4.70. That is a saving of $1.50 on the input side for a few minutes of cleanup, and the saving scales linearly with volume.
Worked example 2: a chat that keeps growing
In a chat app, most APIs are stateless: each request resends the earlier messages. Suppose every turn adds 400 tokens (your message plus the reply).
- Turn 1 sends 400 tokens of history.
- Turn 10 sends about 4,000.
- Turn 20 sends about 8,000.
Total input over 20 turns is the sum 400 + 800 + … + 8,000, which is 84,000 tokens, not 8,000. Summarising old turns into one 300-token note after turn 5 keeps later requests small. The cost grows with the square of the conversation length if you never trim.
How to trim a prompt without losing meaning
- Delete what the model does not need. Email signatures, cookie notices, page menus and legal footers add tokens and no value.
- Remove exact duplicates. Logs, lists of URLs and exported spreadsheets often repeat lines. The duplicate line remover keeps one copy of each line.
- Fix stray line breaks. Text copied from a PDF has a hard break at the end of every line. The remove line breaks tool joins lines into paragraphs and collapses extra spaces. Blank lines and trailing spaces are tokens too.
- Format JSON before you inspect it. Run it through the JSON formatter to find the fields you do not need, delete them, and send the compact version. Pretty-printed JSON with deep indentation uses more tokens than minified JSON, so format to read it and minify to send.
- Cut examples to the useful ones. Two good examples usually do as well as eight.
- Cap the output. Ask for a length (“in three bullet points”) and set the provider’s maximum-output setting so a runaway answer cannot inflate the bill.
A before-you-send checklist
- Did I paste the text into a token counter and compare it with the model’s context limit?
- Is there room left for the reply?
- Have I removed duplicates, boilerplate and blank lines?
- Did I confirm today’s prices on the provider’s pricing page?
- For a batch job, did I run five items first and compare real usage with my estimate?
That last test is the most reliable. Most APIs return the actual token usage with every response, so run a small sample, compare it against your estimate, and adjust the formula before you run the full job.
Worked example 3: will it fit in the context window?
You want a model to answer questions about a 30-page report. The report is about 15,000 words.
- Estimated tokens: 15,000 ÷ 0.75 = 20,000.
- Add a 300-token instruction and a 500-token question: 20,800.
- Reserve 1,000 tokens for the answer: 21,800 tokens in total.
Compare 21,800 with the context window listed for your model in the provider’s documentation (Anthropic explains how context windows work). If the report is too long, split it by chapter with the PDF splitter, send the relevant chapter only, or summarise sections first. Sending everything every time is the most common way people overspend.
Tips for writing shorter prompts that still work
- State the task first. “Summarise the text below in three bullets for a busy manager” beats a long preamble.
- Use a consistent format. Lists and short labels cost fewer tokens than flowery sentences.
- Do not repeat instructions. Put rules in the system or instruction message once instead of in every user message.
- Remove politeness padding in batch jobs. Models do not need “please” and “thank you” across 10,000 calls.
- Send IDs, not blobs. If a record has 40 fields and you need three, send three.
- Test shorter versions. Cut a paragraph, run five samples, and compare quality. Keep the shorter prompt if the answers hold up.
Compare two versions of a prompt in the diff checker to see exactly what you removed, which makes it easier to spot an accidental loss of a key instruction.
Minified or pretty JSON: a quick comparison
Take a small record: {"id": 7, "name": "Ana", "tags": ["a", "b"]}. Pretty-printed with four-space indentation, each field sits on its own line with leading spaces, and the character count roughly doubles. Characters are not the same as tokens, since runs of spaces are often merged, but the extra newlines and indentation still add up across thousands of records. A good routine is to format JSON when you read it, then strip the whitespace before you send it to a model. Paste both versions into the token counter once to see the difference for your own data.
Where the estimate goes wrong
Reasoning models can spend extra output tokens on internal thinking that you are billed for. Images and file attachments are counted by their own rules. Non-English text, emoji and source code can use noticeably more tokens than the English rule suggests. When the budget is tight, use the provider’s counter, then use the AI token counter for quick what-if checks as you edit.