LLM Latency Calculator
Estimate expected LLM response latency — time-to-first-token and total response time — for planning real-time AI applications.
Did this tool work for you?
How to use this calculator
- 1
Select the LLM you are planning to use.
- 2
Enter the expected input token count (your prompt length) and output token count (desired response length).
- 3
Select the expected server load — low for off-peak hours, high for peak traffic.
- 4
Review TTFT and total response time to determine if the model fits your latency budget.
Frequently asked questions
What formula does the latency calculator use?
Total time = TTFT + output tokens ÷ tokens per second. TTFT and tokens per second start from a per-model baseline at medium load, then the load level rescales them: TTFT × 0.7, 1.0 or 1.8 for low, medium or high, and tokens per second ÷ the same multiplier. Both are rounded to whole numbers before the total is computed.
Where do the baseline latency numbers come from?
From two lookup tables in the code, commented as approximate figures at medium load. They are static and are not measurements or benchmarks, so real latency can differ by region, provider load, prompt length and model version. Measure the endpoint you plan to use.
Does the number of input tokens change the result?
No. The input tokens field is accepted but is not used in the calculation, so a 100,000-token prompt returns the same result as a 500-token one. In reality, longer prompts generally take longer before the first token appears, so results for large prompts are optimistic.
How is the real-time verdict decided?
From total response time only: under 3 seconds is Yes, suitable for real-time chat; under 8 seconds is Marginal, consider streaming; otherwise No, use async or background processing. TTFT alone is not part of the label.
What does time to first token mean?
It is the delay between sending the request and the first generated token arriving. In a streamed interface it is roughly how long the user waits before anything appears; the rest of the response then appears at the tokens-per-second rate.
Does streaming change the total time?
Not in this tool: total time is the same formula either way. Streaming changes what users experience, because they can read from the first token onward instead of waiting for the full response.
What do the low, medium and high load options do?
They are fixed multipliers of 0.7, 1.0 and 1.8 applied to the baselines. They are not derived from any provider data, so use them to see sensitivity rather than to predict peak behaviour.
What is left out of the estimate?
Network round trips, queueing beyond the three load levels, tool calls, retries, rate-limit delays and any hidden reasoning tokens generated before the visible answer.
LLM latency calculator: time to first token and total response time
The two-part formula
Total time = time to first token (TTFT) + output tokens ÷ tokens per second. Both values come from lookup tables in the code, one pair of numbers per model at medium load, and the load setting rescales them: TTFT is multiplied by 0.7 at low load, 1.0 at medium and 1.8 at high, while tokens per second is divided by the same multiplier. Results are rounded to whole milliseconds and whole tokens per second before generation time is calculated, so the on-screen figures reproduce exactly from each other. Output is clamped to at least one token.
Baselines stored in the code
These are static, approximate figures built into this tool, not measurements. Real latency depends on region, prompt length, provider load, streaming settings and model version, so benchmark the endpoint you plan to call. Model names appear as listed in the dropdown, without release dates.
| Model entry | TTFT (ms) | Tokens per second |
|---|---|---|
| GPT-4o | 400 | 60 |
| GPT-4o-mini | 200 | 120 |
| Claude Sonnet | 600 | 70 |
| Claude Haiku | 200 | 150 |
| Gemini 1.5 Pro | 400 | 80 |
| Gemini 1.5 Flash | 150 | 200 |
| Llama 3 70B | 300 | 40 |
| Llama 3 8B | 100 | 120 |
Worked examples
Latency runs. The columns show Run, TTFT, Tokens/s, Total time and Verdict, one line per case.
| Run | TTFT | Tokens/s | Total time | Verdict |
|---|---|---|---|---|
| GPT-4o, medium load, 300 output tokens (defaults) | 400 ms | 60 | 5.40 s | Marginal |
| GPT-4o, low load, 600 output tokens | 280 ms | 86 | 7.26 s | Marginal |
| GPT-4o, medium load, 600 output tokens | 400 ms | 60 | 10.40 s | No |
| GPT-4o, high load, 600 output tokens | 720 ms | 33 | 18.90 s | No |
| Gemini 1.5 Flash, low load, 150 output tokens | 105 ms | 286 | 0.63 s | Yes |
| Llama 3 70B, high load, 800 output tokens | 540 ms | 22 | 36.90 s | No |
How the verdict works
The real-time label looks only at total time: under 3 seconds is Yes, under 8 seconds is Marginal, anything longer is No. Streaming is not part of it, so a response that streams can feel faster than the label suggests, because users start reading after the TTFT instead of after the total. If your product shows tokens as they arrive, use the TTFT line for responsiveness and the total line for how long the connection stays open. The GPT-4o runs at 600 output tokens show that output length dominates: 280 ms of TTFT matters little against nearly seven seconds of generation.
What the model leaves out
The input-token field is accepted but does not enter the formula, so a very long prompt returns the same answer as a short one. Real processing time before the first token grows with prompt length, so treat results for long prompts as optimistic. Network round trips, queueing beyond the three load levels, tool calls, retries and any reasoning tokens generated before the visible answer are also outside the calculation. Use the output to compare shapes and sensitivities, then confirm with a timed test against the actual endpoint.
Learn more from an authoritative source:
OpenAI Platform DocsAI Token Counter
Estimate the number of tokens in your text for GPT-4, Claude, Gemini, and other LLMs. Useful for staying within context limits.
AI Prompt Cost Calculator
Calculate the cost of an AI API call based on input/output tokens and model pricing.
Words to Tokens Converter
Convert between words, characters, tokens, and pages for AI models and content planning.
AI API Budget Calculator
Plan your monthly AI API budget based on usage volume, model selection, and request patterns.
Context Window Checker
Check if your text fits within the context window of popular AI models. Paste your prompt and see which models can handle it.
LLM Cost Comparison Calculator
Compare monthly token costs across major LLM models — GPT-4o, Claude, Gemini, and more — for your specific workload.
Results are estimates for informational purposes only and do not constitute professional financial, medical, legal, or technical advice. Read full disclaimer →