Your LLM Benchmark Is Probably Measuring the Wrong Reality

Most LLM benchmarks begin with a clean table. Models go down the left, scores run across the top, and somebody highlights the largest number in green.

Production traffic does not look like that table.

It looks like a half-debugged Python script pasted at 02:13, a 4,000-token legal document, a student asking for a proof, an awkward relationship question, a role-play thread, and a user who keeps saying "shorter" until the answer fits in a Slack message. These requests do not merely test different skills. They create different cost curves, latency profiles, safety risks, and interaction patterns.

The AI Observatory is an unusually useful attempt to measure this mess. It aggregates 24,521 real-world AI conversations and 92,493 user–assistant exchange pairs from seven sources, covering samples dated from 2023 to 2026 and 52+ models. Every conversation is mapped onto the same 145-feature taxonomy, spanning prompts, responses, turns, topics, functions, media formats, interaction styles, and sensitive-use flags.

The headline result is blunt: there is no generic thing called "real AI usage." There are only samples of it, and every sampling method bends the picture.

A dataset is a product surface wearing a lab coat

The Observatory combines WildChat, ShareGPT, AI Archive, Grok, LMSYS-Chat, Chatbot Arena, and the National Internet Observatory dataset. On paper, they are all collections of human–AI conversations. In practice, their collection mechanisms are wildly different.

WildChat came through a free API intermediary and skews toward long, code-heavy conversations. ShareGPT contains chats people voluntarily exported because they considered them worth sharing. Chatbot Arena attracts users who compare two models and actively probe their behavior. Grok conversations are public and tied to X, which likely helps explain why news and current affairs show up far more often. The NIO sample comes from privacy-preserving browser instrumentation rather than public sharing.

That provenance is not a footnote. It is part of the data-generating process.

The resulting distributions are nowhere near interchangeable:

• Grok conversations are tagged for News & Current Affairs 38.5% of the time and Business & Society 64.5% of the time.
• Academic-policy issues range from 44.0% in WildChat to 73.6% in AI Archive.
• Misinformation or misrepresentation appears in 39.0% of Grok conversations, versus 3.0% to 28.2% elsewhere.
• The mean WildChat prompt is 569.5 tokens, compared with 51.8 to 181.0 tokens in the other sources.
• Grok responses average 1,322.9 tokens.
• ShareGPT and AI Archive average 4.42 and 4.34 turns per conversation, while several other sources sit much closer to one or two.

If you evaluate a model on one of these sources and call the result "real-world quality," you have silently shipped the source's collection bias into your conclusion.

This is the same mistake as benchmarking a database using only point reads, then discovering that production is 60% range scans and batched writes. The benchmark may be reproducible. It may even be useful. It is still measuring the wrong workload.

"Work" is only half the traffic

Many reports about AI usage focus on economic impact, so they first filter for occupational tasks. That makes sense if the research question is productivity. It becomes dangerous when the findings are retold as a general account of how people use AI.

The Observatory applies a Clio-style occupational classifier across its pooled data. It finds that 47.9% of conversations would be classified as non-occupational, with the share ranging from 34.2% in AI Archive to 61.9% in LMSYS.

Nearly half the traffic disappears before the work analysis begins.

What gets removed is not noise. It includes personal advice, relationships, health and psychological interests, culture, fiction, entertainment, sexual content, jailbreak attempts, and other safety-relevant behavior. Those conversations may contribute little to a labor-market chart, but they matter a great deal to a product team responsible for abuse handling, privacy, user trust, and incident response.

This is the core measurement bug: scale cannot rescue missing scope. Ten million occupational conversations still tell you almost nothing about the failure modes concentrated in non-occupational use.

For builders, the implication is practical. Do not maintain one golden evaluation score. Keep separate slices for coding, retrieval, editing, long-context synthesis, personal advice, adversarial prompts, and whatever else actually crosses your API. A model can improve on the global average while regressing badly on the slice that pages your on-call engineer.

Your traffic is drifting while you benchmark it

The Observatory also tracks WildChat from April 2023 to July 2025. Over that period:

• Mean prompt length increased by 1,049.5%.
• Mean response length increased by 100.6%.
• Turns per conversation increased by 17.1%.
• Prompts containing code increased by 15.2%.
• Responses containing code increased by 16.6%.
• Formatted lists in responses increased by 25.8%.

Some content and interaction signals moved in the opposite direction: adult and illicit content, academic-policy flags, self-disclosure, and "continue the user's text" answer forms all became less prevalent.

This is concept drift with a token bill attached.

A prompt-length jump of roughly 10× changes more than quality. It changes time to first token, context-window pressure, cache economics, rate-limit behavior, and the probability that an upstream timeout kills the request. Longer, more iterative conversations also make state management and retry semantics more important. A routing policy tuned on 2023 traffic can be perfectly implemented and still be wrong for 2025 traffic.

Static eval sets age in the same way load tests age. The test itself continues to pass; the production workload quietly moves somewhere else.

Model names are weak proxies for usage regimes

The Observatory's model-level comparisons are especially relevant to teams sitting behind an API gateway. Even after controlling for source and time, model variants support different interaction patterns.

Within WildChat, GPT-3.5 variants are associated with shorter, more template-like exchanges. GPT-4o supports longer and more iterative assistance. Reasoning-oriented models such as o1 skew toward long, one-shot technical problem solving.

In Chatbot Arena, Gemini and Grok responses are 83.8% and 77.3% longer than the comparison pool, while Claude responses are 60.8% shorter. In the NIO sample, Gemini conversations are more than twice as long as ChatGPT conversations and lean further toward retrieval and follow-up work.

These numbers should not be read as a universal ranking. That would repeat the exact error the project exposes. They show that changing the model can change the shape of the interaction, not only the answer quality.

This creates a feedback loop:

1. A model responds in a particular style.
2. Users learn what it is good at and adapt their prompts.
3. The traffic distribution shifts.
4. Cost, latency, and safety behavior move with it.
5. Yesterday's routing rule becomes today's weird production incident.

The model is part of the workload generator.

What an evaluation stack should look like

If the Observatory's findings hold for your traffic, an LLM evaluation system should behave less like a leaderboard and more like an observability pipeline.

Start with a versioned taxonomy. You do not need 145 labels on day one, but you do need stable dimensions for task, topic, turn structure, input and output length, media, tool use, refusal behavior, and safety-relevant content. Without a schema, every incident becomes an anecdote and every model migration becomes a vibes-based argument.

Then stratify before you aggregate. Compare models on the same traffic slice and time window. If one model receives mostly coding traffic and another receives mostly casual Q&A, the raw averages are operational fan fiction.

Track joint metrics. Quality without latency is a demo. Quality and latency without cost is a surprise invoice. Cost without failure rate hides retries. A useful route-level record might look conceptually like this:

{
  "route": "code-debugging/long-context",
  "model": "provider/model-version",
  "prompt_tokens": 6120,
  "completion_tokens": 940,
  "ttft_ms": 740,
  "total_ms": 6840,
  "retries": 0,
  "task_success": true,
  "safety_flags": []
}

Finally, keep privacy boundaries hard. Real prompts are valuable precisely because they are real, which also makes them sensitive. Prefer consented sampling, redaction, short retention, access controls, and aggregate features over a permanent warehouse of raw conversations. The Observatory's NIO component is a useful reminder that measurement does not require publishing the underlying text.

The gateway is where the benchmark can meet reality

An OpenAI-compatible gateway has an interesting vantage point: it sees the request shape before routing and the response shape after completion. That makes it a natural place to attach consistent measurement across providers—if the implementation is privacy-conscious and the user controls are explicit.

This is also why model portability matters. With a common API, teams can replay consented evaluation slices, shadow-test a candidate model, or route a narrow task class without rewriting the application around every provider. The useful question stops being "Which model won?" and becomes:

Which model, on this traffic slice, at this point in time, meets our quality target within the latency, cost, and safety budget?

That question has more variables. It is also the one production eventually asks.

A2Agent provides access to multiple model families through one OpenAI-compatible API. The compatibility layer is the simple part. The harder—and more interesting—work is using that flexibility without pretending all requests, users, datasets, and models are interchangeable.

The AI Observatory does not hand us a final benchmark. It demonstrates why a final benchmark is the wrong abstraction.

Production is a moving distribution. Measure it like one.

---

Further reading: Explore The AI Observatory and its public annotation dataset on Hugging Face.