Choosing the Right LLM for Grounded RAG (Without Overpaying for Reasoning)

Retrieval-augmented generation looks like a solved problem until you have to pick the model. The choice most teams face today is between a fast, inexpensive model like GPT-6 Luna, run with its reasoning switched off, and a frontier reasoning model like GPT-6 Astra, which costs a hundred times more per token. The right answer depends less on leaderboards than on the shape of your context and your queries, and on knowing what each answer actually costs you.

The two models are built for different jobs

GPT-6 Luna is OpenAI's most efficient model, built for focused, high-volume work. At $0.10 per million input tokens and $0.50 per million output, it costs a hundredth of Astra's price on both sides (OpenAI). Every GPT-6 model can reason, but Luna lets you set reasoning effort to none, the setting OpenAI suggests for fast information retrieval and classification. For grounded question answering, where every fact the model needs is already in the prompt, that profile is a good fit: quick, cheap, and with no deliberation to pay for on questions that don't need any.

GPT-6 Astra sits at the other end. Its lowest reasoning setting is low; there is no off switch (OpenAI). Its headline long-context result is on OpenAI's MRCR v2 8-needle test, where it scored 100% at 256K–512K tokens and 96.3% at 512K–1M, against 91.5% and 73.8% for GPT-5.6 Sol (Vellum's breakdown of OpenAI's results). That is a real leap, but note where it was measured: at context lengths most RAG pipelines never reach, and past the 272K-token line where OpenAI doubles the input price for the entire request.

One thing has changed underneath this comparison. Reasoning used to be a choice of model. In GPT-6 it is mostly a dial, from none to max, and Luna ships set to medium, so a team that never touches the setting pays for reasoning on every lookup. GPT-6.1 Terra and GPT-6.1 Sol sit between the two models, and everything below applies along that ladder.

What actually happens at 80K tokens

Most production RAG prompts land in the tens of thousands of tokens, not the hundreds. At around 80K, the picture is more balanced than the benchmark deltas suggest, and both models stay well under the 272K threshold where long-context pricing starts.

Single-fact retrieval is the easy case for a small model: find the relevant passage, answer from it. Where small tiers weaken is multi-needle disambiguation and multi-hop synthesis, combining a fact from page 12 with a fact from page 200. The published evidence comes from the previous generation, and it reads as a warning rather than a verdict: on the same 8-needle MRCR test at 256K–512K tokens, GPT-5.6 Luna scored 41.3% to GPT-5.6 Sol's 91.5% (Vellum). We have not seen a comparable figure published for GPT-6 Luna, so treat its multi-needle behavior at your context length as something to measure, not assume. All transformer models also show some "lost in the middle" attention decay, and older generations show more of it.

GPT-6 Astra at 80K is operating near its ceiling, so its advantage is real but concentrated in the hard tail: scattered evidence, cross-document reasoning, and questions where a confidently wrong answer is expensive.

Can prompt engineering close the gap?

Partially. Because a Luna call costs so little, you can afford to scaffold deliberation into it:

1. Evidence extraction first. Have the model quote the relevant passages before answering, converting one hard attention problem into two easier steps.
2. Grounding verification. Add a pass that checks every claim in the draft answer against the extracted evidence, and removes anything unsupported.
3. Structured outputs, ordered correctly. Put evidence and reasoning fields before the answer field in your schema. Generation is autoregressive; if the answer comes first, the reasoning is post-hoc rationalization.

This meaningfully improves grounded accuracy. But it cannot fix retrieval-over-context failures: if the model never surfaces the second fact into its scratchpad, no amount of reasoning over the scratchpad recovers it. And the scaffolding generates several times more output tokens. So does turning up the reasoning dial, because OpenAI bills reasoning tokens as output (OpenAI). Either route erodes the very cost and latency advantage that made the smaller model attractive.

The routing answer

In practice the strongest architecture is not one model but a routed system: a cheap model at little or no reasoning effort for lookup-style questions, and a frontier model, or a higher effort setting, for synthesis-heavy or high-stakes ones. The decision of where to draw that line should come from your own evaluation set (50–100 real queries against your real prompts, graded for 'groundedness'), because at moderate context lengths the gap is workload-dependent.

What the router actually costs

A router is a cost claim until the bill confirms it, and list prices make a poor proxy. On paper the arithmetic is simple. At 80,000 input tokens and 1,000 output tokens, list prices put a Luna answer under a cent and an Astra answer at about 85 cents, with more than 90% of Astra's cost on the input side. That is our arithmetic on OpenAI's published rates, before caching, and it assumes both models produce the same number of tokens. In production they don't.

Reasoning tokens bill as output, so Luna at none and Astra at low will return different output totals for the same question. Pulling the other way, OpenAI says Astra often finishes tasks with substantially fewer output tokens than earlier models, so its cost per task can come in below what its per-token price implies (OpenAI). Which effect wins on your traffic is something you learn from the bill, not the pricing page.

That is the job SpendIQ was built for. We built it for ourselves first: AI is Rezolve.ai's main cost of goods, and our own model bill used to arrive as a single number nobody could attribute. SpendIQ breaks AI spend down by provider, then by individual model, then by token class: input, cached input, cache reads, cache writes and output. It shows the effective price per million tokens for each class. When the bill moves, it compares the last 30 days with the 30 before and attributes the change to rate, volume or mix, down to the model and token class responsible.

Applied to a routed RAG system, that turns the usual failure modes into lines you can read:

• Routing drift. Every five points of traffic that slide from Luna to Astra add about $4,200 a month at 100,000 questions, on the assumptions above. SpendIQ shows that as mix rather than volume, which is the difference between "usage grew" and "the router changed".
• Reasoning creep. Luna defaults to medium effort, and the scaffolding passes from the previous section add output too. Both land in the output token class while question volume stays flat, so you can weigh what they cost against the accuracy they bought on your eval set.
• Cache leakage. Cached input bills at a tenth of fresh input on both models, so the static part of your prompt (system instructions, policies, schemas) should be showing up as cache reads. If it isn't, the cache columns say so.
• The 272K cliff. A retrieval change that pushes some prompts past 272K tokens doubles their input rate. That registers as a rate change on an unchanged model, which is exactly the movement the attribution view exists to explain.

Spend can also be attributed by API key, project and user, so each route can run under its own key and carry its own line. Alerts flag a daily spike against the trailing average before the invoice arrives, not after.

The evaluation set tells you where the line should go. The bill tells you whether production is respecting it.

This is how we think about model selection inside Rezolve.ai. Sidekick answers employees from their organization's own knowledge, grounded and cited, with specialized agents reasoning over one shared conversation rather than a single monolithic prompt. Matching the model, and the amount of deliberation, to the question is how you get answers that are fast when they can be and careful when they must be. Metering what each of those choices costs is how you keep it that way.

If you want to see governed, grounded AI answering in action on your own knowledge base, book a demo and see Rezolve.ai in action.