I Tested 71 AI Model Configurations On My News Pipeline. The Best Isn’t In Production

🔍 Read the full analysis: I Tested 71 AI Model Configurations On My News Pipeline. The Best Isn’t In Production on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get movie nights delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI model configurations for his news pipeline found GPT-6 Astra produced the highest-scoring articles, but GPT-6 Luna was the model he chose to run in production because it cost about one-hundredth as much. The results, published October 13, also led him to remove two backup models after tests found fabricated quotes in one model’s output and unusable articles from another.

Meyer ran the evaluation from September 8 to October 13, 2026, using his production prompt to turn real news sources into English- and German-language articles. The sources were a 6,340-character product filing and a 619-character political news item. He says the shorter item better reflected the thin-source material common in his workflow. Each configuration wrote for both editions, and two AI judges scored the articles on a five-point scale for source fidelity, fabrication, clarity about evidence, news value, structure and polish, with German-language quality also assessed.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a 5.00 for source fidelity. In a further run covering 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reports GPT-6 Luna scored 3.67 in that robustness run, with source fidelity of 4.71 and valid output for every article. He estimated Astra at about $0.12 per article, compared with roughly $0.0012 for Luna; reported generation times were about 50 seconds and 17 seconds, respectively.

The test also found that increased reasoning effort did not produce a clear quality gain for Luna. Across six effort settings, the scores varied, but Meyer said the differences were within the test’s noise; the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort production setting. He also reported that GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a 1.00 source-fidelity score, while local gpt-oss-120b produced no valid articles in four attempts. Meyer says both were removed from the route on October 7.

The Cost of Reliable News Output

The findings highlight a practical choice for publishers using AI: the highest-scoring model may not be affordable for routine, high-volume work. Meyer’s reported cost estimates put Astra at about 100 times Luna’s per-article cost, while the additional quality shown in the tests may not justify that spend for every article. His decision to use Luna reflects a production threshold—consistent, usable output at a low price—rather than a claim that it is the strongest writer overall.

The fallback results point to a separate operational risk. A pipeline can depend on backup models when its main service fails, so poor output from those models may reach publication unless it is caught. In Meyer’s tests, one backup produced invented quotations and another failed the pipeline’s validity requirement. Those results are specific to his setup and test prompts, but they illustrate why model selection needs to include failover behavior, source fidelity and format compliance, not only a headline score.

How Meyer Built the Test

Meyer’s article argues that broad leaderboards do not answer which model should write a particular publisher’s articles. His evaluation instead used the same prompt and repair step as his production pipeline, with two source examples and two language editions. The models included API and subscription products, free-tier options accessed through OpenRouter and open-weight models run on his own hardware.

Two AI judges—GPT-6.1 Sol, set to high reasoning, and Claude Opus 5.5—saw the original source and graded the outputs. Meyer also ran a robustness round in which leading configurations wrote 10 additional articles, mostly from short sources. He says that because the initial test involved only two articles per configuration, he treated score gaps below roughly 0.3 points as noise and relied more on the larger follow-up run for ranking. That approach provides useful detail about the test, but the results remain an evaluation by one publisher on a limited set of sources.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in the October 13 report

Limits of the Model Comparison

The results come from one publisher’s workflow, using two initial source texts and a follow-up set of 10 articles for leading configurations. Meyer reports two AI judges, but the article does not describe an independent human editorial assessment or a blind evaluation. It is also unclear from the provided results how the judge scores were combined, whether models were tested under identical sampling settings, or how performance varies across a wider range of news topics and source formats.

The cost and timing figures are Meyer’s estimates for his setup and should not be read as universal prices or response times. The report presents the outcome as a decision for his pipeline; it does not establish that Luna will be the best choice for other publishers. The supplied account also ends while describing the updated routing setup, so the precise fallback arrangement after the October 7 changes is not fully specified.

Production Use After the Test

Meyer says GPT-6 Luna remains in production at the no-reasoning setting, which he reported costing about $0.0013 per article over the previous week, with a median generation time of 17 seconds. He had decided on October 7 not to use Astra for routine news because of its higher cost, and removed GPT-4.1 nano and gpt-oss-120b from the news route after their test failures.

The report does not announce a new independent benchmark or a planned expansion of the test. For now, the next evidence will come from Luna’s ongoing performance in Meyer’s workflow and any future tests across more sources or model versions. The report does not say whether he will add replacement fallbacks or publish further production metrics.

Key Questions

Which model scored highest in Meyer’s news-writing test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test, with a 5.00 source-fidelity score. Its average across the 12-article robustness run was 3.93.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and produced valid output for every article. He estimated it cost about $0.0012 per article, compared with about $0.12 for Astra.

Did higher reasoning effort improve Luna’s articles?

Meyer tested six reasoning-effort settings and said the score differences were within the test’s noise. The maximum setting took and cost about 12 times as much as the no-effort setting, without a clear quality gain in his results.

What problems did the removed fallback models have?

Meyer reports that GPT-4.1 nano produced invented quotations and received a 1.00 source-fidelity score. He says local gpt-oss-120b produced no valid articles in four attempts, including output-format problems and invented named quotations.

Do these results establish the best AI writer for every newsroom?

No. The comparison tested Meyer’s prompt, pipeline and selected sources. Its results may help publishers design their own evaluations, but the report does not establish how the models would perform across other newsrooms, topics or editorial workflows.

Source: ThorstenMeyerAI.com