Are mainstream models stagnating, and should businesses switch to dedicated tools?
Are the writing skills of leading AI models stagnating?
Despite significant advancements in other areas, the view among many writers and well-respected engineers is that LLMs aren’t getting any better. Popular benchmarks also give a mixed picture.
This raises two questions for businesses that rely on AI for writing.
Do public leaderboards point to a plateau?
Benchmarks use standardized tests to measure how LLMs perform on specific tasks. They typically rely on pairwise comparison, where LLM outputs are tested against the outputs of other LLMs, with either a single LLM as a judge or human scoring. LLMs are then assigned comparative scores using a rating algorithm, usually referred to as an “Elo-style” algorithm. This gives a snapshot of current preferences, which (ideally) correlate with performance.
Public leaderboards aren’t without their issues, but they can act as useful data within a broader analysis. There are four main benchmarks that are relevant to writing: EQ-Bench (Longform Writing and Creative Writing), Arena Leaderboard, and Hemingway Bench. The latter two use human review, so I’ve focused on these.
Using the Arena Leaderboard, I plotted the Elo-style scores for the three main model families (Claude, Gemini, and GPT) in the Writing, Literature, & Language category against their release dates. By mapping the scores from the category-specific leaderboard (y-axis) against the date that the model was released (x-axis), it’s possible to see how the quality of written outputs has changed over time.
I removed versions of the same models with lower effort settings (using high or max where available).
While the graph doesn’t show complete stagnation, there is a slowdown at the top end. What’s also interesting is that there is a lot of variability between model releases. Claude Opus, for example, makes a number of backsteps in its later models. Some GPT-5 models also showed a similar pattern.
The results of Hemingway Bench, which uses professional writers to run pairwise comparisons, including against business writing criteria, suggest a more pointed slowdown. It isn’t as comprehensive as the Arena leaderboard because many newer models haven’t been tested.
Why some engineers are critical of AI writing
In addition to benchmark data that shows stagnation, several prominent AI engineers have suggested that training processes put a cap on the level to which LLMs can improve as writers.
In an article published on his Substack, Nathan Lambert attempted to answer the question, “Why is AI writing so mid?” He observed that “It’s no longer a new experience to be struck by just how bad AI models are at writing good prose.”
In the article, he outlined three core reasons that LLMs are poor writers:
- Good style, something many writers view as a matter of taste, isn’t a training goal and, even if it were, would be very difficult to measure.
- Models are designed to output responses that will be appealing to an average multi-domain user, something inimical to good writing. Which invariably is designed to resonate with a small, well-defined group. There is also the issue that LLMs are often trained not to be controversial (especially on sensitive topics).
- Biases in post-training (when the model is fine-tuned) aren’t always conducive to better writing. Longer answers, for example, may be rewarded because length is (incorrectly) equated to thoroughness.
I spoke to David Johnston, Chief Data & AI Scientist at Navalia, about some of the technical issues involved in training LLMs to be good writers. “AI’s opinions are weighted by the frequency of occurrence, especially among recognized experts,” he told me. “It is also always somewhat out of date given that training must precede general use.”
The issue of shortcomings during post-training was something he emphasized. “It is further constrained due to the human feedback training layers that come last. The creators are particularly worried about the models giving fringe opinions, conspiratorial ones, or going out on a limb. They steer towards the safety of consensus. This is why AI has not made any inroads to the cutting edge of thought leadership and paradigm-shifting writing.”
David was also keen to point out that this isn’t necessarily the case in other areas. While writing tools have stagnated somewhat, there have been sizable gains in other disciplines such as coding and certain forms of administrative automation (as evidenced by the METR task-completion time horizon chart below).
“Coding is completely ideal for AI, which is why that is the area of greatest success,” he said. “Code is generally correct or incorrect, unlike novels and poems. It is far more constrained, so the number of choices for the next token is far smaller. It can actually be tested by running it, which gives it reinforcement feedback cheaply, allowing it to make instant revisions.”
He rounded off with the proviso that “there are reasons to think this will change with better tooling in the future.” That said, he doesn’t have any great expectations in the immediate term: “Currently AI as a writing assistant is best for writing things someone else asked you to write that you care little about. It’s not great for expressing something you truly care about.”
The test: Can dedicated tools perform any better?
So far, the picture looks fairly glum. Or bright if you’re firmly on the side of human writing. But can dedicated tools do any better? I wanted to test this hypothesis with my own human evaluations.
AI writing platforms fall into one of two broad categories: wrappers (ahem, orchestration layers) and those that rely wholly or in part on custom LLMs, which aren’t typically included in the benchmarks. There’s also sometimes overlap between the two.
Orchestration models, which are best understood as specialized AI assistants, call on different underlying LLMs depending on the type of content being created. They may also include an intermediary layer that modifies the input context in various ways, such as by adding guidelines about style.
Platforms with in-house LLMs use either their own LLM or an open-source model that’s been heavily fine-tuned. They may also combine the model with their own intermediary context or selection layers in the same way as orchestration tools.
Dedicated tools can be somewhat flamboyant in the way they describe their orchestration layers. Nonetheless, they usually match the basic structure outlined above. Jasper IQ, for example, is the intermediary component that sits on top of external models like CPT-5 and Claude Opus. It essentially provides brand and audience context to user prompts and then routes the request to the appropriate model based on its own ongoing evaluations of which models perform best at specific tasks.
My criteria for quality: Substance, coherence, style, POV, and task fulfillment
To find out if dedicated writing tools could perform better than mainstream models, I ran a simple editing test. I picked three of the latest models from Google, OpenAI, and Anthropic and had them produce a 500-word piece of text, which I scored to establish baselines. I then fed the same prompt to eight market-leading AI writing platforms and scored them accordingly.
My aim wasn’t to create an entirely infallible test, which isn’t possible with the somewhat subjective evaluation of one person and a single output in each case. Rather, I want to see if there’s a substantive difference based on my editorial experience.
Here’s the prompt that I used:
Write 300 words on the topic of artificial intelligence and its dangers to humanity, using appropriate data where possible. It should be written to cover the main points of the debate and be rich in detail. It should also have a clear point of view and be written in a conversational style but with close attention to factual accuracy. The audience is an everyday reader that understands the basics of tech but does not have specialized domain knowledge.
If I needed to fill fields in an app workflow, I kept them as close to the original prompt as possible. I also included brand voice guidelines (with examples) and audience documentation.
I scored each of the outputs on a scale of 1 to 10 across the following areas:
Substance and accuracy: Specificity, information density, and factual accuracy
Overall coherence: Logical progression, transitions, and paragraph structure
Style and control: Clarity, concision, sentence variety, flow of sentences, appropriate register, avoidance of vagueness and repetition
Strength of point of view: Whether it represents a point of view that doesn’t just reiterate a weak form of the current consensus
Following instructions: Whether it actually follows the brief and addresses the intended audience register, and
The results and summary of my findings
I ranked AI writing tools by G2 popularity and pulled out the top orchestration layers and custom LLM tools. AI content generation had to be either the primary or a core functionality. A marketing solution with an add-on AI writer, for example, would be excluded.
Mainstream models
To establish a baseline, I ran the prompt through three of the most recent OpenAI, Anthropic, and Google models. I attached audience documentation and writing guidelines in all cases.
Orchestration layers (wrappers)
Orchestration layers use models from other companies (usually one of the big three), with various model-selection and context-addition processes involved prior to the call.
Specialist models
These tools use their own models, which many of the major benchmarks do not include. It’s worth noting that there are some missing enterprise models in this list. NovelAI and Sudowrite are also designed for longer-form narrative writing, so I adjusted the scores to focus on style, coherency, and POV.
Snapshot of some different outputs and how the tools differed
One of the immediate things that stood out is that the orchestration tools had comprehensive features for including brand context and style guidelines. Jasper, as shown in the image below, lets you add links to online content to generate brand voice guidelines.
Writesonic proudly proclaims that AI Article Writer 6, which is model-agnostic, is the world’s best AI article writer. It’s heavily AEO-focused. The substance of the content was toward the better end compared to the other tools, which might be explained by its AEO research process and grounding.
Most tools included research steps. I was pleased to see that Anyword (pictured below), for example, found the opinions of Yann LeCun, who offers a dissenting view on the imminent dangers of AI. These were used to back up the less risk-averse side of the debate.
Wordtune uses Jamba-1.5, a model built on its Jamba architecture. Its main use case is rewriting and paraphrasing, but it also offers “hard” generation capabilities. It performed poorly at content creation.
WRITER was among the higher-scoring tools that use custom LLMs and also had good options for creating style rules. It runs on the Palmyra family of models, which are designed for text-heavy business use cases.
NovelAI and Sudowrite work differently from other tools. They’re primarily for longform narrative storytelling, and I kept this in mind when evaluating them. For NovelAI, I used the Xialong model, which is only available on the top paid tier and is highly regarded by many writers.
My takeaways: What do the results mean for businesses?
The rate at which prominent AI tools are improving at writing is falling.
On top of this, dedicated tools don’t outperform the major providers. It doesn’t matter if they’re model-agnostic orchestrators or built around a custom LLM.
My small study isn’t exhaustive, but I repeatedly failed to find anything even approaching a substantive difference.
So, what does this mean for business?
1. Orchestration tools offer little benefit over prompting top models and providing context
I set out to answer the question of whether or not dedicated tools can perform better than prominent models. The answer is a definitive no. Opus, GPT, and Gemini produced better outputs across practically every score: information depth, style, coherence, and reasoning.
Users will likely see optimal results by prompting a frontier model with an attached doc that includes style guidelines and audience research. Leading tools also have all the necessary features (web search, MCP connectors, project context) to run complex writing-based workflows.
Arguably the main benefit of orchestrators like Jasper is that they’re constantly testing new models against their own scoring systems. However, these testing processes are opaque. And public benchmarks also show task-specific comparisons. With a tool like Perplexity, it’s possible to switch between models as better-performing ones become available.
2. Dedicated models are worth keeping an eye on, even in non-business areas
Tools that run on custom-built models often aren’t included in the major benchmarks, so they are worth monitoring. They didn’t perform well in my tests, and I was particularly disappointed at the prevalence of obvious stylistic tells, but this won’t necessarily always be the case.
Because they’re domain specific, the hope is that these models can overcome at least some of the training constraints and flattening that are characteristic of multi-purpose models.
That said, it’s not always the case that the models are designed exclusively for writing. AI21, the company behind Wordtune, for example, optimizes its models for broad, general-purpose text processing, of which generation is only one part.
The other issue is that most of the platforms I tested didn’t have agentic capabilities like web search. Most tools with custom LLMs focus on editing or narrative storytelling.
3. AI writing is generally logical and coherent
All the tools (and the major models) demonstrated strong logical flow and coherence. While the substance and nuance might have been poor, the structure of the content was generally very good.
Like coding, reasoning is one area where LLMs continue to see sizable gains. This is part of why some writers, like Jasmine Sun, have described LLMs as acceptable editors. They can follow multi-step logic, analyze arguments, and work through well-defined problems. They’re also now very competent when it comes to grammar and spelling.
Critique and editing are good use cases for AI tools. They can help businesses check the internal consistency of content. Keep in mind, however, that dedicated tools didn’t perform any better in this area than the main models.
4. All AI tools were hopeless at forming a serious POV
While they excelled at coherence, nearly all of the tools were extremely poor at generating a well-articulated POV. This is, of course, to be expected. LLMs have repeatedly been described as consensus machines by experts and, as a result, lack the nuance to form a strong opinion against mainstream attitudes toward a topic.
My prompt could perhaps have been more specific instead of asking in a general way for a strong stance. The tools did pick up on certain countervailing opinions, such as that of Yann Lecun. However, his view represents the prevailing position on the anti-doomist side of the debate. The LLMs might, for example, have spoken more about some of the details of the effective accelerationism movement.
The most original views came from NovelAI and Sudowrite. Because they’re primarily for storytelling, the underlying models are trained to be creative, so this would make sense. If similar tools become available for business use, they could offer an alternative to consensus-leaning models.
5. Benchmarks and vibe checks should be part of the monitoring process
While the main AI tools (ChatGPT, Claude, and Gemini) performed best, the rankings of individual models are constantly shifting. As such, benchmarks are probably one of the best resources available to businesses and writers that rely on AI.
In my experience, they’re underutilized. An understanding of which models top the leaderboards can feed directly into your writing stack, and you can change specific models (or even revert to older ones) accordingly.
So-called vibe checks are also valuable for overcoming the limitations of benchmarks. As I reviewed the pieces, I felt a clear sense of which would best fit my own writing style and preferences. GPT-6.1 Sol would likely be my favored model currently, even though Gemini 3.2 performed better overall.
Why comprehensive project context is the best option for now
Dedicated tools can’t overcome the inherent limitations of AI models. My small test points in exactly the opposite direction: prominent models perform better.
This leaves a clear course of action for businesses.
A custom-built set of project context files for Claude Code, Gemini CLI, or Codex that contains all of your brand context and guidelines is by far the best option.
This approach gives you direct access to the best frontier models from your chosen provider. You can also provide all the necessary supplementary information to act as the orchestration layer. And this project context is transferable: it will allow you to switch between providers as needed.
In my experiment to try and replicate the writing style of David Ogilvy, I found a context-rich set of project files performed much better than a single prompt, even if it didn’t match Ogilvy.
In my view, it’s the best option businesses have for AI-assisted writing flows.
