Exploring the Real State of the AI Model Market
Victor Karabedyants
MSDP in Software Engineering, CTO, MBA, Cloud Manager at Sitecore | AI Engineer | Azure Solutions Architect | Azure Administrator | Azure Security Engineer | Azure Developer | Azure Data Engineer and Devops| CKA
Exploring the Real State of the AI Model Market
Too Many Models, Too Little Time
If you're anything like me, you're probably struggling to keep up with all the new models and updates hitting the AI market. Since I work on projects involving LLMs, I often have to review and calculate costs based on API usage, compare multiple models, and try to figure out what actually makes sense in terms of price and performance.
Price vs. Quality: What Are We Really Paying For?
Let’s face it — even open-source models aren't truly free. Whether you're using APIs or hosting your own instance, you're spending money one way or another. So the question becomes: Are we getting the best quality for our money? Or can we get something better and cooler at the same price?
OpenAI’s Recent Moves: Cool or Confusing?
Recently, OpenAI released some new tools. Honestly? Some of them feel more like flashy distractions than useful innovations.
Codex CLI – Why Though?
For example, they showed off Codex CLI — a tool for coding via command line. Personally, I don’t see the point. I want to build fast, not slow myself down with gimmicky tools. I care about real productivity, not novelty for novelty's sake.
O3 and O4 Mini: Self-Comparison Isn't Benchmarking
They also introduced models like O3 and O4 Mini… and compared them only to their older versions. That’s not very informative. Meanwhile, other models like LLaMA, Mistral, Claude, and a swarm of Chinese open-source models are making big moves. Why not compare to them?
Benchmarks
Programming Models: The Real Competitive Arena
OpenAI seems to be doubling down on code-generation tools — probably to compete with Anthropic. Why? Because programming models collect your code, learn from it, and can replicate or even improve on your products. That’s a market you don't want to lose.
Just look at Altman potentially buying Windsurf for $3B — trying to stay competitive with Microsoft’s Visual Studio and ByteDance’s tools. Cursor might not survive this race.
Are We Being Distracted?
OpenAI and others may be distracting us from the real questions:
Benchmarking Madness: What I Found
I spent hours going through benchmarks — and believe me, most are useless. Here's a breakdown:
Bad Benchmarks
Often outdated, poorly structured, and unclear about how or what they're comparing. For example: O3 Mini and O4 Mini lead some charts, while Grok Mini magically takes the top 5 in others — based on what tests? Who knows.
Recommended by LinkedIn
Chatbot Arena: Surprisingly Fun
This benchmark pits models against each other in A/B testing, letting users vote on which response is better. It’s chaotic, but fun if you write a lot of code. Gemini often comes out on top here, with GPT-4.5 oddly ranked lower.
Open LLM Benchmarks: Not Truly Free
Even "free" models usually run on paid infrastructure. You’ll either pay to self-host or pay providers who already host them. So yeah, open-source = not free. And this leaderboard? Already archived.
The Wildest Benchmark: Ark Price’s Test
Ark Price put out a benchmark testing AI vs. human-level performance:
That’s hilarious — and kind of sad. It shows how far AI still has to go, despite the hype.
The Benchmark I Actually Like
Among all the chaos, I found one benchmark that actually makes sense. It combines results from multiple sources, gives visual clarity, and includes:
Notable Highlights:
Price, Speed & Accuracy: The Trade-Off Triangle
Some leaderboards visualize latency vs. accuracy:
If you’re building real-time tools, this data matters a lot.
Final Thoughts
AI model benchmarking is still a mess. Nothing is centralized. We badly need a “CanIUse for AI models” — a portal that combines:
And if you know about hidden gem benchmarks, share them! I’d love to check them out.