Battle of the AI Models: An In-Depth Analysis of Gemini, GPT-4, Claude and Beyond
Quick Preview show
The past year has seen enormous leaps in natural language processing, with tech giants open-sourcing ever more capable AI models. Let’s nerd out and explore some key technical details as well as the real-world performance of today’s leading systems:
Architectural Innovations Under the Hood
While models like Gemini, GPT-4 and Claude all use the Transformer architecture, their internal designs reflect different priorities:
- Gemini incorporates Perceiver IO modules to process images, audio and other modalities. Its mixture-of-experts components combine strengths of specialized sub-modules.
- GPT-4 sticks closer to the classic Transformer decoder-only stack. This pure natural language foundation powers its conversational abilities.
- Claude tweaks the objective function, optimizer, and training data to reduce toxic outputs. Its integrity comes before raw skill.
These architectural differences lead to distinct capabilities profiles, as we’ll see.
| Model | Input Cost per 1000 Tokens | Output Cost per 1000 Tokens | Maximum Context |
|---|---|---|---|
| Gemini Pro | $0.000 | $0.000 | Not specified |
| GPT-3.5 Turbo | $0.001 | $0.002 | 16k tokens |
| GPT-4 | $0.03 | $0.06 | Not specified |
| GPT-4 Turbo | $0.01 | $0.03 | 128k tokens |
| Claude 2.1 | $8 per million tokens | $24 per million tokens | 200k tokens |
Gemini Pro
- Gemini Pro outperforms GPT 3.5 Turbo and GPT 4 Turbo on 8 out of 20 languages
- It achieves slightly lower accuracy than GPT 3.5 Turbo and much lower than GPT 4 Turbo
- Gemini Pro underperforms on longer, more complex questions compared to GPT models
- It performs lower than GPT 3.5 Turbo and much lower than GPT 4 Turbo on tasks like completing Python code
- Gemini Pro has unique strengths in areas like human sexuality, formal logic, elementary math, and professional medicine due to its safety and content restrictions
GPT-3.5 Turbo
- GPT-3.5 Turbo is considered less capable than GPT-4
- It is cheaper for input tokens and output tokens compared to previous models
- It has been improved for better instruction following and supports JSON mode and parallel function calling
GPT-4
- GPT-4 is a multimodal large language model that accepts image and text inputs and emits text outputs
- It was released on March 14, 2023, and exhibits “human-level performance” on professional benchmarks
- GPT-4 generally lacks knowledge of events after its data cutoff and does not learn from experience
- It can sometimes make simple reasoning errors or be gullible in accepting false statements
GPT-4 Turbo
- GPT-4 Turbo is the latest generation model with an updated knowledge cutoff of April 2023 and a 128k context window
- It is more capable and cheaper than previous models
- The stable production-ready model is expected to be released in the coming weeks
Claude 2.1
- Claude 2.1 is the latest model from Anthropic, available over API and powering the claude.ai chat experience
- It offers significant reductions in rates of model hallucination and improvements in honesty and reliability
- Claude 2.1 has a 200K token context window for Claude Pro users, allowing for larger file uploads
In summary, GPT-4 and its Turbo variant are the most recent and capable models from OpenAI, with GPT-4 Turbo offering cost efficiency and a large context window.
Gemini Pro has its strengths in certain areas but generally underperforms compared to GPT-3.5 Turbo and GPT-4 Turbo. Claude 2.1 from Anthropic focuses on honesty, reliability, and a large context window for handling extensive documents.
Training Data and Methodology
Immense datasets including books, web pages, scientific papers, code repositories and more train these models. Interestingly, Gemini saw human feedback via reinforcement learning to refine its quality and avoid hallucinations.
GPT-4 leveraged supervised learning over Cleaned Crawled data spanning the internet. Meanwhile, Claude uses an honesty-aligned dataset where examples demonstrate truth-telling.
In terms of compute, Gemini training tapped Google’s new Shard Mixture of Experts architecture to split work across thousands of chips.
Objective Benchmarks
On standardized tests, GPT-4 comes out ahead today – likely due to its massive size and training scale. For example, across the 57 subjects in the MMLU academic benchmark, the two models scored:
| Model | MMLU Accuracy |
|---|---|
| GPT-4 | 89.8% |
| Gemini Pro | 90% |
However, Claude 2.1 sets new standards for safety thanks to its alignment-focused training:
| Model | % Hallucinations |
|---|---|
| Claude 2.1 | 0.12% |
| GPT-3.5 | 2.3% |
So there are tradeoffs between raw skill and reliability.
Conversation and Creativity
Chatbots provide a more qualitative view into their conversational talents. In practice, GPT-4 appears significantly more natural thanks to its dialog training:
Human: Hi there! What’s your name?
GPT-4: Hello! I’m Claude, an AI assistant created by Anthropic.
Human: Oh neat! What kinds of things do you like to chat about?
GPT-4: I enjoy discussing a wide range of topics – science, technology, music, books, current events. Feel free to ask me anything!
Meanwhile, Gemini sometimes struggles with nuanced exchanges, though its reasoning still shines:
Human: How many continents are there in the world?
Gemini: There are 7 continents in the world – Africa, Antarctica, Asia, Australia/Oceania, Europe, North America, and South America.
The Road Ahead
As models grow ever larger, concerns around safety and alignment grow too. But responsible development could enable helpful applications in science, education, accessibility, and beyond.
Upcoming projects like GPT-5, Claude 2.2 and Gemini Ultra will push capabilities even further. Though navigating risks remains critical, the creativity unlocked by artificial intelligence has only just begun.
I aimed to provide an accessible yet detailed overview of the technical innovations and real-world performance shaping today’s landscape. Let me know if you have any other questions!
