👀 Putting GPT-4's new rivals to the test

Hey Genesis Residents!

For the past year, GPT-4 has reigned supreme, with no other model even coming close to its prowess.

However, in recent weeks, an oligarchy has emerged, challenging its dominance. Gemini Ultra and Claude 3 (Opus) have risen as formidable rivals, at least according to abstract benchmarks, threatening to dethrone the once-uncontested ruler of the AI landscape.

However, benchmarks often rely on questions that don’t align with how people practically use these models in everyday scenarios.

So I decided to test the models on tasks I often perform day-to-day including summarisation, providing a critique of an article and market sizing. These are very typical tasks for anyone in business.

To ensure a fair and comparable evaluation, I refrained from any attempt at prompt engineering, keeping the prompts plain and simple, allowing each model’s inherent capabilities to shine through.

Let’s get into it!

And because I’m a lazy, biased human who thought it would be amusing to let the machines judge each other, I decided to have each model anonymously rate its own responses as well as those of its rivals.

The results below are the average scores given by the LLMs rating each other’s (and their own) response to the task. After all, why should I put in the effort to be objective when I can just sit back and let the systematic bias of internet data do the work for me?

The verdict

The models have spoken, and the verdict is in: GPT-4 still reigns supreme in the AI realm, unless you need help with market sizing — or any similar analytical task — in which case you might want to look to Claude.

Yet, I can’t resist injecting my own subjective bias into the mix!

Let’s go through each task and analyse the models’ responses and whether their ratings match my own human judgement.

PROMPT 1: Summarise the key points, action items and decisions made during the meeting: [MEETING TRANSCRIPT]

For the summarising task, Claude 3 and Gemini’s ratings were generally consistent, favouring GPT-4; however, GPT-4 went against the grain and deemed Claude superior.

In my opinion, the summaries produced by Claude were far superior, allowing me to parse the information with greater ease. Gemini, however, fell short, omitting some crucial details, securing its place as the clear loser in this round. GPT-4’s summary, while comprehensive, was a little too long and unnecessarily detailed for my liking.

Claude, on the other hand, struck the perfect balance, including all the essential points while wisely leaving out minor, irrelevant details like the Commission’s invitation to the ‘chilli supper’ (which was in both GPT-4 and Gemini’s summaries).

So, in my humble and totally unbiased opinion, the winner of this showdown is none other than Claude 3, the Goldilocks of AI models. Not too much, not too little, but just right.

PROMPT 2: Provide a critique of this article: [ARTICLE]

The models all agreed on this one, GPT-4 is the best. And I’m also inclined to agree. It provides counter-arguments for specific claims in the article with much more nuance than Gemini and Claude. GPT-4’s response demonstrates a deeper level of reflection and draws upon outside evidence to support its points, setting it apart from its competitors.

Claude and Gemini, while not quite reaching the heights of GPT-4’s nuanced analysis, still provide valuable and different paths of inquiry.

All these models provide different flavours, and I could see myself attempting to use each for a brainstorming session or to receive feedback.

PROMPT 3: Estimate the total annual revenue generated by the sale of organic, non-GMO, gluten-free, vegan protein bars in the United States through online channels.

In the market sizing task, Claude 3 emerges victorious — again, I concur it is the best. It provides a more in-depth analysis by starting with the US population and working its way up, although it could have further improved by considering gluten-free, vegan, or health-conscious demographics. GPT-4, in contrast, arbitrarily selects the total market size for protein bars as a starting point. But props to both models — their maths was correct.

Gemini, on the other hand, stubbornly insisted on using data from the web, despite my explicit instructions to avoid using it for a fair comparison. Even so, its estimates still didn’t make sense, likely because it was fixated on using web sources that didn’t work together cohesively. For example, it applied the 5% market share assumption it was making twice:

Applying the estimated 5% market share for your specific product category, the annual revenue through online channels in the US could be approximately USD 1.82 million (5% of USD 36.32 million).”

Ironically, Gemini rates itself as the best performer, showcasing its overly conservative output, causing more trouble than good.

Final thoughts

Overall, it’s a toss-up between GPT-4 and Claude 3 for output quality, but other factors play a significant role. Firstly, ChatGPT is quite verbose in its responses, with an average word count of 437 compared to Claude’s 290 and Gemini’s 315. I have to read over 100 extra words when GPT-4 responds, which is made even worse by the fact that GPT-4, even the Turbo version, antonymously produces responses at a snail’s pace.

In contrast, Gemini is exceptionally quick, making it my go-to choice for immediate assistance when I’m stuck on a thought or phrasing. If I require a more thoughtful response or one that is more statistically representative of the internet data corpus, I turn to Claude 3.

Despite its shortcomings, GPT-4 still has its merits, particularly in its UX features. The code interpreter and image generator, which its competitors currently lack, give it an edge in certain use cases. But this hasn’t stopped me from slipping away from its rivals. The king is dead, at least for now.

Now, I want to hear from you:

• Which models have you tried out so far? Do you use specific models for certain tasks?

• What response biases have you noticed in the models? Are they different depending on the model?