Comparing Six AI Assistants
Say you need to make an important call at 2 p.m., but a work meeting you can't move is booked for the same hour. The meeting is on your calendar, while the call was arranged over email, so it's hard to notice they overlap. Would AI assistants catch it for you?
The American tech publication WIRED staged this situation on purpose to see whether six AI assistants would notice. Only the paid OpenAI Dots and Claude Duet spotted the overlap.
One difference, though. Dots went further and suggested calling before the meeting or after 3 p.m., while Claude Duet only flagged the conflict. Meta Muse, which is basically free, didn't notice the problem at all.
✦ The Five Tasks WIRED Set
One day, six agents, one iPhone
This comparison is a hands-on review by WIRED reporter Reece Rogers, who set up six agents on an iPhone and gave them five tasks over a single day.
The five tasks were: coming up with news ideas in the morning, putting together a lunch order, finding a schedule conflict between email and calendar, planning a two-week trip to China, and answering a late-night worry.
✦ Total Scores Out of 25
My sum of the published per-task scores
Each task was scored out of 5. Adding up the published scores gives Dots 20 points, Muse 18, Claude Duet and Grok Bot 16 each, and Gemini Spark and Instinct 12 each.
Of course, the agents didn't all work the same way. Instinct took its tasks by text message, Dots ran inside ChatGPT, and Claude Duet ran in a dedicated app.
So you can't say the score gaps are 100% down to the models themselves, and the scoring is pretty subjective too. Still, it's an interesting way to see how far AI assistants have come.
So Did Dots Do the Best Job?
✦ Highest Score on Each Task
Dots wasn't first everywhere
As mentioned earlier, Dots was best at keeping schedule conflicts straight. But across all five tasks, Dots didn't do everything well.
Take 'news ideas.' Going by the final output, Gemini Spark got the top rating with 5 points, while Dots managed only 2.
But results aren't everything. When the reporter asked for 'thirty minutes of deep research, then a plan,' only Muse and Dots actually tried to put in that 30 minutes. The rest gave ideas right away or sent something back within a few minutes.
It's true Spark scored highest on the final ideas. But it didn't follow the process it was told to follow.
If you care about process more than results, you might have given Muse or Dots better marks than Spark.
On the 'China trip' plan, Muse took 5 points. The reporter asked for a two-week trip for two people, with no more than four hotels, mixing historic sites, big cities, food and nightlife.
Muse laid out tables and practical tips neatly, but Dots' plan ran to 24 pages. A good travel plan isn't just about gathering a lot of information. Trimming the options down to a sensible length matters too, right?
In that sense, Muse did the better job. Then again, some people might think 'A 24-page travel plan? Cool!' and score Dots higher.
And on the late-night worries task, Grok Bot reportedly got the top score of 5.
The 'lunch order' task came down to how the menu was put together. The reporter asked, 'Get me meat and coffee, and pick a proper Vietnamese lunch for my vegan boss.'
✦ Lunch Order: Dots vs. Gemini Spark
Same request, very different menus
Dots built a proper menu, but Gemini Spark recommended just a single $5.85 tofu spring roll for the boss.
So the reporter would be having a real lunch while the boss sat across the table picking at one tofu spring roll. That's why the reporter gave Dots 5 points and Gemini Spark 1.
Are Paid AI Assistants Worth the Money?
Of the six AI assistants compared, only Meta Muse and Instinct are realistically free to use.
✦ Price and Access at Test Time
A snapshot from when WIRED tested
Meta Muse limits sign-ups by region, but it gives you 100 million tokens free each week. Instinct works if an existing user gives you an invite code.
Dots, Claude Duet, Grok Bot and Gemini Spark each required a subscription plan of $20 to $100 a month back when WIRED was testing them.
Personally, going just by WIRED's results, I think the free Muse is the best of the bunch at 18 points. It wasn't far behind paid Dots at 20, and it actually scored higher than the other paid services.
But that's for light personal use with nothing sensitive involved.
For example, let's look at what happened when a different reporter handed real orders to AI assistants.
Business Insider gave Muse, Instinct and Dots the jobs of sorting email, ordering lunch and building a personal website. The lunch order wasn't smooth on any of the three.
✦ Where the Real Lunch Orders Got Stuck
A separate experiment by Business Insider
Dots got stuck on the login code that arrived by text and on typing into the browser, but the order still went through. With Muse, the reporter didn't give the final confirmation, so the order never got sent.
Instinct took a lot of texting back and forth to place the order, and it never asked about adding a tip.
Getting through several steps in a row, from login to payment to final approval, without a slip is still hard. And if an AI assistant makes a mistake there, you're in for trouble.
In conclusion, I think Dots is the most reliable AI assistant right now. It costs money, but if you'd use an assistant for work, or run into frequent schedule conflicts, Dots seems like the safest choice.
Of course, Muse, Gemini Spark and Grok Bot scored better at idea suggestions, light trip planning and personal late-night advice, so no AI assistant is clearly better across the board yet.
So if you've never used an AI assistant and don't plan to give it sensitive work, why not start by trying the free Muse first?
That's all for this post. See you next time.
Related posts
- We Made AIs Vote, and They Started Copying Each Other
- GPT-6 Astra and Claude Fable 5.1 Failed a Paper Review
- ChatGPT Ads: Do They Change the Answers You Get?
🌐 Korean version: 무료 AI비서보다 유료 AI비서가 일을 더 잘 할까? 비교실험 결과를 살펴보자
No comments:
No comments yet. Want to leave the first one?
Post a Comment