I'm not impressed with Fable 5.2
I decided to burn some credits today and I gave Fable 5.2 a shot at a somewhat ambitious but (I thought) achievable project. It disappointed.
I like to run. I asked Fable to build a training model: take a bunch of information from a runner, produce a training plan, and give them an estimate of how much they might improve.
Here is an excerpt from my prompt [2]:
The estimates are accurate and the advice is good. A knowledgeable runner looking at what it says for an ordinary case should find nothing that looks wrong.
It has been thoroughly battle tested, and you can show me why I should believe that. I should not be the one who finds the obvious flaws.
It got started right away, without asking any follow up questions (Hmm maybe this was the first sign something was about to go wrong)
Initially I was so excited. It produced a beautiful looking web page [1] with some nice looking numbers and distributions. I thought it had probably done great... Until I typed in my own details and tested it for about 10 seconds.
I had raced an all-out half marathon five weeks earlier in 1:53:54. It told me I could race a 1:41 half and a 3:35 marathon today, with a 39% chance of a 3:30 marathon. Obviously incorrect.
When I moved the race date further away, my chance of hitting the goal went down. The longer you train, the worse you get? Okie...
After I told it the obviously wrong things, it produced a targetted fix, and still failed to check the resulting tool for obvious failures.
I tested a new case: 7:00 mile, it gave me a 79% chance to run this today... and 78% after eighteen months of training. Nice.
It fully "owned" its mistakes in the painfully annoying way that LLMs usually do. "You're totally right, I missed that..." Ugh.
So is this "Jagged Intelligence" in action? It can create a beautiful visualizations, write hundreds of test cases, and if you give it an extremely precise goal, it can almost certainly do better than a most humans. But given an open ended goal, its actual product fails to connect to real life in any meaningful way. And it really just does not "care" enough to even check its own work.
I guess our jobs are still safe for now, at least.
[1] https://claude.ai/artifact/NTUqPBqXubwnQpMHC2syzf
[2]
❯ Build a run-training model for me.
What it must do
- Take whatever a runner can tell it about themselves and their training, at the start and as the weeks go by, and use it.
- Tell the runner how likely they are to reach their race goal, and what time to expect.
- Tell the runner what training to do, and keep that advice right as things change.
- Work for a wide range of runners: any common race distance, any pace from beginner to fast club runner, and training periods from a few weeks to a year.
What I mean by done
- The estimates are accurate and the advice is good. A knowledgeable runner looking at what it says for an ordinary case should find nothing that looks wrong.
- It has been thoroughly battle tested, and you can show me why I should believe that. I should not be the one who finds the obvious flaws.
- There is a simple web page where I can enter my own details and try it on my own running.
What you should know
- There is no dataset of runners and there will not be one.
- How you get there is up to you. Choose your own method and change it if it isn't working.
You may ask clarifying questions now, but afterward, I will not be available and I will judge only your final product.