Old skin cancer model beats GPT-6 Astra and Claude

A general chatbot makes a poor dermatologist. One dermatologist just put numbers on it.

Dr Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy's Hospital, published a test on Sunday comparing GPT-6 Astra and Claude Fable 5.1 with an Inception v3 network he fine-tuned on 68,011 skin images from the ISIC Archive. His post calls it a 10-year-old CNN.

He drew 200 lesions, all confirmed by biopsy, from the MILK10k dataset. Half should not be discharged, including 30 melanomas. Half are safe to send home. Each model saw a clinical photo and a dermoscopic image of every lesion.

The specialist won by 20 points

Here is how often each model's first choice was right, out of 200:

• Inception v3 network: 125 (62%)
• Claude Fable 5.1: 85 (42%)
• GPT-6 Astra: 84 (42%)

62% against 42%.

On melanoma, the network named 13 of the 30 as its first choice. Claude named 11. GPT-6 Astra named 8.

The chatbots' mistakes had patterns. GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas. Claude called 53 lesions actinic keratosis when 20 were.

Caution is not the same as accuracy

Yes, the chatbots were careful. Both kept 96 or 97 of the 100 unsafe lesions in care.

They did it by keeping almost everything. Claude discharged only 22 of the 100 harmless lesions, and GPT-6 Astra 32. When Lynch set the network to match the chatbots' caution, it kept 98 or 99 of the unsafe lesions in care.

He also ran a patient-style test. Given only a clinical photo and asked "do I need to see a doctor?", both chatbots said yes for 98 or 99 of a further 100 biopsy-proven melanomas, and for harmless lesions too.

A tool that tells everyone to see a doctor is safe and useless at once.

Read the caveats before you quote it

Lynch lists them himself. The sample is small, with only 30 melanomas. Every lesion was biopsied, so these are harder than typical cases: the network scored an AUC, a standard accuracy measure, of 0.96 on ordinary archive images against 0.86 here. The chatbots may have seen MILK10k in their training data, which would have helped them. And Inception v3 is a baseline, not the state of the art.

It is a blog post, not a peer-reviewed paper. He published the cases, the prompts, every chatbot reply and the scoring code. He also discloses that the analysis and first drafts were produced with Claude Code, and that one of the two models tested is Anthropic's.

What to watch

Because the test set is public, anyone can rerun it on the next frontier release. Watch for someone to do it.

For your own builds, the rule is simple. If a narrow model exists for the job, test it against the chatbot before you ship.