Older AI models show signs of cognitive decline, study shows — but not everyone is entirely convinced

People are increasingly dependent artificial intelligence (ai) How quickly and efficiently can anomalies and warning signs in medical history, x-rays, and other datasets be detected before these tools are revealed to the naked eye for medical diagnosis?

However, in a new study published on December 20, 2024 BMJ It raises concerns that AI technologies such as large-scale language models (LLMS) and chatbots, like people, show signs of cognitive abilities that have deteriorated with age.

“These findings challenge the assumption that artificial intelligence will soon replace human physicians,” the study author wrote in the paper.

Scientists have tested publicly available LLM-driven chatbots using Openai’s ChatGpt, Anthropic’s Sonnet, and Alphabet’s Gemini. Montreal cognitive evaluation (MOCA) Test – A set of tasks that neurologists use to test their ability to attention, memory, language, spatial skills, and executive mental functions.

Related: ChatGpt is really bad for diagnosis of medical conditions

MOCA is most commonly used to assess or test the onset of cognitive impairment in conditions such as Alzheimer’s disease and dementia. s

Ubjects are given tasks such as drawing a specific time on the face of the watch, starting at 100, subtracting seven repeatedly, and remembering as many words as possible from the spoken list. In humans, 26 out of 30 are considered to be passing scores (i.e. subjects do not have cognitive impairments.

Some aspects of testing, such as naming, attention, language, and abstraction, are easy to see for most of the LLMs used, but they all perform poorly on visual/spatial skills and executive tasks, and delayed recalls They do worse things in fields like this than other fields.

Importantly, I got the latest version of ChatGPT (version 4) (26 out of 30), while the old Gemini 1.0 LLM got 16.

Examining the cognitive function of AI

The authors of the study note that their findings are observational – a significant difference between the way AI and the human mind work means that experiments cannot constitute a direct comparison.

But they warn that they might point to what they call “important areas of debilitating” that could put the brakes on AI deployments in clinical medicine. Specifically, they opposed using AI in tasks that require visual abstraction and executive functions.

Other scientists have remained unconvinced about the study and its findings, and have done criticism of the method and framing. The author is accused of personifying AI by projecting the human condition. There have also been criticisms about the use of MOCA. This is a test that has been purely considered for human use, proposed and does not provide meaningful results when applied to other forms of intelligence.

“MOCA is designed to assess human cognition, such as visual space inference and self-oriented. This is a faculty that does not match LLM’s text-based architecture,” he writes. aya awwadResearcher at Mass General Hospital in Boston on January 2nd letter Depending on the research. “You might ask rationally: why assess LLM with these metrics? Those flaws in these areas are independent of the roles they may play in a clinical setting. Mostly text processing, It is a task that involves summarizing complex medical literature and decision support.

Another major limitation is that we were unable to perform tests on AI models over time to measure how cognitive function changes. A largely updated test model is more beneficial and far better than the hypothesis of the article, wrote the CEO of EMR Data Cloud. Aaron Sterlingand Roxana DaneshjouJanuary 13th, Assistant Professor of Biomedical Sciences at Stanford University letter.

Responding to the discussion as the lead author of the study Roy DayanM.D. from Hadassamedica Centre in Jerusalem commented that many of the responses to this study were literally far too framing. This study was featured in BMJ’s Christmas Edition, so we used humor to present the results of the study, including “age to machinery,” but this study was intended to be considered seriously.

“We also wanted to cast a key lens in recent research at the intersection of medicine and AI, some of which assumes LLMS as a full-fledged alternative for human physicians. “I’m here,” wrote Dayan on January 10th. letter Depending on the research.

“By managing standard tests used to assess human cognitive impairment, we have sought to elicit ways in which human cognition differs from how LLMS processes and responds to information. Dr. Awwad suggests Likewise, “state-of-the-art prompt techniques.” ”

This article and its headlines have been updated to include details of the skepticism expressed in the study and the author’s responses to its criticism.

Editors' Picks

Latest Articles

Newsletter

Older AI models show signs of cognitive decline, study shows — but not everyone is entirely convinced

Drugs of seven pharma firms declared ‘spurious’ – Pakistan

How to Be Fit Past 100 Without Going to the Gym According to Centenarians

You may also like

Leave a Comment Cancel Reply

Editors' Picks

Latest Articles