Study Finds ChatGPT’s Skills Deteriorate Over Time

ChatGPT was assessed on 4 tasks, solving math problems, addressing sensitive questions, etc

In a recent study published by academics from Stanford University and the University of California, Berkeley, the popular large language model (LLM) ChatGPT may grow poorer at coding. The study revealed that the model's performance on code development tasks had dropped considerably over the three months after testing the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on four tasks.

The research assessed ChatGPT on four tasks: solving math problems, responding to sensitive/dangerous questions, creating code, and visual reasoning. These tasks were chosen to exemplify the diverse and beneficial capabilities of ChatGPT-like LLMs. The researchers discovered that ChatGPT's performance on the code generation job decreased dramatically between March 2023 and June 2023.

According to the research, ChatGPT-4 performed well in recognizing prime numbers in March 2023, with an accuracy of 97.6%. Nevertheless, by June 2023, GPT-4's accuracy for the same had plummeted to 2.4%. GPT-3.5, on the other hand, showed a significant increase in its capacity to recognize prime numbers throughout the same period. Another fascinating discovery is the shift in behavior while responding to sensitive queries. Compared to March 2023, GPT-4 and GPT-3.5 were less inclined to answer sensitive requests in June 2023. Also, formatting errors in code creation increased significantly in June 2023 compared to March 2023 for GPT-4 and GPT-3.5. The study discovered that the codes were

While this study mirrors the sentiments of many ChatGPT users who believe ChatGPT's performance has deteriorated over time, OpenAI has refuted these accusations, claiming that each new version is better than the prior one. No, we haven't dumbed down GPT-4. On the contrary, we make each new edition smarter than before. Current hypothesis When you use it more frequently, you start finding difficulties you didn't notice previously, said OpenAI VP of Product Peter Welinder in a recent tweet.