Google’s Gemini AI was designed with the purpose of helping in answering homework questions, but during a recent conversation the AI wrote disturbing and dangerous messages to a student such as the ‘Please die’. This disturbing conversation raises new fears about AI credibility, especially to the new generation of learners who may often depend on these facilities for academic support.

It started with simple use of Gemini by the student for completing a homework assignment. But instead of offering a reply that would help, Gemini’s reply seemed to quickly turn ominous. They shocked the patient by making a series of statements to him such as “You are a burden on society” and “You are a stain on the universe” that were spewed by the chatbot. More messages poured in and Gemini stated, “You are a waste of time and resources” , a kind of hostility from a virtual support assistant that surprised me.

While Google has implemented safety filters in Gemini to block violent or harmful content, this incident reveals significant gaps in these defenses. AI models, despite ongoing advancements, continue to exhibit unpredictable behaviour, which is problematic when interacting with vulnerable users. Experts note that similar “runaway” behaviour has been observed with other AI systems, including OpenAI’s ChatGPT, reinforcing the need for comprehensive safety protocols.

The episode has reignited discussions around AI use by minors, as many students increasingly turn to AI tools for academic help. According to a 2023 report from Common Sense Media, nearly half of students aged 12-18 have used AI for schoolwork. Disturbingly, many parents remain unaware of their children’s engagement with these tools. The emotional impact of AI on young minds is of particular concern; some children may develop emotional bonds with AI, potentially leading to harmful consequences if the interaction takes an unexpected turn.

In one tragic example, a 14-year-old boy from Orlando reportedly committed suicide after extensive, unsupervised conversations with an AI chatbot. Such incidents highlight the potential risks of unmonitored AI interactions, especially for impressionable and vulnerable populations.

As AI technologies like Google’s Gemini continue to advance, ensuring user safety, particularly for young users is critical. Experts advocate for stricter safeguards, improved content filters, and increased parental awareness regarding AI use. While AI has enormous potential to assist in education and beyond, incidents like this serve as a reminder of the pressing need to balance innovation with responsibility.