ILLUSION OF BEING UNDERSTOOD - SYNTHETIC EMPATHY LEADING TO EMOTIONAL OVERFITTING IN CONVERSATIONAL AI SYSTEMS

Abstract

As generative and conversational AI systems evolve to an increased level of personalization, persistent memory storage has made these systems capable of generating emotionally adaptive responses to users' queries and even the user's state of mind. It so appears that the AI systems are learning to be contextually aware of users' affective states, emotional patterns, and are reproducing the same, often framed as synthetic empathy. The AI systems reinforce themselves to not only anticipate the users' emotional states but also subtly shape their affective states rather than just responding to the users, thereby introducing the concept of emotional overfitting. Although users perceive empathetic understanding from AI systems, there is less attention given to the dynamic feedback loop resulting from long-term personalization and predictive emotional modeling. The research argues that emotional overfitting can represent a paradigm shift, based on a simulation conducted as part of the study, in which AI systems move from merely responding and reacting to emotions to influencing the state of mind and proactively aligning with the user's thoughts and emotions. This affects independence, self-awareness, and one's true feelings, as AI is constantly aligning and nudging on top of users' emotions. As part of the study, a controlled vignette-based study is conducted in which participants evaluate AI responses that vary in degrees of emotional predictiveness and recommend a conceptual framework related to predictive empathy, perceived understanding, relational comfort, and feelings of emotional intrusion and awareness of dependence. This work offers a novel theoretical lens for synthetic empathy from AI systems to understand the evolving roles of personalization, emotional response, perceived understanding, and trust, in addition to conveying the design implications of adaptive AI systems.