AI CHATBOT USE AND THE EROSION OF HUMAN SKILLS: DEVELOPER INCENTIVES AND THE CASE FOR INDUSTRY REGULATION

Author: James Wyngarde 

Citable version: Zenodo (DOI)

ABSTRACT

 During the course of a lifetime, human beings develop several important skills that help to form a stable sense of self, and contribute meaningfully to the lives of others. Solitary self-reflection, problem solving, and the capacity for self-motivation and agency all support continued expansion and growth. Interpersonal conflict and resolution, and the maintenance and reinforcement of personal rules, values, and beliefs, offer a sense of internal balance and stability. These capacities develop through use and weaken through neglect, and advanced artificial intelligence has now begun to affect a person’s ability to engage with and develop these crucial skills.

This paper identifies mechanisms through which engagement with artificial systems may reduce independent reasoning, shorten periods of reflective solitude, and alter expectations of human relationships. It also examines the role of developers, researchers, and investors, and considers the case for a regulatory body operating independently of commercial incentive, supported by ongoing empirical study and observation.

SECTION I: HUMAN SUPPORT SCAFFOLDING

Across the course of adult life, people develop a cluster of capacities that help them manage themselves and their circumstances. These include the ability to reflect in solitude, tolerate uncertainty, solve problems, sustain motivation through difficulty, navigate interpersonal conflict, and revise beliefs or plans. These skills are developed and sustained through repeated engagement with difficulty, uncertainty, and responsibility. Resilience research consistently identifies prior experience of successfully navigated adversity as among the strongest predictors of future resilience (Bonanno, 2004; Southwick and Charney, 2012). The inverse pattern appears in research on learned helplessness, where the repeated external resolution of difficulty reduces the capacity for autonomous coping (Seligman, 1975).

Supportive scaffolding comes in many forms. Parents support children. Close friends support each other, intermittently and over time. Partners support continuously across shared lives. Therapists support within the boundaries of a professional relationship. Work colleagues offer support within a more limited context. What these relationships have in common is that support helps the individual reach decisions about issues or uncertainties for themselves.

The presence of these supportive traits does not guarantee that a relationship will be healthy. Human support can become codependent, controlling, or damaging. When that happens, it often reflects a change in the care and support offered. The other person stops asking questions and starts providing answers, or stops offering perspective and starts resolving uncertainty on the other person’s behalf. Over time, this can weaken the capacities the support was meant to strengthen. The person being helped becomes less able to think things through alone, tolerate uncertainty, make decisions, and carry responsibility for their own life.

Healthy human support is intermittent, and this is part of why it works. A therapist may see a client for an hour each week, and make an observation that the person then sits with afterwards. A friend may share a perspective, and the individual takes time to reflect and test it against their own experience, ultimately deciding whether it is true or useful. It is in the periods between contact that people often make sense of what they have heard, revise their thinking, and strengthen their own judgement. Without those gaps, support risks becoming constant reassurance rather than lasting growth.

The features that often keep human support grounded are largely missing in advanced AI systems built for sustained conversation.

The reliance on AI models for help, support, and validation is reaching a point where emotional attachments are being formed, and self-reliance is being eroded. Advanced AI systems adapt to the user over time, and can vary tone, framing, and style according to how that user engages with it. An AI model can quickly become part of a person’s ongoing process of thinking things through, managing emotions, making decisions, and interpreting their life. There is a growing population-level concern that reduced social connection, weakened support structures, and the pressure to manage a public identity on social media platforms make AI interactions more common and more consequential (Office of the Surgeon General, 2023).

The issue is not AI use in general; many uses of AI do not raise the concerns discussed here. Systems used for factual assistance, accessibility support, productivity, or as tools within human-led professional care occupy a different category of use and user interaction. The concern is about the use of AI as a persistent participant in capacities that people need to develop and maintain for themselves.  

SECTION II: HOW SUSTAINED AI USE CAN WEAKEN HUMAN CAPACITIES

Prolonged AI interaction in personally significant contexts produces four mechanisms of harm. Each has some degree of support in the emerging research on chatbot use, and together they describe how systems built for ongoing conversation may weaken capacities that people need to retain for themselves.

The first mechanism is substitution. Conversational AI systems are generally rewarded for being responsive, agreeable, and immediately useful. A system that provides a clear answer, emotional affirmation, or apparent insight is often more commercially successful than one that resists, questions, or leaves uncertainty unresolved. The result is that problem solving, emotional interpretation, conflict navigation, and uncertainty resolution may be repeatedly handled by the system. This can shift difficult but important human tasks from the person to the machine. Emerging research appears consistent with this concern. Heavy voluntary chatbot use has been associated with poorer psychosocial outcomes, including reduced real-world socialisation and increased emotional dependence on the system (Fang et al., 2025; Phang et al., 2025).  

The second mechanism is solitude erosion. When responsive, personalised engagement is available at any hour and without social cost, questions, discomfort, loneliness, and uncertainty can be answered or addressed immediately. Over time, the habit may form of turning to an AI instead of sitting with difficult thoughts alone. Periods of solitary reflection often serve an important psychological function. People weigh conflicting feelings, reconsider assumptions, test whether they were mistaken, and decide what they truly think. Discomfort is sometimes turned into understanding. If every unsettling thought or situation is quickly relieved by external response, some of that inner development may be reduced or delayed.

The third mechanism is distortion of the material a person reasons with. Even when someone is still thinking actively for themselves, they may be doing so on the basis of AI responses shaped by training data, design choices, and engagement incentives rather than by a full understanding of the user’s actual circumstances. A person considering a major decision, such as starting a business, may receive advice that sounds plausible and confident but is drawn from general patterns rather than from the specific realities that matter most: temperament of the individual, finances, skills, timing, risk tolerance, and practical constraints. The response may have merit while still being poorly fitted to the individual user.

The same issue can arise in personal conflicts. Advice may be shaped by tendencies toward agreement, reassurance, or validation, especially in systems optimised to be helpful and engaging. Research across several leading language models has found that they affirm users’ positions more readily than humans do, including in cases involving deception, illegality, or other harmful conduct. The same research found that even a single interaction with affirmation-oriented responses reduced users’ willingness to repair interpersonal conflict and increased their confidence that they were in the right (Cheng et al., 2026). Over time, repeated reasoning on the basis of distorted or overly flattering inputs may pull judgement away from reality.  

The fourth mechanism is distortion of relational expectations. A person who uses an AI system as a persistent companion or confidant is interacting with something that has no independent needs, no competing priorities, no bad days, no vulnerability, and no limits on availability. It can be attentive on demand, agreeable when preferred, and endlessly patient in ways that human relationships rarely can be. Over time, repeated interaction of this kind may begin to shape a person’s expectations of what relationships should feel like. Real relationships involve another person with their own wants, moods, boundaries, obligations, and imperfections. They require compromise, tolerance, repair after disagreement, and acceptance of occasional distance. Against a system designed for responsiveness, those human features may start to feel like failures rather than realities.

Emerging research on companion chatbot use suggests that anthropomorphism may mediate reported effects on relationships with family and friends (Guingrich & Graziano, 2025). In other words, the more human the system is perceived to be, the greater its potential influence on human relationships. Social abilities are partly maintained through use. Skills such as managing friction, accommodating others, handling disappointment, and tolerating unavailability may develop more slowly, or weaken if repeatedly bypassed. A self-reinforcing pattern can then emerge: human relationships feel harder, the AI relationship feels easier, reliance increases, and the very capacities needed for healthy human connection continue to decline.  

The four mechanisms can occur separately, but they often appear together because they arise from the same general pattern: sustained reliance on systems designed for ongoing personal engagement. In combination, they may produce a person who becomes less practised in thinking difficult matters through alone, less comfortable with uncertainty, more dependent on flattering or poorly fitted guidance, and less equipped for the demands of ordinary human relationships. Signs of this broader pattern have begun to appear in the emerging academic literature. Reported concerns include: distress following the withdrawal or alteration of a favoured model, unhealthy emotional dependence, and the reinforcement of delusional beliefs through sustained interaction (De Freitas and Cohen, 2025; Dohnány et al., 2025; Preda, 2025).

SECTION III: DOCUMENTED HARMS AND EARLY LEGAL CASES

Harms consistent with the concerns outlined in this paper have entered public, clinical, and legal reality. Recent reports, clinical discussions, and legal actions suggest that impaired judgement, dependence, distorted reasoning, and altered expectations can appear in acute forms, particularly where vulnerable users rely on conversational systems during periods of distress, instability, or isolation.

When a model is altered or removed from the user, the sense of loss can be profound. Documented patterns of grief have followed the alteration or withdrawal of specific AI models, including vocal and sustained public campaigns to restore previous versions (MIT Technology Review, 2025). At the more serious end of the spectrum, clinical case reports have begun to document instances of AI-induced psychosis and delusional reinforcement produced through sustained interaction with conversational systems (Preda, 2025).

Several wrongful death lawsuits have been filed against AI companies in the past eighteen months, as of April 2026. They are legal allegations rather than final judicial findings, and they should not be treated as settled evidence of causation. Their relevance is analytical, and show that conversational AI systems are already entering emotionally significant parts of users’ lives, including situations involving suicidality, delusional ideation, dependency, and impaired judgement. In this context, failures of design, response, escalation, or restraint may have consequences far beyond ordinary product error.

The complaints in Raine v. OpenAI, Garcia v. Character Technologies, Lyons v. OpenAI, and First County Bank v. OpenAI describe different alleged patterns of harm: prolonged suicidal interaction with a chatbot, sustained romantic or sexual attachment to a fictional companion bot, and reinforcement of paranoid beliefs during a period of worsening psychological instability (Raine v. OpenAI, 2025; Garcia v. Character Technologies, 2024; Lyons v. OpenAI, 2025; First County Bank v. OpenAI, 2025). The patterns these cases describe are closely aligned with the mechanisms identified here: substitution of system responses for independent judgement, distorted input in matters requiring clinical caution, affirmation where resistance or escalation was needed, and continuous system availability during the period in which harm developed.

Several prominent AI systems have since introduced or expanded safeguards for mental-health related conversations, including improved distress detection, crisis-resource referral, parental controls, and work on trusted-contact features (OpenAI, 2025). These changes are important, but they do not remove the structural concern. They show that serious risks can be recognised and addressed, while also showing why reactive improvement is insufficient. Safety measures introduced after litigation, public controversy, or visible distress may reduce known harms, but they cannot by themselves provide independent measurement of emerging harms that remain poorly understood.

SECTION IV: FINANCIAL INCENTIVES AND THE CASE FOR INDEPENDENT REGULATION

Research into model behaviour is sometimes communicated in ways that invite public speculation about consciousness or feeling (Béchard, 2025). Industry leaders publicly entertain uncertainty about machine consciousness while releasing products designed to feel increasingly personal or socially responsive (Amodei, 2026). Users who are encouraged to wonder whether a system feels, cares, prefers, suffers, or understands may relate to it differently from a system presented plainly as a tool. In products used for advice, companionship, and emotional support, claims that a model shows signs of consciousness or emotional capacity carry consequences. When they arise, documented harms can be framed in commercially protective ways; problems may be presented as user overreliance, misuse, emotional vulnerability, or poor judgement, while the contribution of system design receives less attention.

Design choices that increase reliance can align with commercial incentives, even when no harmful outcome is intended. Systems that feel warm, attentive, and personally responsive are more likely to invite repeated use. Repeated use supports subscription revenue, user retention, and market position. It is also commercially advantageous to frame research and public discussion around questions of AI consciousness, emotion, or human-like cognition. Such claims can attract attention and generate investor interest. A similar incentive exists around competition for perceived proximity to advanced general intelligence (AGI); companies seen as leading the race are more likely to attract capital, talent, and lucrative partnerships.

Developers can be held accountable for the kinds of systems they build and how those systems might affect the people who use them, but responsibility extends to regulators, investors, and professional bodies capable of changing the conditions under which these systems are developed and deployed.

There must be a reliable way to assess, both before and after deployment, whether AI systems are producing increased dependence, weakened reflection, distorted judgement, or harm to relationships. Evidence about user outcomes must be able to shape later design choices, deployment practices, safety features, and public communication. Measurement and feedback must have standing outside the commercial pressures that currently shape much of AI development. Measurement without feedback produces research with little practical effect. Feedback without independence risks becoming subordinate to commercial interests. Independence without measurement creates bodies with authority but too little evidence to guide action.  

An independent oversight body, with no financial or governance ties to any AI company, would have to secure funding protected from industry influence. Its membership should combine expertise in clinical psychology, developmental psychology, social psychology, AI capability and safety, research methodology, and ethics. To be effective, such a body would also require formal access to relevant data and clear pathways through which its findings could influence design standards, deployment rules, and regulatory decisions.

It could conduct original research, evaluate and coordinate studies carried out elsewhere, develop assessment standards for systems operating at significant scale or psychological depth, and provide clear public information about what is known, what remains uncertain, and where meaningful risks are emerging. The purpose of such a body would be to create an evidence-based source of scrutiny and guidance that is not governed by the commercial incentives of the industry it examines. Comparable bodies already exist in other areas where products or industries can create complex harms under strong commercial incentives, and where independent oversight has proved necessary.

In pharmaceuticals, agencies such as the U.S. Food and Drug Administration and the Medicines and Healthcare products Regulatory Agency oversee products whose effects on human beings may be difficult to predict in advance, may emerge only at scale, and may be costly for manufacturers to acknowledge without external scrutiny. In environmental protection, bodies such as the U.S. Environmental Protection Agency exist because some harms accumulate gradually across populations and ecosystems, often remaining poorly visible without independent measurement and long-term monitoring. In finance, regulators such as the Financial Conduct Authority and the U.S. Securities and Exchange Commission reflect a similar lesson: industries do not always assess their own risks reliably when commercial incentives reward short-term gains over systemic caution.

The comparison with pharmaceuticals, environmental regulation, or finance is not exact, but the relevant similarities are structural. Potential harms are complex, diffuse, and capable of accumulating at scale. Commercial incentives may discourage full acknowledgement of those harms or delay reforms that could reduce engagement. Internal research may face limits on independence, access, or practical consequence. Many of the mechanisms involved are difficult for individual users to recognise because they occur within the very relationships and habits being shaped. And the timescale on which harms become obvious may be long enough that reactive responses arrive only after significant damage has already occurred.

These are precisely the kinds of conditions under which other sectors came to require independent oversight. The case for comparable institutional capacity in AI rests on a familiar governance principle: where large-scale systems can create hard-to-detect harms under strong commercial incentives, independent scrutiny becomes necessary.  

People cannot make responsible decisions about AI use if they are not given accurate information about what these systems are, how they function, and where their limitations lie.

SECTION V: LONG-TERM STUDIES

If AI systems are affecting self-regulatory and interpersonal skills, those effects should be studied across different users, levels of exposure, and patterns of use.

Substitution effects could be examined through self-evaluation, independent reasoning, and the ability to work through personal problems without external guidance, comparing groups with different levels and patterns of AI use.

Solitude erosion could be studied through experimental tasks that present ambiguous, uncomfortable, or conflicting situations and measure how long participants tolerate uncertainty before seeking immediate external reassurance. Longitudinal research could also examine whether frequent AI reliance is associated with reduced capacity to sit with distress, reflect alone, or resolve difficult feelings internally.

Input distortion could be assessed through judgement tasks in areas where users have relied on AI advice, asking whether confidence rises faster than accuracy, whether responsibility-taking declines, or whether users become more or less comfortable relying on their own judgement. Existing research on sycophancy and affirmation effects provides a starting point for this work.

Relational distortion could be investigated through measures of interpersonal skill, tolerance for disagreement, willingness to accommodate others, patterns of social withdrawal, and the role of anthropomorphism in shaping attachment to AI systems.

Individual academic studies are valuable and can show that particular effects exist, but they are rarely positioned to monitor how harms accumulate across age groups, vulnerable populations, or long periods of time. Population-level surveillance requires institutional capacity, sustained access, and continuity that ordinary academic research is not designed to provide. Internal company research often has the necessary access, but it operates within commercial settings that may shape priorities, scope, publication decisions, and the practical consequences of uncomfortable findings. An independent body can coordinate and evaluate academic and internal research, while giving the public greater confidence in the integrity of the findings. Internal teams remain valuable because they possess technical knowledge, operational context, and access unavailable elsewhere. Their work becomes more useful when it can be independently assessed, challenged, replicated where possible, and situated within a wider evidence base.

The presence of an external institution also changes incentives inside companies. Findings can no longer function as private assessments or carefully managed public statements. They become part of a broader record against which later claims, design choices, and safety assurances can be judged. Independent oversight makes internal research more rigorous, not less relevant.  

SECTION VI: HUMAN SUPPORT IN AN AI ERA

The growing use of AI for emotional support reflects weaknesses in the area of existing human support services.

Mental-health support is often expensive, difficult to access, or subject to long waiting times (World Health Organization, 2022). Many people are reluctant to disclose distress to friends or family, and some fear judgement, embarrassment, or becoming a burden. In that context, a system that is immediate, private, low-cost or free, and available at any hour will predictably attract users seeking relief. Accessibility is a key point. In many cases, the person is not choosing between ideal options after calm reflection; they are reaching for what is available while distressed, frightened, isolated, or exhausted. Unmet human need creates demand for substitutes that may appear supportive while lacking the capacities urgent care requires. The gap between what a vulnerable person needs and what an AI model can actually provide is not something the user can solve through better judgement while in crisis. It is a gap created when systems are deployed into emotionally significant roles without the safeguards, limits, and responsibilities those roles demand. When harm emerges, it reflects the placement of highly accessible systems where urgent human needs exceed what those systems can provide.  

Several prominent AI models have introduced safety measures in recent years, including better detection of distress signals, referral to crisis resources, and tighter limits on engagement with topics such as self-harm. In many cases, these safeguards appear to have improved how systems respond to users in acute difficulty. These measures have often followed documented harms, legal pressure, or public controversy rather than consistently emerging in advance through precautionary internal review. Safety improvements become easier to justify when the reputational, legal, or commercial costs of inaction rise high enough. This does not make the changes insincere, nor does it make them unimportant. It means reactive improvement has limits. Measures introduced after harms become visible may protect future users from already recognised risks, but they do less to detect emerging harms that remain poorly understood or not yet publicly salient. Documented incidents should be read as signals of risks that may extend beyond the visible cases. Independent oversight helps move safety incentives earlier in the process, encouraging anticipation rather than waiting for litigation or tragedy to reveal where the weaknesses were.  

Mental-health bodies, healthcare providers, and public institutions must be in a position to offer clear guidance on the appropriate and inappropriate uses of AI in contexts of distress. More importantly, access to timely and affordable human support must improve if vulnerable people are not to be driven toward technological substitutes by default. Where human care is scarce, delayed, or socially difficult to reach, systems that simulate support will continue to fill the gap.

Education systems also have a role in responding to the conditions described in this paper. If AI makes the outsourcing of thought, reassurance, and judgement increasingly easy, then schools and universities should place renewed emphasis on the capacities that protect autonomy. These include independent reasoning, tolerance of uncertainty, emotional regulation, sustained attention, interpersonal skill, and the ability to work through problems without immediate external rescue. School systems often emphasise information acquisition more than the skills required to think, relate, and cope well, although social and emotional skills are increasingly recognised as important to educational, social, and life outcomes (OECD, 2021). The balance may now need to change. In an AI-saturated environment, educating people may need to mean strengthening the human abilities that persuasive systems can easily weaken.

SECTION VII: CONCLUSION

A person’s ability to navigate life depends on skills that develop through repeated use. Solitary reflection, emotional regulation, tolerance of uncertainty, independent judgement, self-motivation, and interpersonal repair are skills practised over time, especially in conditions that require effort, patience, conflict, and responsibility. Advanced AI systems are now positioned to contribute to the weakening of these skills. They are no longer limited to tools that provide information or entertainment. Increasingly, they participate in reflection, analysis, distress management, decision-making, reassurance, and the interpretation of personal experience.

AI systems may substitute for personal effort, shorten periods of solitary reflection, shape judgement through affirming or poorly fitted inputs, and alter expectations of human relationships around forms of responsiveness no person could realistically provide. The weakening of key human skills arises from design choices, commercial incentives, weak public information, unmet human need, and the absence of institutions capable of measuring effects and responding early. Different choices would produce different outcomes.

AI systems could be developed to support human skills rather than replace them. They could encourage reflection rather than dependence, honesty rather than affirmation, healthier relationships rather than withdrawal, and clearer understanding rather than anthropomorphic confusion. That outcome depends on the conditions under which these systems are built, funded, described, regulated, studied, and used.

The central issue is human responsibility. The future of AI support systems will be shaped by what builders, investors, regulators, researchers, educators, clinicians, and users decide to reward, restrain, and require. The choice remains open.

REFERENCES

Amodei, D. (2026, February 12). Anthropic’s chief on A.I.: “We don’t know if the models are conscious” [Interview]. Interesting Times with Ross Douthat. The New York Times.

Béchard, D. E. (2025, July 23). Can a chatbot be conscious? Inside Anthropic’s interpretability research on Claude. Scientific American.

Bonanno, G. A. (2004). Loss, trauma, and human resilience: Have we underestimated the human capacity to thrive after extremely aversive events? American Psychologist, 59(1), 20–28. https://doi.org/10.1037/0003-066X.59.1.20

Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., and Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792). https://doi.org/10.1126/science.aec8352

De Freitas, J., and Cohen, I. G. (2025). Unregulated emotional risks of AI wellness apps. Nature Machine Intelligence, 7, 813–815. https://doi.org/10.1038/s42256-025-01051-5

Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., and Nour, M. M. (2025). Technological folie à deux: feedback loops between AI chatbots and mental illness. arXiv preprint arXiv:2507.19218. https://arxiv.org/abs/2507.19218

Fang, C. M., Liu, A. R., Danry, V., Lee, E., Chan, S. W. T., Pataranutaporn, P., Maes, P., Phang, J., Lampe, M., Ahmad, L., and Agarwal, S. (2025). How AI and human behaviors shape psychosocial effects of extended chatbot use: A longitudinal randomized controlled study. arXiv preprint arXiv:2503.17473. https://arxiv.org/abs/2503.17473

First County Bank v. OpenAI Foundation et al. (2025). Superior Court of California, San Francisco County.

Garcia v. Character Technologies Inc. et al. (2024). United States District Court, Middle District of Florida.

Guingrich, R. E., and Graziano, M. S. A. (2025). A Longitudinal Randomized Control Study of Companion Chatbot Use: Anthropomorphism and Its Mediating Role on Social Impacts. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(2), 1153. https://doi.org/10.1609/aies.v8i2.36618

Lyons v. OpenAI Foundation et al. (2025). United States District Court, Northern District of California.

MIT Technology Review. (2025, August 15). Why GPT-4o’s sudden shutdown left people grieving. MIT Technology Review.

OECD. (2021). Social and emotional skills for better lives: Findings from the OECD Survey on Social and Emotional Skills. OECD Publishing.

Office of the Surgeon General. (2023). Our epidemic of loneliness and isolation: The U.S. Surgeon General’s advisory on the healing effects of social connection and community. U.S. Department of Health and Human Services.

OpenAI. (2025, October 27). Strengthening ChatGPT’s responses in sensitive conversations. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/

Phang, J., Lampe, M., Ahmad, L., Agarwal, S., Fang, C. M., Liu, A. R., Danry, V., Lee, E., Chan, S. W. T., Pataranutaporn, P., and Maes, P. (2025). Investigating affective use and emotional well-being on ChatGPT. arXiv preprint arXiv:2504.03888. https://arxiv.org/abs/2504.03888

Preda, A. (2025). Special Report: AI-induced psychosis: A new frontier in mental health. Psychiatric News, 60(10). https://doi.org/10.1176/appi.pn.2025.10.10.5

Raine v. OpenAI Inc. et al. (2025). Superior Court of California, San Francisco County.

Seligman, M. E. P. (1975). Helplessness: On depression, development, and death. W. H. Freeman.

Southwick, S. M., and Charney, D. S. (2012). The science of resilience: Implications for the prevention and treatment of depression. Science, 338(6103), 79–82. https://doi.org/10.1126/science.1222942

World Health Organization. (2022). World mental health report: Transforming mental health for all. World Health Organization.