Abstract

Access to quality mental health care remains a global challenge, especially in underserved regions. While large language models (LLMs) show promise in providing AI-driven therapeutic support, their massive size often limits practical deployment due to high computational and memory  requirements. This study presents a fine-tuned version of Phi-3, a compact yet powerful language model, tailored specifically for empathetic mental health counseling. By leveraging Low-Rank Adaptation (LoRA) and the Unsloth framework, we significantly improved response relevance and emotional sensitivity, while achieving a 2x increase in training speed and reducing memory consumption by 70%. The model was fine-tuned on three diverse dataset of publicly available mental health dialogues, enabling it to generate supportive and context-aware responses. Evaluation using BERTScore metrics demonstrated strong performance, with an F1-score of 0.8429 during training and 0.8385 on unseen test data. Our results suggest that small-scale, efficiently fine tuned LLMs like Phi-3 can offer accessible, accurate, and scalable mental health support bridging the gap between technological capability and real-world usability for the average user.

Keywords

  • Mental health
  • Anxiety
  • Depression
  • Stigma
  • Small Language Models
  • AI-driven Therapy
  • Fine-tuning
  • Emotional Intelligence
  • Personalized Interaction
  • Digital Mental Health
  • AI-driven Therapy
  • Low-Resource Deployment

References

  1. 1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
  2. 2. Saleh Afroogh, Yasser Poreesmaiel, and Junfeng Jiao. Hallucinations vs. predictions: Reframing uncertainty in llm-generated medical responses. In ICLR 2025 Workshop on Machine Learning for Genomics Explorations.
  3. 3. Antonio Agliata, Antonio Pilato, Sorrentino Mariacarmen, Salvatore Bottiglieri, Emanuel Di Nardo, and Angelo Ciaramella. Generative ai and emotional health: Innovations with haystack. In 2024 IEEE Symposium on Computers and Communications (ISCC), pages 1–4. IEEE, 2024.
  4. 4. Arun Brahma. Mental health chatbot dataset, 2023.
  5. 5. Michael Han Daniel Han and Unsloth team. Unsloth, 2023.
  6. 6. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. Predicting depression via social media. In Proceedings of the international AAAI conference on web and social media, volume 7, pages 128–137, 2013.
  7. 7. Kerstin Denecke, Alaa Abd-Alrazaq, and Mowafa Househ. Artificial intelligence for chatbots in mental health: opportunities and challenges. Multiple perspectives on artificial intelligence in healthcare: Opportunities and challenges, pages 115–128, 2021.
  8. 8. Graciela Gonzalez-Hernandez, Abeed Sarker, Karen O’Connor, and Guergana Savova. Capturing the patient’s perspective: a review of advances in natural language processing of healthrelated text. Yearbook of medical informatics, 26(01):214–227, 2017.
  9. 9. Jagoda Grzejszczak, Dominik Strzelecki, Agata Gabryelska, and Magdalena Kotlicka-Antczak. Evaluation of covid-19 effect on mental health, self-harm, and suicidal behaviors in children and adolescents population. Journal of clinical medicine, 13(3):744, 2024.
  10. 10. Steven D Hollon and Aaron T Beck. Cognitive therapy of depression. Cognitive-behavioral interventions: Theory, research, and procedures, pages 153–203, 1979.
  11. 11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  12. 12. Shaoxiong Ji, Shirui Pan, Xue Li, Erik Cambria, Guodong Long, and Zi Huang. Suicidal ideation detection: A review of machine learning methods and applications. IEEE Transactions on Computational Social Systems, 8(1):214–226, 2020.
  13. 13. Sijie Ji, Xinzhe Zheng, Jiawei Sun, Renqi Chen, Wei Gao, and Mani Srivastava. Mindguard: Towards accessible and sitgma-free mental health first aid via edge llm. arXiv preprint arXiv:2409.10064, 2024.
  14. 14. Nikita Kanodia, Khandakar Ahmed, and Yuan Miao. Question answering model based conversational chatbot using bert model and google dialogflow. In 2021 31st International Telecommunication Networks and Applications Conference (ITNAC), pages 19–22. IEEE, 2021.
  15. 15. Joyce HL Lui, David K Marcus, and Christopher T Barry. Evidence-based apps? a review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice, 48(3):199, 2017.
  16. 16. Laura Martinengo, Elaine Lum, and Josip Car. Evaluation of chatbot-delivered interventions for self-management of depression: Content analysis. Journal of affective disorders, 319:598–607, 2022.
  17. 17. Augustine Meier, Micheline Boivin, and Molisa Meier. Working through the transference of an unresolved separation/individuation pattern: a case study using theme-analysis. The helping relationship: Healing and change in community context, pages 101–129, 2010.
  18. 18. Mpingale. Mental health chat dataset, Year.
  19. 19. World Health Organization. The world health report 2001: Mental health: new understanding, new hope. 2001.
  20. 20. Aravind Pillai. Screening for common mental disorders in primary care in low and middle income countries: A rational approach to address the mental health treatment gap? PhD thesis, Columbia University, 2020.
  21. 21. Eugene Sadler-Smith and Vita Akstinaite. Human hubris, anthropogenic climate change, and an environmental ethic of humility. Organization & Environment, 35(3):446–467, 2022.
  22. 22. Vasu Saini, Luis Guada, and Dileep R Yavagal. Global epidemiology of stroke and access to acute ischemic stroke interventions. Neurology, 97(20 Supplement 2):S6–S16, 2021.
  23. 23. Talha Abdul Hakeem Shaikh and Manisha Mhetre. Autonomous ai chat bot therapy for patient with insomnia. In 2022 IEEE 7th International conference for Convergence in Technology (I2CT), pages 1–5. IEEE, 2022.
  24. 24. Manoj Kumar Tamrakar and Abhishek Badholia. Scientific study of technological chatbot adoption in customer service. In 2022 3rd International Conference on Electronics and Sustainable Communication Systems (ICESC), pages 1117–1123. IEEE, 2022.
  25. 25. Adam Timmis, Nick Townsend, Chris P Gale, Aleksandra Torbica, Maddalena Lettino, Steffen E Petersen, Elias A Mossialos, Aldo P Maggioni, Dzianis Kazakiewicz, Heidi T May, et al. European society of cardiology: cardiovascular disease statistics 2019. European heart journal, 41(1):12–85, 2020.
  26. 26. Jacob White. Pubmed 2.0. Medical reference services quarterly, 39(4):382–387, 2020.