YaBeSH Engineering and Technology Library

    • Journals
    • PaperQuest
    • YSE Standards
    • YaBeSH
    • Login
    View Item 
    •   YE&T Library
    • ASME
    • Journal of Mechanical Design
    • View Item
    •   YE&T Library
    • ASME
    • Journal of Mechanical Design
    • View Item
    • All Fields
    • Source Title
    • Year
    • Publisher
    • Title
    • Subject
    • Author
    • DOI
    • ISBN
    Advanced Search
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Archive

    Large Language Models for Predicting Empathic Accuracy Between a Designer and a User

    Source: Journal of Mechanical Design:;2024:;volume( 147 ):;issue: 004::page 41401-1
    Author:
    Fabunmi, Oluwatoba
    ,
    Halgamuge, Saman
    ,
    Beck, Daniel
    ,
    Holtta-Otto, Katja
    DOI: 10.1115/1.4067227
    Publisher: The American Society of Mechanical Engineers (ASME)
    Abstract: Empathic design research aims to gain deep and accurate user understanding. We can measure the designer's empathic ability as empathic accuracy (EA) in understanding the user's thoughts and feelings during an interview. However, the EA measure currently relies on human rating and is thus time-consuming, making the use of large language models (LLMs) an attractive alternative. It is essential to consider two significant constraints when implementing LLMs as a solution: the choice of LLM and the impact of domain-specific datasets. Datasets of the interactions between the designer and the user are not generally available. We present such a dataset consisting of the EA task employed in user interviews to measure empathic understanding. It consists of over 400 pairs of user thoughts or feelings matched with a designer's guess of the same and the human ratings of the accuracy. We compared the performance of six sentence embedding state-of-the-art LLMs with different pooling techniques on the EA task. We used the LLMs to extract semantic information before and after fine-tuning. We conclude that directly using LLMs based on their reported performance in general language tasks could result in errors when judging a designer's empathic ability. We also found that fine-tuning LLMs on our dataset improved their performance, but the model's ability to fit the EA task and pooling method also determined the LLM's performance. The results will provide insight for other LLM-based similarity analyses in design.
    • Download: (776.5Kb)
    • Show Full MetaData Hide Full MetaData
    • Get RIS
    • Item Order
    • Go To Publisher
    • Price: 5000 Rial
    • Statistics

      Large Language Models for Predicting Empathic Accuracy Between a Designer and a User

    URI
    http://yetl.yabesh.ir/yetl1/handle/yetl/4306615
    Collections
    • Journal of Mechanical Design

    Show full item record

    contributor authorFabunmi, Oluwatoba
    contributor authorHalgamuge, Saman
    contributor authorBeck, Daniel
    contributor authorHoltta-Otto, Katja
    date accessioned2025-04-21T10:38:48Z
    date available2025-04-21T10:38:48Z
    date copyright12/10/2024 12:00:00 AM
    date issued2024
    identifier issn1050-0472
    identifier othermd_147_4_041401.pdf
    identifier urihttp://yetl.yabesh.ir/yetl1/handle/yetl/4306615
    description abstractEmpathic design research aims to gain deep and accurate user understanding. We can measure the designer's empathic ability as empathic accuracy (EA) in understanding the user's thoughts and feelings during an interview. However, the EA measure currently relies on human rating and is thus time-consuming, making the use of large language models (LLMs) an attractive alternative. It is essential to consider two significant constraints when implementing LLMs as a solution: the choice of LLM and the impact of domain-specific datasets. Datasets of the interactions between the designer and the user are not generally available. We present such a dataset consisting of the EA task employed in user interviews to measure empathic understanding. It consists of over 400 pairs of user thoughts or feelings matched with a designer's guess of the same and the human ratings of the accuracy. We compared the performance of six sentence embedding state-of-the-art LLMs with different pooling techniques on the EA task. We used the LLMs to extract semantic information before and after fine-tuning. We conclude that directly using LLMs based on their reported performance in general language tasks could result in errors when judging a designer's empathic ability. We also found that fine-tuning LLMs on our dataset improved their performance, but the model's ability to fit the EA task and pooling method also determined the LLM's performance. The results will provide insight for other LLM-based similarity analyses in design.
    publisherThe American Society of Mechanical Engineers (ASME)
    titleLarge Language Models for Predicting Empathic Accuracy Between a Designer and a User
    typeJournal Paper
    journal volume147
    journal issue4
    journal titleJournal of Mechanical Design
    identifier doi10.1115/1.4067227
    journal fristpage41401-1
    journal lastpage41401-12
    page12
    treeJournal of Mechanical Design:;2024:;volume( 147 ):;issue: 004
    contenttypeFulltext
    DSpace software copyright © 2002-2015  DuraSpace
    نرم افزار کتابخانه دیجیتال "دی اسپیس" فارسی شده توسط یابش برای کتابخانه های ایرانی | تماس با یابش
    yabeshDSpacePersian
     
    DSpace software copyright © 2002-2015  DuraSpace
    نرم افزار کتابخانه دیجیتال "دی اسپیس" فارسی شده توسط یابش برای کتابخانه های ایرانی | تماس با یابش
    yabeshDSpacePersian