An Update to a Functional Modeling Scoring Rubric With Overall and Question-Level Inter-Rater Reliability

Abstract

Functional modeling allows an engineer or a designer to separate component architecture from the purpose or the objective of a system. Many engineering design textbooks introduce functional modeling as a tool to aid with conceptual design, to assist with idea generation, to allow for conceptual decomposition of complex systems, to convey abstractions of systems at both high and low levels, and to reduce design fixation. This article presents an update to a previously published functional modeling rubric that increases the overall inter-rater reliability and the question-by-question inter-rater reliability through the inclusion of student examples, expert examples, two additional scoring questions, and detailed guidelines. These changes are validated by novice raters rescoring a prior set of data with the newly updated rubric to compare the results for improved inter-rater reliability. These rubric changes also make the assessment tool more accessible to individuals new to functional modeling. The analysis shows that the updated rubric improved the overall Cohen’s kappa from 0.65 to 0.76 and improved the lowest question-level Cohen’s kappa from 0.03 to 0.47, which indicates a substantial increase in reliability at the question level that subsequently improved the overall reliability. These results are discussed in detail followed by an abbreviated presentation of the updated rubric in the Appendix.