Get the latest Science News and Discoveries

Large Language Models on Wikipedia Editing: The Stylistic Mask Effect in a Randomized Controlled Intervention in Health Education

The effects of large language models (LLMs) on content, references, and language quality in Wikipedia assignments remain unclear. This randomized controlled study assessed how model assistance influences the quality of Wikipedia edits created by undergraduate audiology students. Thirty-six participants were assigned to two groups: Group 1 (G1) edited without model support, and Group 2 (G2) edited with ChatGPT support. Twenty blinded expert reviewers evaluated a sample of 30 texts (15 per group) using a six-item Likert-scale instrument covering Content, References, and Language. Inter-rater reliability was high (Gwet's AC2 = 0.80). The analysis suggests that LLM assistance may lead to a significant improvement in Language (β = 0.400; p = 0.023; Cliff's δ = 0.400). G2 improved by 0.68 ± 0.47, while G1 improved by 0.28 ± 0.57. The analysis found no significant differences for Content (β = 0.167; p = 0.214) or References (β = −0.267; p = 0.859), although G2 scores in the latter category trended lower. Output counts were not associated with quality improvement in either group. This asymmetry is proposed as a potential Stylistic Mask Effect. It describes a gap between surface language gains and limited substantive development. LLM assistance appears to risk substituting constructive contribution with linguistic polish. This record contains the preprint manuscript. The manuscript has been submitted for peer review and has not yet been published. Dataset and code are made available to support transparency of the reported analyses.

None

Get the Android app