From Vanderbilt University:
In April of this year, Turnitin released an update to their product that reviewed submitted papers and presented their determination of how much of a paper was written by AI. As we outlined at that time, many people had important concerns and questions about this new tool, namely how the product exactly works and how reliable the results would be. After several months of using and testing this tool, meeting with Turnitin and other AI leaders, and talking to other universities who also have access, Vanderbilt has decided to disable Turnitin’s AI detection tool for the foreseeable future. This decision was not made lightly and was made in pursuit of the best interests of our students and faculty.
When Turnitin launched its AI-detection tool, there were many concerns that we had. This feature was enabled for Turnitin customers with less than 24-hour advance notice, no option at the time to disable the feature, and, most importantly, no insight into how it works. At the time of launch, Turnitin claimed that its detection tool had a 1% false positive rate (Chechitelli, 2023). To put that into context, Vanderbilt submitted 75,000 papers to Turnitin in 2022. If this AI detection tool was available then, around 750 student papers could have been incorrectly labeled as having some of it written by AI. Instances of false accusations of AI usage being leveled against students at other universities have been widely reported over the past few months, including multiple instances that involved Turnitin (Fowler, 2023; Klee, 2023). In addition to the false positive issue, AI detectors have been found to be more likely to label text written by non-native English speakers as AI-written (Myers, 2023).
Additionally, there is a larger question of how Turnitin detects AI writing and if that is even possible. To date, Turnitin gives no detailed information as to how it determines if a piece of writing is AI-generated or not. The most they have said is that their tool looks for patterns common in AI writing, but they do not explain or define what those patterns are. Other companies that offer popular AI detectors have either begun to either pivot to other business models (Edwards, 2023) or closed down entirely (Coldewey, 2023). Even if other third-party software claimed higher accuracy than Turnitin, there are real privacy concerns about taking student data and entering it into a detector that is managed by a separate company with unknown privacy and data usage policies. Fundamentally, AI detection is already a very difficult task for technology to solve (if it is even possible) and this will only become harder as AI tools become more common and more advanced. Based on this, we do not believe that AI detection software is an effective tool that should be used.
Link to the rest at Vanderbilt University