How Confident is ChatGPT in Kurdish? A Deep Dive into Its Sorani Skills
How Confident is ChatGPT in Kurdish?
A Deep Dive into Its Sorani Skills
Given the global impact of AI on communication and learning, there is an interest in testing the capabilities of AI tools like ChatGPT in less common and under-resourced languages. A study published in the UHD Journal of Science and Technology placed the test of ChatGPT in a systematic way against the abilities of the Kurdish language and, in particular, the Sorani dialect.
Under the Hood: How researchers tested the Bot
Researchers then formulated 50 multiple-choice questions about general knowledge, geography, culture, language structure and regional history to evaluate AI.
•Multi-Account Approach:The team employed four different user accounts and had the same 50 questions.•Repeated Rounds:Ten separate testing sessions over five days were completed by each account, for a total of 2,000 responses.
•Variability Factors:The study compared response accuracy by user profiles, time and topic types, for three variability factors.
Behind the Big Reveal: Surprising AI Insights
1.Decent Overall Accuracy:For Kurdish Sorani, ChatGPT scored an average of roughly 70% on all tests, indicating relatively strong general knowledge and basic language abilities.2.Accuracy was not consistent (account and time variability).User performance varied from an error ratio of 39.4% on the worst-performing user account to 24.8% on the best-performing account, depending on the timing of the request.3.Cultural and Literary Blindspots:The AI had a great deal of trouble with topics that are specialised or local. While there were many questions about classical Kurdish literature, renowned Sorani poets, and particular events from history, these were often forgotten in all accounts.4.Comparison with Arabic:When presented with the same questions translated into Arabic, ChatGPT scored slightly higher, averaging around 73.6% accuracy. This is because there is more training data available in Arabic than Kurdish.