Evaluation of social bot detection models

Social bots are employed to automatically perform online social network activities; thereby, they can also be utilized in spreading misinformation and malware. Therefore, many researchers have focused on the automatic detection of social bots to reduce their negative impact on society. However, it is challenging to evaluate and compare existing studies due to difficulties and limitations in sharing datasets and models. In this study, we conduct a comparative study and evaluate four different bot detection systems in various settings using 20 different public datasets. We show that high-quality datasets covering various social bots are critical for a reliable evaluation of bot detection methods. In addition, our experiments suggest that Botometer is preferable to others in order to detect social bots.