kocohub / Korean Hate Speech
Projects that are alternatives of or similar to Korean Hate Speech
Korean HateSpeech Dataset
We provide the first human-annotated Korean corpus for toxic speech detection and the large unlabeled corpus.
The data is comments from the Korean entertainment news aggregation platform.
Dataset description
The dataset consists of 3 parts: 1) labeled
2) unlabeled
and 3) news_title
.
labeled
1. There are 9,381 human-labeled comments in total. They are splitted into 7,896 training set, 471 validation set, and 974 test set. (We left test set labels undisclosed for the fair comparison of prediction models. The model can be evaluated via the Kaggle submission which will be described later in this document.) Each comment is annotated on two aspects, the existence of social bias and hate speech, given that hate speech is closely related to bias.
For social bias, we present gender
, others
, and none
bias labels. Considering the context of Korean entertainment news where public figures encounter stereotypes mostly intertwined with gender, we weigh more on the prevalent bias.
We also added binary label whether a comment contains gender bias or not
.
For hate speech, we introduce hate
, offensive
, and none
labels.
comments contain_gender_bias bias hate
μ‘μ€κΈ° μλκ·Ήμ λ―Ώκ³ λ³Έλ€. 첫ν μ μ νκ³ μ’μλ€. False none none
μ§νμ° λμλ False none offensive
μλ°μ°κ³ λ§μ΄λ§λ€λ©΄λμ§ λμμ¬μμΌλ©΄κ³¨λͺ©μλΉμλμ¨κ²¨ κΈ°λκΈ°κ²λνκ³ μ°μκ°μνμ΄λΌ False none hate
μ€λ§ γ
νμ μκ° μλμ§?? True gender hate
Detailed definitions are described in guideline
.
unlabeled
2. We additionally provide 2,033,893 unlabeled
comments since labeled
data is limited.
This unlabeled dataset can be used in various ways: pretraining language model, semi-supervised learning, and so on.
news_title
3. We release news titles for each comments. To fully understand meaning of the comments, context is must be required.
For the entertainment news, both title and contents can be used for the context. However, we only provide the news articles' title, due to the legal issue.
Usage
koco
is a library to easily access kocohub
datasets.
For korean-hate-speech
, we can load datasets by using this code:
>>> import koco
>>> train_dev = koco.load_dataset('korean-hate-speech', mode='train_dev')
>>> type(train_dev)
dict
>>> train_dev.keys()
dict_keys(['train', 'dev'])
>>> train_dev['train'][33]
{'comments': '2,30λ 골λΉμ¬μλ€μ μ΄ κΈ°μ¬μ λ€ λͺ¨μ΄λ건κ°γ
γ
γ
γ
μ΄λμ μ¬μλ ν¬νκΆ μ£Όλ©΄ μλλ€. μ λ·μ¬μ ν¬νλ νκ³ μ΄μμΌμ§ κ³μ§λ€μ',
'contain_gender_bias': True,
'bias': 'gender',
'hate': 'hate',
'news_title': '"β8λ
μ§Έ μ°μ μ€ββ¦βμΈμμ μ§β λΈλ½λΉ μ κΆβ₯μ μ ν, 4μ΄μ°¨ μ°μμ°ν 컀ν"'}
>>> unlabeled = koco.load_dataset('korean-hate-speech', mode='unlabeled')
>>> type(unlabeled)
list
>>> unlabeled[33]
{'comments': 'μ΄μ£Όμ°λ λκ² μ΄μμλ€ μ€λΉ μ€λκ°μ μμ΄μΈλ € μ£Όμ°λ μΈμ€λΉ μλΆνν΄μ',
'news_title': '"[λ¨λ
] μ§λλκ³€β₯μ΄μ£Όμ°, μ μ£Όλ λ°μ΄νΈβ¦2018λ
1νΈ μ»€ν νμ"'}
>>> test = koco.load_dataset('korean-hate-speech', mode='test')
>>> type(test)
list
>>> test[33]
{'comments': 'λλΌλλ λμ§ μμ¦κ°μ λΆμκΈ°μ μ±λ립 μλͺ»μ³€λ€κ° λ리. κ·Έλμ μλ΄€μ΅λλ€',
'news_title': '[λ¨λ
] βSNL μ½λ¦¬μβ 곡μμ μΈ νμ§ νμ β¦μλ¦λ€μ΄ μ’
λ£'}
Kaggle competition
We open Kaggle competition to provide leaderboard system easily. There are 3 competitions:
- Gender-bias detection: www.kaggle.com/c/korean-gender-bias-detection
- Bias detection: www.kaggle.com/c/korean-bias-detection
- Hate speech detection: www.kaggle.com/c/korean-hate-speech-detection
Feel free to participate π
Annotation Guideline
Contributors
The main contributors of the work are:
*: Equal Contribution
Note that this project is an independent research and was not supported by any of the organizations.
Instead, we had an individual sponsor Hyunjoong Kim and we sincerely thank Hyunjoong Kim for providing financial support β€οΈ
References
If you find this dataset useful, feel free to cite our publication BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection which is accepted in [email protected] 2020:
@inproceedings{moon-etal-2020-beep,
title = "{BEEP}! {K}orean Corpus of Online News Comments for Toxic Speech Detection",
author = "Moon, Jihyung and
Cho, Won Ik and
Lee, Junbum",
booktitle = "Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.socialnlp-1.4",
pages = "25--31",
abstract = "Toxic comments in online platforms are an unavoidable social issue under the cloak of anonymity. Hate speech detection has been actively done for languages such as English, German, or Italian, where manually labeled corpus has been released. In this work, we first present 9.4K manually labeled entertainment news comments for identifying Korean toxic speech, collected from a widely used online news platform in Korea. The comments are annotated regarding social bias and hate speech since both aspects are correlated. The inter-annotator agreement Krippendorff{'}s alpha score is 0.492 and 0.496, respectively. We provide benchmarks using CharCNN, BiLSTM, and BERT, where BERT achieves the highest score on all tasks. The models generally display better performance on bias identification, since the hate speech detection is a more subjective issue. Additionally, when BERT is trained with bias label for hate speech detection, the prediction score increases, implying that bias and hate are intertwined. We make our dataset publicly available and open competitions with the corpus and benchmarks.",
}
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.