Jigsaw Unintended Bias in Toxicity Classification
This respository contains my code for competition in kaggle.
7th Place Solution for Jigsaw Unintended Bias in Toxicity Classification
Team: Abhishek Thakur, Duy, R0seNb1att, atfujita
All models(Team)
Public LB: 0.94729(3rd)
Private LB: 0.94660(7th)
Note: This repository contains only my models and only train script.
My models(5 Model averaging)
Public LB: 0.94719
Private LB: 0.94651
Thanks to Abhishek and Duy's wonderful models and support, I was able to get better results.
Set up
- Particularly important libraries are listed in requirements.txt
Models
I created 5 models
-
LSTM
- Based on the Quora competition model
- Architecture: LSTM + GRU + Self Attention + Max pooling
- Word embeddings: concat glove and fasttext.
- Optimizer: AdamW
- Train:
- max_len = 220
- n_splits = 10
- batch_size = 512
- train_epochs = 7
- base_lr, max_lr = 0.0005, 0.003
- Weight Decay = 0.0001
- Learning schedule: CyclicLR
-
BERT
- The model is based on yuval reina's graet kernel
- Changes are loss function and preprocessing.
- I created 4 BERT models.
- BERT-Base Uncased
- BERT-Base Cased
- BERT-Large Uncased(Whole Word Masking)
- BERT-Large Cased(Whole Word Masking)
- Train:
- max_len = 220
- train samples = 1.7M, val samples= 0.1M
- batch_size = 32(Base), 4(Large)
- accumulation_steps = 1(Base), 16(Large)
- train_epochs = 2
- lr = 2e-5
Worked well
The loss function was very important in this competition.
In fact, all winners used different loss functions.
My loss function is below.
y_columns = ['target']
y_aux_train = train_df[['target', 'severe_toxicity', 'obscene',
'identity_attack', 'insult',
'threat',
'sexual_explicit'
]]
y_aux_train = y_aux_train.fillna(0)
identity_columns = [
'male', 'female', 'homosexual_gay_or_lesbian', 'christian', 'jewish',
'muslim', 'black', 'white', 'psychiatric_or_mental_illness']
# Overall
weights = np.ones((len(train_df),)) / 4
# Subgroup
weights += (train_df[identity_columns].fillna(0).values >= 0.5).sum(
axis=1).astype(bool).astype(np.int) / 4
# Background Positive, Subgroup Negative
weights += (((train_df['target'].values >= 0.5).astype(bool).astype(np.int) +
(train_df[identity_columns].fillna(0).values < 0.5).sum(
axis=1).astype(bool).astype(np.int)) > 1).astype(
bool).astype(np.int) / 4
# Background Negative, Subgroup Positive
weights += (((train_df['target'].values < 0.5).astype(bool).astype(np.int) +
(train_df[identity_columns].fillna(0).values >= 0.5).sum(
axis=1).astype(bool).astype(np.int)) > 1).astype(
bool).astype(np.int) / 4
y_train = np.vstack(
[(train_df['target'].values >= 0.5).astype(np.int), weights]).T
y_train = np.hstack([y_train, y_aux_train])
def custom_loss(data, targets):
''' Define custom loss function for weighted BCE on 'target' column '''
bce_loss_1 = nn.BCEWithLogitsLoss(
weight=targets[:, 1:2])(data[:, :1], targets[:, :1])
bce_loss_2 = nn.BCEWithLogitsLoss()(data[:, 1:2], targets[:, 2:3])
bce_loss_3 = nn.BCEWithLogitsLoss()(data[:, 2:3], targets[:, 3:4])
bce_loss_4 = nn.BCEWithLogitsLoss()(data[:, 3:4], targets[:, 4:5])
bce_loss_5 = nn.BCEWithLogitsLoss()(data[:, 4:5], targets[:, 5:6])
bce_loss_6 = nn.BCEWithLogitsLoss()(data[:, 5:6], targets[:, 6:7])
bce_loss_7 = nn.BCEWithLogitsLoss()(data[:, 6:7], targets[:, 7:8])
bce_loss_8 = nn.BCEWithLogitsLoss()(data[:, 7:8], targets[:, 8:9])
return bce_loss_1 + bce_loss_2 + bce_loss_3 + bce_loss_4 \
+ bce_loss_5 + bce_loss_6 + bce_loss_7 + bce_loss_8