本文将会介绍如何使用keras-bert实现文本多标签分类任务，其中对BERT进行微调。
项目结构
本项目的项目结构如下：
其中依赖的Python第三方模块如下：
pandas==0.23.4Keras==2.3.1keras_bert==0.83.0numpy==1.16.4
数据集介绍
本文采用的数据集与文章NLP（二十八）多标签文本分类中的一致，以事件抽取比赛的数据集为参考，形成文本与事件类型的多标签数据集，一共为65种事件类型。样例数据（csv格式）如下：
label,content司法行为-起诉|组织关系-裁员,最近，一位前便利蜂员工就因公司违规裁员，将便利蜂所在的公司虫极科技（北京）有限公司告上法庭。组织关系-裁员,思科上海大规模裁员人均可获赔100万官方澄清事实组织关系-裁员,日本巨头面临危机，已裁员1000多人，苹果也救不了它！组织关系-裁员|组织关系-解散,在硅谷镀金失败的造车新势力们：蔚来裁员、奇点被偷窃、拜腾解散
在label中，每个事件类型用|隔开。
在该数据集中，训练集一共11958个样本，测试集一共1498个样本。
模型训练
模型训练的脚本model_train.py的完整代码如下：
# -*- coding: utf-8 -*-import jsonimport codecsimport pandas as pdimport numpy as npfrom keras_bert import load_trained_model_from_checkpoint, Tokenizerfrom keras.layers import *from keras.models import Modelfrom keras.optimizers import Adam# 建议长度<=510maxlen = 256BATCH_SIZE = 8config_path = './chinese_L-12_H-768_A-12/bert_config.json'checkpoint_path = './chinese_L-12_H-768_A-12/bert_model.ckpt'dict_path = './chinese_L-12_H-768_A-12/vocab.txt'token_dict = {}with codecs.open(dict_path, 'r', 'utf-8') as reader:for line in reader:token = line.strip()token_dict[token] = len(token_dict)class OurTokenizer(Tokenizer):def _tokenize(self, text):R = []for c in text:if c in self._token_dict:R.append(c)else:R.append('[UNK]') # 剩余的字符是[UNK]return Rtokenizer = OurTokenizer(token_dict)def seq_padding(X, padding=0):L = [len(x) for x in X]ML = max(L)return np.array([np.concatenate([x, [padding] * (ML - len(x))]) if len(x) < ML else x for x in X])class DataGenerator:def __init__(self, data, batch_size=BATCH_SIZE):self.data = dataself.batch_size = batch_sizeself.steps = len(self.data) // self.batch_sizeif len(self.data) % self.batch_size != 0:self.steps += 1def __len__(self):return self.stepsdef __iter__(self):while True:idxs = list(range(len(self.data)))np.random.shuffle(idxs)X1, X2, Y = [], [], []for i in idxs:d = self.data[i]text = d[0][:maxlen]x1, x2 = tokenizer.encode(first=text)y = d[1]X1.append(x1)X2.append(x2)Y.append(y)if len(X1) == self.batch_size or i == idxs[-1]:X1 = seq_padding(X1)X2 = seq_padding(X2)Y = seq_padding(Y)yield [X1, X2], Y[X1, X2, Y] = [], [], []# 构建模型def create_cls_model(num_labels):bert_model = load_trained_model_from_checkpoint(config_path, checkpoint_path, seq_len=None)for layer in bert_model.layers:layer.trainable = Truex1_in = Input(shape=(None,))x2_in = Input(shape=(None,))x = bert_model([x1_in, x2_in])cls_layer = Lambda(lambda x: x[:, 0])(x) # 取出[CLS]对应的向量用来做分类p = Dense(num_labels, activation='sigmoid')(cls_layer)# 多分类model = Model([x1_in, x2_in], p)pile(loss='binary_crossentropy',optimizer=Adam(1e-5), # 用足够小的学习率metrics=['accuracy'])model.summary()return modelif __name__ == '__main__':# 数据处理, 读取训练集和测试集print("begin data processing...")train_df = pd.read_csv("data/train.csv").fillna(value="")test_df = pd.read_csv("data/test.csv").fillna(value="")select_labels = train_df["label"].unique()labels = []for label in select_labels:if "|" not in label:if label not in labels:labels.append(label)else:for _ in label.split("|"):if _ not in labels:labels.append(_)with open("label.json", "w", encoding="utf-8") as f:f.write(json.dumps(dict(zip(range(len(labels)), labels)), ensure_ascii=False, indent=2))train_data = []test_data = []for i in range(train_df.shape[0]):label, content = train_df.iloc[i, :]label_id = [0] * len(labels)for j, _ in enumerate(labels):for separate_label in label.split("|"):if _ == separate_label:label_id[j] = 1train_data.append((content, label_id))for i in range(test_df.shape[0]):label, content = test_df.iloc[i, :]label_id = [0] * len(labels)for j, _ in enumerate(labels):for separate_label in label.split("|"):if _ == separate_label:label_id[j] = 1test_data.append((content, label_id))# print(train_data[:10])print("finish data processing!")# 模型训练model = create_cls_model(len(labels))train_D = DataGenerator(train_data)test_D = DataGenerator(test_data)print("begin model training...")model.fit_generator(train_D.__iter__(),steps_per_epoch=len(train_D),epochs=10,validation_data=test_D.__iter__(),validation_steps=len(test_D))print("finish model training!")# 模型保存model.save('multi-label-ee.h5')print("Model saved!")result = model.evaluate_generator(test_D.__iter__(), steps=len(test_D))print("模型评估结果:", result)
模型结构如下：
__________________________________________________________________________________________________Layer (type)Output Shape Param #Connected to ==================================================================================================input_1 (InputLayer) (None, None) 0 __________________________________________________________________________________________________input_2 (InputLayer) (None, None) 0 __________________________________________________________________________________________________model_2 (Model) (None, None, 768) 101677056 input_1[0][0]input_2[0][0]__________________________________________________________________________________________________lambda_1 (Lambda)(None, 768)0 model_2[1][0]__________________________________________________________________________________________________dense_1 (Dense) (None, 65) 49985 lambda_1[0][0] ==================================================================================================Total params: 101,727,041Trainable params: 101,727,041Non-trainable params: 0__________________________________________________________________________________________________
从中我们可以发现，该模型结构与文章NLP（三十五）使用keras-bert实现文本多分类任务中给出的文本多分类模型结构大体一致，修改之处在于BERT后接的网络结构，所接的依然是dense层，但激活函数采用sigmoid函数，同时损失函数为binary_crossentropy。就其本质而言，该模型结构是对输出的65个结果采用0-1分类，故而激活函数采用sigmoid，这当然是文本多分类模型转化为多标签标签的最便捷方式，但不足之处在于，该模型并未考虑标签之间的依赖关系。
模型评估
模型评估脚本model_evaluate.py的完整代码如下：
# -*- coding: utf-8 -*-# @Time : /12/23 15:28# @Author : Jclian91# @File : model_evaluate.py# @Place : Yangpu, Shanghai# 模型评估脚本,利用hamming_loss作为多标签分类的评估指标，该值越小模型效果越好import jsonimport numpy as npimport pandas as pdfrom keras.models import load_modelfrom keras_bert import get_custom_objectsfrom sklearn.metrics import hamming_loss, classification_reportfrom model_train import token_dict, OurTokenizermaxlen = 256# 加载训练好的模型model = load_model("multi-label-ee.h5", custom_objects=get_custom_objects())tokenizer = OurTokenizer(token_dict)with open("label.json", "r", encoding="utf-8") as f:label_dict = json.loads(f.read())# 对单句话进行预测def predict_single_text(text):# 利用BERT进行tokenizetext = text[:maxlen]x1, x2 = tokenizer.encode(first=text)X1 = x1 + [0] * (maxlen - len(x1)) if len(x1) < maxlen else x1X2 = x2 + [0] * (maxlen - len(x2)) if len(x2) < maxlen else x2# 模型预测并输出预测结果prediction = model.predict([[X1], [X2]])one_hot = np.where(prediction > 0.5, 1, 0)[0]return one_hot, "|".join([label_dict[str(i)] for i in range(len(one_hot)) if one_hot[i]])# 模型评估def evaluate():test_df = pd.read_csv("data/test.csv").fillna(value="")true_y_list, pred_y_list = [], []true_label_list, pred_label_list = [], []common_cnt = 0for i in range(test_df.shape[0]):print("predict %d samples" % (i+1))true_label, content = test_df.iloc[i, :]true_y = [0] * len(label_dict.keys())for key, value in label_dict.items():if value in true_label:true_y[int(key)] = 1pred_y, pred_label = predict_single_text(content)if true_label == pred_label:common_cnt += 1true_y_list.append(true_y)pred_y_list.append(pred_y)true_label_list.append(true_label)pred_label_list.append(pred_label)# F1值print(classification_report(true_y_list, pred_y_list, digits=4))return true_label_list, pred_label_list, hamming_loss(true_y_list, pred_y_list), common_cnt/len(true_y_list)true_labels, pred_labels, h_loss, accuracy = evaluate()df = pd.DataFrame({"y_true": true_labels, "y_pred": pred_labels})df.to_csv("pred_result.csv")print("accuracy: ", accuracy)print("hamming loss: ", h_loss)
Hamming Loss为多标签分类所特有的评估方式，其值越小代表多标签分类模型的效果越好。运行上述模型评估代码，输出结果如下：
micro avg0.9341 0.9578 0.94581657macro avg0.9336 0.9462 0.93701657weighted avg0.9367 0.9578 0.94561657samples avg0.9520 0.9672 0.95311657accuracy: 0.8985313751668892hamming loss: 0.001869158878504673
在这里，笔者希望与之前的文章NLP（二十八）多标签文本分类中的模型对比一下。当时采用的模型为用ALBERT提取特征向量，再用Bi-GRU+Attention+FCN进行分类，模型结构如下：
对该模型同样采用上述评估办法，输出的结果如下：
micro avg0.9424 0.8292 0.88221657macro avg0.8983 0.7218 0.77911657weighted avg0.9308 0.8292 0.86691657samples avg0.8675 0.8496 0.85171657accuracy: 0.7983978638184246hamming loss: 0.0037691280681934887
可以发现，采用BERT微调的模型，在accuracy方面高出了约10%，各种F1值高出约5%-10%，Hamming Loss也小了很多。因此，BERT微调的模型比之前的模型效果好很多。
总结
本项目已经开源，Github地址为：/percent4/keras_bert_multi_label_cls 。
12月27日于上海浦东
