手把手教你用 spacy3 训练中文NER

  • 开发
  • 12

文章目录

模型文件下载

https://github.com/explosion/spacy-models/releases?q=zh&expanded=true
在这里插入图片描述
简单测试一下ner效果，发现根本不能用
在这里插入图片描述

训练模型

准备数据

在这里插入图片描述

转化成 doc_bin 格式

from spacy.tokens import DocBin
from tqdm import tqdm
from spacy.util import filter_spans


nlp = spacy.blank('zh')   # 选择中文空白模型
doc_bin = DocBin()
for training_example in tqdm(TRAIN_DATA):
    text = training_example['text']
    labels = training_example['entities']
    doc = nlp.make_doc(text)
    ents = []
    for start, end, label in labels:
        span = doc.char_span(start, end, label=label, alignment_mode="contract")
        if span is None:
            print("Skipping entity")
        else:
            ents.append(span)
    filtered_ents = filter_spans(ents)
    doc.ents = filtered_ents
    doc_bin.add(doc)


doc_bin.to_disk("train.spacy")

train.spacy 和 dev.spacy 分别用来训练和测试

模型训练配置

生成初始配置

模型配置文件不用自己写，直接到官网上点击配置：https://spacy.io/usage/training#quickstart
在这里插入图片描述
通过简单勾选，得到一个初始配置文件 base_config.cfg

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = "zh_core_web_lg"
[system]
gpu_allocator = null

[nlp]
lang = "zh"
pipeline = ["tok2vec","ner"]
batch_size = 1000

[components]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[initialize]
vectors = ${paths.vectors}

补全完整配置

接下来，需要用命令 spacy init fill-config [初始配置] [完整配置] 把上述初始配置补全为完整的训练配置

python -m spacy init fill-config spacy/base_config.cfg spacy/config.cfg

得到 config.cfg 文件如下，其中做了一些人工改动，例如 paths.vectors 默认选的是 zh_core_web_lg，我改成了 zh_core_web_md

[paths]
train = null
dev = null
vectors = "zh_core_web_md"
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "zh"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
vectors = {"@vectors":"spacy.Vectors.v1"}

[nlp.tokenizer]
@tokenizers = "spacy.zh.ChineseTokenizer"
segmenter = "char"

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]
pkuseg_model = null
pkuseg_user_dict = "default"

开始训练

python -m spacy train spacy/config.cfg --output ./spacy/ --paths.train ./train.spacy --paths.dev ./dev.spacy

参数：

  • output 输出目录
  • paths.train 训练集文件
  • paths.dev 验证集文件

训练日志：

>>> python -m spacy train spacy/config.cfg --output ./spacy/ --paths.train ./train.spacy --paths.dev ./dev.spacy
ℹ Saving to output directory: spacy
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     49.29    0.00    0.00    0.00    0.00
  0     200        609.43   3515.82    6.61    7.99    5.63    0.07
  0     400       1104.85   3590.05   10.22   10.26   10.19    0.10
  0     600       1120.82   5038.80   16.23   17.95   14.81    0.16
  0     800       1071.70   5578.76   10.95   14.11    8.95    0.11
  0    1000       1151.26   6506.03   20.62   23.73   18.23    0.21
  0    1200       1100.93   6840.94   26.60   32.95   22.30    0.27
  0    1400       2058.58   7959.36   34.93   39.60   31.25    0.35
  0    1600       1642.29   9632.10   40.32   45.09   36.46    0.40
  1    1800       2580.55  11209.10   38.82   47.18   32.98    0.39
  1    2000       2907.86  13187.84   44.31   52.42   38.38    0.44
  1    2200       3575.63  15214.04   42.97   50.06   37.63    0.43
  2    2400       4790.03  18126.32   48.39   51.29   45.80    0.48
  2    2600       5653.69  17209.21   51.27   54.42   48.47    0.51

测试模型

nlp = spacy.load("spacy/model-best")
text = "我的名字是michal johnson,我的手机号是13425456344，我家住在东北松花江上8幢8单元6楼5号房。我叫王大，喜欢去旺角餐厅吃牛角包, 今年买了阿里巴巴的股票，我家住在新洲花园3栋4单元 8988-1室"
doc = nlp(text)

for ent in doc.ents:
    print({
            "start": ent.start,
            "end": ent.end,
            "text": ent.text,
            "entity_group": ent.label_,
        })

在这里插入图片描述

参考文献

  1. https://ubiai.tools/fine-tuning-spacy-models-customizing-named-entity-recognition/
  2. https://spacy.io/usage/training
  3. https://ner.pythonhumanities.com/03_02_train_spacy_ner_model.html
阅读全部

相关推荐

  7. AI-GC-手把手教你写一个小说推文生成器(对接生成大模型）

    2024-05-12 16:34:08       59 阅读

最近更新

  4. TCP协议是安全的吗?

    2024-05-12 16:34:08       18 阅读

  12. 阿里云服务器执行yum，一直下载docker-ce-stable失败

    2024-05-12 16:34:08       19 阅读

  13. 【Python教程】压缩PDF文件大小

    2024-05-12 16:34:08       19 阅读

  18. 通过文章id递归查询所有评论（xml）

    2024-05-12 16:34:08       20 阅读

热门阅读

  5. 递归求fabonacci数列 pta

    2024-05-12 16:34:08       11 阅读

  8. python爱心代码高级

    2024-05-12 16:34:08       7 阅读

  9. MongoDB聚合运算符：$trunc

    2024-05-12 16:34:08       9 阅读

  11. Python从0到POC编写--模块（三）

    2024-05-12 16:34:08       10 阅读

  12. 【Python 常用脚本及命令系列 1.3 -- 使用Python实现串口读写】

    2024-05-12 16:34:08       9 阅读

  13. [C++][数据结构]哈希3：unordered_map和unordered_set的模拟实现

    2024-05-12 16:34:08       8 阅读

  18. RustGUI学习（iced）之小部件（十一）：如何使用滚动条scrollable部件来进行滚动显示？

    2024-05-12 16:34:08       10 阅读

  19. edge 的使用心得与深度搜索

    2024-05-12 16:34:08       8 阅读

  24. 【Spring Boot】Spring Boot 的常用注解

    2024-05-12 16:34:08       9 阅读

  30. Vim常用快捷键

    2024-05-12 16:34:08       11 阅读