Elasticsearch提供了丰富的内置分析器，以满足不同场景下的文本分析需求。本文将详细介绍几个核心的内置分析器，包括它们的工作原理和使用示例，帮助开发者更好地理解如何在索引和搜索过程中应用这些分析器。
1. Standard Analyzer（标准分析器）
标准分析器是最常用的分析器之一，它基于Unicode文本分割算法切分单词，同时删除大部分标点符号，并将所有单词转为小写形式，以实现标准化处理。此外，标准分析器还支持过滤停用词列表中的词汇，进一步优化索引内容。
示例
POST _analyze
{
"analyzer": "standard",
"text": "The 2019头条新闻 has spread out。"
}
{
"tokens" : [
{
"token" : "the",
"start_offset" : 0,
"end_offset" : 3,
"type" : "<ALPHANUM>",
"position" : 0
},
{
"token" : "2019",
"start_offset" : 4,
"end_offset" : 8,
"type" : "<NUM>",
"position" : 1
},
{
"token" : "头",
"start_offset" : 8,
"end_offset" : 9,
"type" : "<IDEOGRAPHIC>",
"position" : 2
},
{
"token" : "条",
"start_offset" : 9,
"end_offset" : 10,
"type" : "<IDEOGRAPHIC>",
"position" : 3
},
{
"token" : "新",
"start_offset" : 10,
"end_offset" : 11,
"type" : "<IDEOGRAPHIC>",
"position" : 4
},
{
"token" : "闻",
"start_offset" : 11,
"end_offset" : 12,
"type" : "<IDEOGRAPHIC>",
"position" : 5
},
{
"token" : "has",
"start_offset" : 13,
"end_offset" : 16,
"type" : "<ALPHANUM>",
"position" : 6
},
{
"token" : "spread",
"start_offset" : 17,
"end_offset" : 23,
"type" : "<ALPHANUM>",
"position" : 7
},
{
"token" : "out",
"start_offset" : 24,
"end_offset" : 27,
"type" : "<ALPHANUM>",
"position" : 8
}
]
}
2. Simple Analyzer（简单分析器）
简单分析器在非字母字符处进行切分，并将所有字符转为小写。与标准分析器相比，它更加“简单粗暴”，不保留任何非字母字符，适用于那些需要快速分割且不关心标点符号和数字的场景。
示例
POST _analyze
{
"analyzer": "simple",
"text": "The 2019头条新闻 hasn’t spread out。"
}
{
"tokens" : [
{
"token" : "the",
"start_offset" : 0,
"end_offset" : 3,
"type" : "word",
"position" : 0
},
{
"token" : "头条新闻",
"start_offset" : 8,
"end_offset" : 12,
"type" : "word",
"position" : 1
},
{
"token" : "hasn",
"start_offset" : 13,
"end_offset" : 17,
"type" : "word",
"position" : 2
},
{
"token" : "t",
"start_offset" : 18,
"end_offset" : 19,
"type" : "word",
"position" : 3
},
{
"token" : "spread",
"start_offset" : 20,
"end_offset" : 26,
"type" : "word",
"position" : 4
},
{
"token" : "out",
"start_offset" : 27,
"end_offset" : 30,
"type" : "word",
"position" : 5
}
]
}
3. Whitespace Analyzer（空格分析器）
空格分析器以空格作为分隔符来切分文本，保留文本中所有的字符，包括数字、标点符号等，适合于需要精确控制分词边界的场景。
示例
POST _analyze
{
"analyzer": "whitespace",
"text": "The 2019头条新闻hasn’t spread out。"
}
{
"tokens" : [
{
"token" : "The",
"start_offset" : 0,
"end_offset" : 3,
"type" : "word",
"position" : 0
},
{
"token" : "2019头条新闻hasn’t",
"start_offset" : 4,
"end_offset" : 18,
"type" : "word",
"position" : 1
},
{
"token" : "spread",
"start_offset" : 19,
"end_offset" : 25,
"type" : "word",
"position" : 2
},
{
"token" : "out。",
"start_offset" : 26,
"end_offset" : 30,
"type" : "word",
"position" : 3
}
]
}
4. Stop Analyzer（停用词分析器）
停用词分析器在简单分析器的基础上增加了停用词过滤功能，移除了英语中最常见的非实质意义词汇（如“a”，“the”，“is”等），适用于希望进一步减少噪声、提高索引效率的情况。
示例
POST _analyze
{
"analyzer": "stop",
"text": "The quick brown fox jumps over the lazy dog."
}
{
"tokens" : [
{
"token" : "quick",
"start_offset" : 4,
"end_offset" : 9,
"type" : "word",
"position" : 1
},
{
"token" : "brown",
"start_offset" : 10,
"end_offset" : 15,
"type" : "word",
"position" : 2
},
{
"token" : "fox",
"start_offset" : 16,
"end_offset" : 19,
"type" : "word",
"position" : 3
},
{
"token" : "jumps",
"start_offset" : 20,
"end_offset" : 25,
"type" : "word",
"position" : 4
},
{
"token" : "over",
"start_offset" : 26,
"end_offset" : 30,
"type" : "word",
"position" : 5
},
{
"token" : "lazy",
"start_offset" : 35,
"end_offset" : 39,
"type" : "word",
"position" : 7
},
{
"token" : "dog",
"start_offset" : 40,
"end_offset" : 43,
"type" : "word",
"position" : 8
}
]
}
5. Keyword Analyzer（关键字分析器）
关键字分析器不对文本进行任何切分，而是将整个文本作为一个不可分割的整体（token）处理。适用于索引那些不应被拆分的字符串，如电子邮件地址、主机名等。
示例
POST _analyze
{
"analyzer": "keyword",
"text": "ContactUs@example.com"
}
{
"tokens" : [
{
"token" : "ContactUs@example.com",
"start_offset" : 0,
"end_offset" : 21,
"type" : "word",
"position" : 0
}
]
}
6. Pattern Analyzer（模式分析器）
模式分析器允许使用正则表达式来定义文本的分割规则，同时支持小写转换和停用词过滤，为文本分析提供了高度的定制性。
示例
POST _analyze
{
"analyzer": "pattern",
"text": "2023-04-01 Release Notes",
"pattern": "\\d{4}-\\d{2}-\\d{2} | [A-Za-z]+"
}
7. Language Analyzers（语言分析器）
Elasticsearch内置了一系列针对特定语言的分析器，如english、chinese等，它们能够根据相应语言的特点进行文本分析，包括正确处理词形变化、停用词等，适合多语言环境下的索引构建。
示例（中文）
POST _analyze
{
"analyzer": "ik_max_word",
"text": " ElasticSearch是一个开源的分布式全文搜索引擎。"
}
{
"tokens" : [
{
"token" : "elasticsearch",
"start_offset" : 1,
"end_offset" : 14,
"type" : "ENGLISH",
"position" : 0
},
{
"token" : "是",
"start_offset" : 14,
"end_offset" : 15,
"type" : "CN_CHAR",
"position" : 1
},
{
"token" : "一个",
"start_offset" : 15,
"end_offset" : 17,
"type" : "CN_WORD",
"position" : 2
},
{
"token" : "一",
"start_offset" : 15,
"end_offset" : 16,
"type" : "TYPE_CNUM",
"position" : 3
},
{
"token" : "个",
"start_offset" : 16,
"end_offset" : 17,
"type" : "COUNT",
"position" : 4
},
{
"token" : "开源",
"start_offset" : 17,
"end_offset" : 19,
"type" : "CN_WORD",
"position" : 5
},
{
"token" : "的",
"start_offset" : 19,
"end_offset" : 20,
"type" : "CN_CHAR",
"position" : 6
},
{
"token" : "分布式",
"start_offset" : 20,
"end_offset" : 23,
"type" : "CN_WORD",
"position" : 7
},
{
"token" : "分布",
"start_offset" : 20,
"end_offset" : 22,
"type" : "CN_WORD",
"position" : 8
},
{
"token" : "式",
"start_offset" : 22,
"end_offset" : 23,
"type" : "CN_CHAR",
"position" : 9
},
{
"token" : "全文",
"start_offset" : 23,
"end_offset" : 25,
"type" : "CN_WORD",
"position" : 10
},
{
"token" : "搜索引擎",
"start_offset" : 25,
"end_offset" : 29,
"type" : "CN_WORD",
"position" : 11
},
{
"token" : "搜索",
"start_offset" : 25,
"end_offset" : 27,
"type" : "CN_WORD",
"position" : 12
},
{
"token" : "索引",
"start_offset" : 26,
"end_offset" : 28,
"type" : "CN_WORD",
"position" : 13
},
{
"token" : "引擎",
"start_offset" : 27,
"end_offset" : 29,
"type" : "CN_WORD",
"position" : 14
}
]
}
8. Fingerprint Analyzer（指纹分析器）
指纹分析器通过一种特殊算法生成文本的“指纹”，它会将文本转为小写，删除扩展词和重复词，并将每个分词按字典序排序输出，常用于生成唯一标识或简化文本比较。
示例
POST _analyze
{
"analyzer": "fingerprint",
"text": "This is a test sentence with duplicate words."
}
{
"tokens" : [
{
"token" : "a duplicate is sentence test this with words",
"start_offset" : 0,
"end_offset" : 45,
"type" : "fingerprint",
"position" : 0
}
]
}
通过合理选择和配置这些内置分析器，开发者可以高效地处理各种文本数据，优化搜索性能和结果的相关性。
