2 Star 7 Fork 1

tenlee/elasticsearch-analysis-hao

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

HAO ES 分词器

简介

一个elasticsearch 中文分词 插件。

QQ交流群:743457803

如何开发一个ES分词插件请参考 这里

主要参考了 IKHanLP

特性

  • 支持复杂汉字,有的汉字在java中长度不是1,比如𡃁,而IK等不支持。

  • 支持单字分词和搜索,而ik_max_word模式不支持。

  • 支持自定义长度分词,适合短文本下的人名等识别。

根据空格标点符号字母数字等分隔后的汉字文本长度<=autoWordLength会自动识别为一个词语。

  • 支持emoji搜索

  • 相比IK,比IK更智能,更准确。

    • 示例:比如IK ik_max_word是穷举所有可能词,导致搜索一些不相关的也会被搜到。 任性冲动过分词结果居然有任性 性冲动 动过,那么搜性冲动就会把这个doc搜索到。 南京市长江大桥,结果是南京市 市长 长江大桥,那么搜市长会把这个doc搜索到,而hao分词器不会,通过词频计算最短路,识别出可能性最高的词组。还可以根据自己场景,随意调节词频。
  • ik_smart 分词结果不是 ik_max_word 的子集,hao_search_mode 分词结果是 hao_index_mode 分词结果的子集

  • 相比HanLp,比HanLP更轻量,分词更可控,没有一些智能的人名等预测功能,可能会导致分词不稳定不准确,机器学习对于长短文本不同,预测分词结果也不同。并且HanLP也没有官方的ES插件。

  • 根据词频计算最短路,穷举出可能的词,而不是所有的词,如果穷举的词不对,可以调词频来纠正,词频文件是可读性更好txt文件

  • 支持元词,比如俄罗斯不会再拆分成罗斯罗斯是常用人名)。这样搜罗斯就不会把俄罗斯相关文档召回

  • 但是不支持词性

提供 Analyzer: hao_search_mode, hao_index_mode Tokenizer: hao_search_mode, hao_index_mode

Versions

Git tag ES version
master ES最新稳定版
v7.17.1 7.17.1
vX.Y.Z X.Y.Z

使用

安装

方式1. bin/elasticsearch-plugin install file:///Users/xiaoming/Download/analysis-hao.zip

方式2. 解压后,放在es plugins目录即可。

最后重启ES

ES 版本升级

如果没有你需要的对应ES版本,要修改一下几个地方:

  1. 修改pom.xml->elasticsearch.version的值为对应版本。
  2. 编译,按照响应报错修改代码,比如可能有HaoTokenizerFactory.java的构造方法。 最后执行 mvn clean package -Dmaven.test.skip=true,就可以得到插件的zip安装包。

分词器

下面是自定义分词器可用的配置项

参数


配置项参数 功能 默认值
enableIndexMode 是否使用index模式,index模式为细颗粒度。 hao_search_modefalsehao_index_modetrue,细颗粒度适合Term Query,粗颗粒度适合Phrase查询
enableFallBack 如果分词报错,是否启动最细粒度分词,即按字分。建议search_mode使用,不至于影响用户搜索。index_mode不启动,以便及时报错告警通知。 false不启动降级
enableFailDingMsg 是否启动失败钉钉通知,通知地址为HttpAnalyzer.cfg.xmldingWebHookUrl字段。 false
enableSingleWord 是否使用细粒度返回的单字。比如体力值,分词结果只存体力值,体力,而不存 false
autoWordLength 根据空格标点符号字母数字等分隔后的汉字文本长度小于autoWordLength会自动识别为一个词语。 默认-1不开启,>=2视为开启 -1

内置分词器介绍

  • hao_index_mode

会根据词库的词条和权重,递归分词,直到该词不可分。如果设置了enableSingleWord=true,会一直分到单字为止。

例如这段文本南京市长江大桥

  1. 南京市长江大桥 ==> 南京市, 长江大桥
  2. 南京市==>南京,, 长江大桥==>长江,大桥
  3. 如果enableSingleWord=false,递归停止,得到分词为南京市,南京,,长江大桥,长江,大桥
  4. 如果enableSingleWord=true,继续递归,直到单字位置,得到分词为南京市,南京,,,,长江大桥,长江,,,大桥,,
  • hao_search_mode

该模式下,相当于hao_index_mode模式只递归一次。 分词结果为南京市, 长江大桥。因为该模式下enableIndexMode=false,如果改成true,则和hao_index_mode一样的效果。

HaoAnalyzer.cfg.xml 配置


参数 功能 备注
baseDictionary 基础词库文件名 放在插件config目录或者es的config目录,不用更改
customerDictionaryFile 用户自定义远程词库文件,多个文件用英文分号;分隔 会存储在插件config目录或者es的config目录
remoteFreqDict 远程用户自定义词库文件 方便热更新,热更新通过下面两个参数定时更新。
syncDicTime 远程词库下次同步时间 hh:mm:ss 不填使用syncDicPeriodTime作为下次同步时间
syncDicPeriodTime 远程词库同步时间间隔,秒,最小值30 比如 syncDicTime=20:00:00,syncDicPeriodTime=86400,则是每天20点同步
dingWebHookUrl 钉钉机器人url 用于分词异常,同步词库异常/成功通知
dingMsgContent 机器人通知文案 注意配置钉钉机器人的时候关键词要和这个文案匹配,不然会消息发送失败

词库说明

优先读取 {ES_HOME}/config/analysis-hao/目录,没有读取 {ES_HOME}/plugins/analysis-hao/config目录下的文件

  • 基础词库 基础词库是base_dictionary.txt,以逗号分割,后面的数字表示词频。 例如:奋发图强 分词结果是 , 发图, , 是因为发图这个词的词频太高了(因为出现次数高),则可以降低词频,手动修改base_dictionary.txt文件就好了。
  • 远程词库 用户自定义词库会按照配置的时间和周期定期执行。 从远程词库更新完成后会自动覆盖现在的customerDictionaryFile。 远程词库的文件格式每行格式为 {词},{词频},{是否元词}, 例如俄罗斯,1000,1。 是否元词字段解释: 1代表是元词,不会再细拆分,俄罗斯不会再拆分成罗斯(罗斯是常用人名)。这样搜罗斯就不会把俄罗斯相关文档召回。 0就是可以继续细拆分,比如奋发图强
  • 远程词库是否重新加载是根据 http head 请求返回头中的两个字段是否有至少有一个发生变化,两个字段为:Last-Modified、ETag

示例索引demo

建索引:

PUT test/
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "search_analyzer": {
            "filter": [
              "lowercase"
            ],
            "char_filter": [
              "html_strip"
            ],
            "type": "custom",
            "tokenizer": "my_search_token"
          },
          "index_analyzer": {
            "char_filter": [
              "html_strip"
            ],
            "type": "custom",
            "tokenizer": "my_index_token"
          }
        },
        "tokenizer": {
          "my_index_token": {
            "enableFailDingMsg": "true",
            "type": "hao_index_mode",
            "enableSingleWord": "true",
            "enableFallBack": "true",
            "autoWordLength": 3
          },
          "my_search_token": {
            "enableFailDingMsg": "true",
            "type": "hao_search_mode",
            "enableSingleWord": "true",
            "enableFallBack": "true",
            "autoWordLength": 3
          }
        }
      },
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "index_options": "offsets",
        "analyzer": "index_analyzer",
        "search_analyzer": "search_analyzer"
      }
    }
  }
}

测试分词

test/_analyze
{
  "analyzer": "index_analyzer",
  "text": "徐庆年 奋发图强打篮球有利于提高人民生活,有的放矢,中华人民共和国家庭宣传委员会宣。🐶"
}

test/_analyze
{
  "analyzer": "search_analyzer",
  "text": "徐庆年 奋发图强打篮球有利于提高人民生活,有的放矢,中华人民共和国家庭宣传委员会宣。🐶"
}

徐庆年并不在词库中,但是通过autoWordLength识别为一个词。

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

可能是东半球hao用的elasticsearch(es)中文分词器插件 展开 收起
Java 等 2 种语言
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
Java
1
https://gitee.com/tenlee/elasticsearch-analysis-hao.git
git@gitee.com:tenlee/elasticsearch-analysis-hao.git
tenlee
elasticsearch-analysis-hao
elasticsearch-analysis-hao
main

搜索帮助

Cb406eda 1850385 E526c682 1850385