Compare commits
40 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a00336d239 | |||
| 32d874b1ba | |||
| 7b0d009d32 | |||
| e3695f9192 | |||
| 33ac83d619 | |||
| fa1451042b | |||
| c2e408473a | |||
| 712da9e7fc | |||
| 76a4499777 | |||
| e6023b1317 | |||
| 2b2f551f0e | |||
| 39a4fe434f | |||
| c73e99133a | |||
| 1dc338cbba | |||
| 76c5092fd9 | |||
| e0d799605b | |||
| ac74bac851 | |||
| c42c6c0739 | |||
| 4b692bcb13 | |||
| f4cf971956 | |||
| d1481b0cd7 | |||
| 2a61cd2469 | |||
| 98ee5afe64 | |||
| 720278edd1 | |||
| bac4768ed7 | |||
| 319f3a9f5e | |||
| 0aa7d7d0b1 | |||
| 93f58a6238 | |||
| cb6d72896d | |||
| 9403ef6dd7 | |||
| b83f833128 | |||
| 5c361fa330 | |||
| 304e978b91 | |||
| 67dc15cf74 | |||
| c61b0a1114 | |||
| c772277647 | |||
| 5a43f340f6 | |||
| 3e118d8073 | |||
| 9618da0a11 | |||
| 988caebe85 |
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Screenshots**
|
||||
If applicable, add screenshots to help explain your problem.
|
||||
|
||||
**Desktop (please complete the following information):**
|
||||
- OS: [e.g. iOS]
|
||||
- Browser [e.g. chrome, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Smartphone (please complete the following information):**
|
||||
- Device: [e.g. iPhone6]
|
||||
- OS: [e.g. iOS8.1]
|
||||
- Browser [e.g. stock browser, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
@@ -0,0 +1,3 @@
|
||||
language: java
|
||||
jdk:
|
||||
- oraclejdk8
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
10.You are not allowed to remove or modify any donation links/images in
|
||||
this project, otherwise we will reclaim all authorization.
|
||||
|
||||
11.You're free to use this software at your own/you company/your orgnization 's
|
||||
use. BUT You don't have the permission to sell the copy or the
|
||||
redistribution of this software to others in any form.
|
||||
|
||||
12.If you've forked this project, you must use the same lisence.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
Copyright 2018 Magese magese@live.cn
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,58 @@
|
||||
|
||||
## Solr-Cloud说明
|
||||
|
||||
##### 因为`Solr-Cloud`中的配置文件是交由`zookeeper`进行管理的, 所以为了方便更新动态词典, 所以也要将动态词典文件上传至`zookeeper`中,目录与`solr`的配置文件目录一致。
|
||||
|
||||
#### 注意:因为`zookeeper`中的配置文件大小不能超过`1m`,当词典列表过多时,需将词典文件切分成多个。
|
||||
|
||||
|
||||
1. 将jar包放入每台服务器的Solr服务的`Jetty`或`Tomcat`的`webapp/WEB-INF/lib/`目录下;
|
||||
|
||||
2. 将`resources`目录下的`IKAnalyzer.cfg.xml`、`ext.dic`、`stopword.dic`放入solr服务的`Jetty`或`Tomcat`的`webapp/WEB-INF/classes/`目录下;
|
||||
```console
|
||||
① IKAnalyzer.cfg.xml (IK默认的配置文件,用于配置自带的扩展词典及停用词典)
|
||||
② ext.dic (默认的扩展词典)
|
||||
③ stopword.dic (默认的停词词典)
|
||||
```
|
||||
注意:与单机版不同,`ik.conf`及`dynamicdic.txt`请不要放在`classes`目录下!
|
||||
|
||||
3. 将`resources`目录下的`ik.conf`及`dynamicdic.txt`放入solr配置文件夹中,与solr的`managed-schema`文件同目录中;
|
||||
```console
|
||||
① ik.conf (动态词典配置文件)
|
||||
files (动态词典列表,可以设置多个词典表,用逗号进行分隔,默认动态词典表为dynamicdic.txt)
|
||||
lastupdate (默认值为0,每次对动态词典表修改后请修改该值,必须大于上次的值,不然不会将词典表中新的词语添加到内存中。)
|
||||
② dynamicdic.txt (默认的动态词典,在此文件配置的词语不需重启服务即可加载进内存中。以#开头的词语视为注释,将不会加载到内存中。)
|
||||
```
|
||||
|
||||
4. 配置Solr的`managed-schema`,添加`ik分词器`,示例如下;
|
||||
```console
|
||||
<!-- ik分词器 -->
|
||||
<fieldType name="text_ik" class="solr.TextField">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
```
|
||||
|
||||
5. 将配置文件上传至`zookeeper`中,首次使用请重启服务或reload Collection。
|
||||
|
||||
6. 测试分词:
|
||||
* 此时的动态词典文件为空
|
||||

|
||||
* 配置文件lastupdate为0
|
||||

|
||||
* 测试分词
|
||||

|
||||
|
||||
7. 测试动态词典:
|
||||
* 增加动态词典词语并上传至`zookeeper`
|
||||

|
||||
* 修改配置文件并上传至`zookeeper`
|
||||

|
||||
* 测试分词
|
||||

|
||||
@@ -1,84 +1,122 @@
|
||||
# ik-analyzer-solr7
|
||||
ik-analyzer for solr7.x
|
||||
<p>IKAnalyzer的原作者为林良益(linliangyi2007@gmail.com),项目网站为http://code.google.com/p/ik-analyzer/</p>
|
||||
# ik-analyzer-solr
|
||||
ik-analyzer for solr 7.x-8.x
|
||||
|
||||
* 该项目根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改,其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
<!-- Badges section here. -->
|
||||
[](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
[](https://github.com/magese/ik-analyzer-solr/releases)
|
||||
[](./LICENSE)
|
||||
[](https://travis-ci.org/magese/ik-analyzer-solr)
|
||||
|
||||
<h4>适配最新版solr7,并添加动态加载字典表功能;</h4>
|
||||
<h4>在不需要重启solr服务的情况下加载新增的字典。</h4>
|
||||
[](https://github.com/magese/ik-analyzer-solr/network/members)
|
||||
[](https://github.com/magese/ik-analyzer-solr/stargazers)
|
||||
<!-- /Badges section end. -->
|
||||
|
||||
>更新说明
|
||||
>* 2018-10-10: 升级lucene版本为7.5.0
|
||||
>* 2018-09-03: 优化注释与输出信息,取消部分中文输出避免不同字符集乱码,现会打印被调用inform方法的hashcode
|
||||
>* 2018-08-23:
|
||||
<br> ⑴完善了动态更新词库代码注释;
|
||||
<br> ⑵将ik.conf配置文件中的lastUpdate属性改为long类型,现已支持时间戳形式
|
||||
>* 2018-08-13: 更新maven仓库地址
|
||||
>* 2018-08-01: 移除默认的扩展词与停用词
|
||||
>* 2018-07-23: 升级lucene版本为7.4.0
|
||||
## 简介
|
||||
#### 适配最新版本solr 7&8;
|
||||
#### 扩展IK原有词库:
|
||||
| 分词工具 | 词库中词的数量 | 最后更新时间 |
|
||||
| :------: | :------: | :------: |
|
||||
| ik | 27.5万 | 2012年 |
|
||||
| mmseg | 15.7万 | 2017年 |
|
||||
| word | 64.2万 | 2014年 |
|
||||
| jieba | 58.4万 | 2012年 |
|
||||
| jcesg | 16.6万 | 2018年 |
|
||||
| sougou词库 | 115.2万 | 2018年 |
|
||||
#### 将以上词库进行整理后约188.5万条词汇;
|
||||
#### 添加动态加载词典表功能,在不需要重启solr服务的情况下加载新增的词典。
|
||||
* IKAnalyzer的原作者为林良益<linliangyi2007@gmail.com>,项目网站为<http://code.google.com/p/ik-analyzer>
|
||||
* 该项目动态加载功能根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改,其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
|
||||
<hr>
|
||||
<H2>使用说明:</H2><BR>
|
||||
|
||||
* jar包下载地址:[ik-analyzer-7.5.0.jar](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/7.5.0/ik-analyzer-7.5.0.jar)
|
||||
* 历史版本:[Central Repository](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
## 使用说明
|
||||
* jar包下载地址:[](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/8.0.0/ik-analyzer-8.0.0.jar)
|
||||
* 历史版本:[](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
|
||||
<pre>
|
||||
<!-- Maven仓库地址 -->
|
||||
<dependency>
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>7.5.0</version>
|
||||
</dependency>
|
||||
</pre>
|
||||
<ul>
|
||||
<li>
|
||||
<p>1. 将jar包放入solr服务的jetty或tomcat的webapp/WEB-INF/lib/目录下;</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>2. 将resources目录下的5个配置文件放入solr服务的jetty或tomcat的webapp/WEB-INF/classes/目录下;</p>
|
||||
<pre>
|
||||
①IKAnalyzer.cfg.xml
|
||||
②ext.dic
|
||||
③stopword.dic
|
||||
④ik.conf
|
||||
⑤dynamicdic.txt
|
||||
</pre>
|
||||
</li>
|
||||
<li>
|
||||
<p>3. 配置solr的managed-schema,添加ik分词器,示例如下;</p>
|
||||
<pre>
|
||||
<!-- ik分词器 -->
|
||||
<fieldType name="text_ik" class="solr.TextField">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
</pre>
|
||||
</li>
|
||||
<li>
|
||||
<p>4. 启动solr服务测试分词;</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>5. ik.conf文件说明:</p>
|
||||
<pre>
|
||||
files=dynamicdic.txt
|
||||
lastupdate=0
|
||||
</pre>
|
||||
<p>files为动态字典列表,可以设置多个字典表,用逗号进行分隔,默认动态字典表为dynamicdic.txt;</p>
|
||||
<p>lastupdate默认值为0,每次对动态字典表修改后请+1,不然不会将字典表中新的词语添加到内存中。<s>lastupdate采用的是int类型,不支持时间戳,如果使用时间戳的朋友可以把源码中的int改成long即可;</s></p>
|
||||
<p>2018-08-23 已将源码中lastUpdate改为long类型,现可以用时间戳了。</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>5-dynamicdic.txt 为动态字典,在此文件配置的词语不需重启服务即可加载进内存中;</p>
|
||||
</li>
|
||||
</ul>
|
||||
<hr>
|
||||
```console
|
||||
<!-- Maven仓库地址 -->
|
||||
<dependency>
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>8.0.0</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
<p>有问题可以联系作者邮箱magese@live.cn;</p>
|
||||
<p>欢迎大家一起交流~</p>
|
||||
### Solr-Cloud
|
||||
* [Solr-Cloud说明](./README-CLOUD.md)
|
||||
|
||||
### 单机版Solr
|
||||
1. 将jar包放入Solr服务的`Jetty`或`Tomcat`的`webapp/WEB-INF/lib/`目录下;
|
||||
|
||||
2. 将`resources`目录下的5个配置文件放入solr服务的`Jetty`或`Tomcat`的`webapp/WEB-INF/classes/`目录下;
|
||||
```console
|
||||
① IKAnalyzer.cfg.xml
|
||||
② ext.dic
|
||||
③ stopword.dic
|
||||
④ ik.conf
|
||||
⑤ dynamicdic.txt
|
||||
```
|
||||
|
||||
3. 配置Solr的`managed-schema`,添加`ik分词器`,示例如下;
|
||||
```console
|
||||
<!-- ik分词器 -->
|
||||
<fieldType name="text_ik" class="solr.TextField">
|
||||
<analyzer type="index">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="false" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
<analyzer type="query">
|
||||
<tokenizer class="org.wltea.analyzer.lucene.IKTokenizerFactory" useSmart="true" conf="ik.conf"/>
|
||||
<filter class="solr.LowerCaseFilterFactory"/>
|
||||
</analyzer>
|
||||
</fieldType>
|
||||
```
|
||||
|
||||
4. 启动Solr服务测试分词;
|
||||
|
||||

|
||||
|
||||
5. `ik.conf`文件说明:
|
||||
```console
|
||||
files=dynamicdic.txt
|
||||
lastupdate=0
|
||||
```
|
||||
|
||||
1. `files`为动态词典列表,可以设置多个词典表,用逗号进行分隔,默认动态词典表为`dynamicdic.txt`;
|
||||
2. `lastupdate`默认值为`0`,每次对动态词典表修改后请+1,不然不会将词典表中新的词语添加到内存中。<s>`lastupdate`采用的是`int`类型,不支持时间戳,如果使用时间戳的朋友可以把源码中的`int`改成`long`即可;</s> `2018-08-23` 已将源码中`lastUpdate`改为`long`类型,现可以用时间戳了。
|
||||
|
||||
6. `dynamicdic.txt` 为动态词典
|
||||
|
||||
在此文件配置的词语不需重启服务即可加载进内存中。
|
||||
以`#`开头的词语视为注释,将不会加载到内存中。
|
||||
|
||||
|
||||
## 更新说明
|
||||
- `2019-05-15:` 升级lucene版本为`8.0.0`,并支持Solr8使用
|
||||
- `2019-03-01:` 升级lucene版本为`7.7.1`
|
||||
- `2019-02-15:` 升级lucene版本为`7.7.0`
|
||||
- `2018-12-26:`
|
||||
- 升级lucene版本为`7.6.0`
|
||||
- 兼容solr-cloud,动态词典配置文件及动态词典可交由`zookeeper`进行管理
|
||||
- 动态词典增加注释功能,以`#`开头的行将视为注释
|
||||
- `2018-12-04:` 整理更新词库列表`magese.dic`
|
||||
- `2018-10-10:` 升级lucene版本为`7.5.0`
|
||||
- `2018-09-03:` 优化注释与输出信息,取消部分中文输出避免不同字符集乱码,现会打印被调用inform方法的hashcode
|
||||
- `2018-08-23: `
|
||||
- 完善了动态更新词库代码注释;
|
||||
- 将ik.conf配置文件中的lastUpdate属性改为long类型,现已支持时间戳形式
|
||||
- `2018-08-13:` 更新maven仓库地址
|
||||
- `2018-08-01:` 移除默认的扩展词与停用词
|
||||
- `2018-07-23:` 升级lucene版本为`7.4.0`
|
||||
|
||||
|
||||
## 感谢 Thanks
|
||||
|
||||
[](https://www.jetbrains.com/?from=ik-analyzer-solr)
|
||||
|
||||
[](https://www.java.com)
|
||||
|
||||
|
||||
## BUG & 疑问 & 其它
|
||||
如果您在使用过程中遇到了BUG,或者有不清楚的地方,请挂ISSUE或者联系作者:<magese@live.cn>
|
||||
|
||||
如果您觉得该项目对您有帮助,请别忘记给这个项目一个`star`
|
||||
|
||||
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 93 KiB |
|
After Width: | Height: | Size: 84 KiB |
|
After Width: | Height: | Size: 67 KiB |
|
After Width: | Height: | Size: 59 KiB |
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 67 KiB |
@@ -0,0 +1,66 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!-- Generator: Adobe Illustrator 19.1.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
|
||||
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
|
||||
width="120.1px" height="130.2px" viewBox="0 0 120.1 130.2" style="enable-background:new 0 0 120.1 130.2;" xml:space="preserve"
|
||||
>
|
||||
<g>
|
||||
<linearGradient id="XMLID_2_" gradientUnits="userSpaceOnUse" x1="31.8412" y1="120.5578" x2="110.2402" y2="73.24">
|
||||
<stop offset="0" style="stop-color:#FCEE39"/>
|
||||
<stop offset="1" style="stop-color:#F37B3D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3041_" style="fill:url(#XMLID_2_);" d="M118.6,71.8c0.9-0.8,1.4-1.9,1.5-3.2c0.1-2.6-1.8-4.7-4.4-4.9
|
||||
c-1.2-0.1-2.4,0.4-3.3,1.1l0,0l-83.8,45.9c-1.9,0.8-3.6,2.2-4.7,4.1c-2.9,4.8-1.3,11,3.6,13.9c3.4,2,7.5,1.8,10.7-0.2l0,0l0,0
|
||||
c0.2-0.2,0.5-0.3,0.7-0.5l78-54.8C117.3,72.9,118.4,72.1,118.6,71.8L118.6,71.8L118.6,71.8z"/>
|
||||
<linearGradient id="XMLID_3_" gradientUnits="userSpaceOnUse" x1="48.3607" y1="6.9083" x2="119.9179" y2="69.5546">
|
||||
<stop offset="0" style="stop-color:#EF5A6B"/>
|
||||
<stop offset="0.57" style="stop-color:#F26F4E"/>
|
||||
<stop offset="1" style="stop-color:#F37B3D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3049_" style="fill:url(#XMLID_3_);" d="M118.8,65.1L118.8,65.1L55,2.5C53.6,1,51.6,0,49.3,0
|
||||
c-4.3,0-7.7,3.5-7.7,7.7v0c0,2.1,0.8,3.9,2.1,5.3l0,0l0,0c0.4,0.4,0.8,0.7,1.2,1l67.4,57.7l0,0c0.8,0.7,1.8,1.2,3,1.3
|
||||
c2.6,0.1,4.7-1.8,4.9-4.4C120.2,67.3,119.7,66,118.8,65.1z"/>
|
||||
<linearGradient id="XMLID_4_" gradientUnits="userSpaceOnUse" x1="52.9467" y1="63.6407" x2="10.5379" y2="37.1562">
|
||||
<stop offset="0" style="stop-color:#7C59A4"/>
|
||||
<stop offset="0.3852" style="stop-color:#AF4C92"/>
|
||||
<stop offset="0.7654" style="stop-color:#DC4183"/>
|
||||
<stop offset="0.957" style="stop-color:#ED3D7D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3042_" style="fill:url(#XMLID_4_);" d="M57.1,59.5C57,59.5,17.7,28.5,16.9,28l0,0l0,0c-0.6-0.3-1.2-0.6-1.8-0.9
|
||||
c-5.8-2.2-12.2,0.8-14.4,6.6c-1.9,5.1,0.2,10.7,4.6,13.4l0,0l0,0C6,47.5,6.6,47.8,7.3,48c0.4,0.2,45.4,18.8,45.4,18.8l0,0
|
||||
c1.8,0.8,3.9,0.3,5.1-1.2C59.3,63.7,59,61,57.1,59.5z"/>
|
||||
<linearGradient id="XMLID_5_" gradientUnits="userSpaceOnUse" x1="52.1736" y1="3.7019" x2="10.7706" y2="37.8971">
|
||||
<stop offset="0" style="stop-color:#EF5A6B"/>
|
||||
<stop offset="0.364" style="stop-color:#EE4E72"/>
|
||||
<stop offset="1" style="stop-color:#ED3D7D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3057_" style="fill:url(#XMLID_5_);" d="M49.3,0c-1.7,0-3.3,0.6-4.6,1.5L4.9,28.3c-0.1,0.1-0.2,0.1-0.2,0.2l-0.1,0
|
||||
l0,0c-1.7,1.2-3.1,3-3.9,5.1C-1.5,39.4,1.5,45.9,7.3,48c3.6,1.4,7.5,0.7,10.4-1.4l0,0l0,0c0.7-0.5,1.3-1,1.8-1.6l34.6-31.2l0,0
|
||||
c1.8-1.4,3-3.6,3-6.1v0C57.1,3.5,53.6,0,49.3,0z"/>
|
||||
<g id="XMLID_3008_">
|
||||
<rect id="XMLID_3033_" x="34.6" y="37.4" style="fill:#000000;" width="51" height="51"/>
|
||||
<rect id="XMLID_3032_" x="39" y="78.8" style="fill:#FFFFFF;" width="19.1" height="3.2"/>
|
||||
<g id="XMLID_3009_">
|
||||
<path id="XMLID_3030_" style="fill:#FFFFFF;" d="M38.8,50.8l1.5-1.4c0.4,0.5,0.8,0.8,1.3,0.8c0.6,0,0.9-0.4,0.9-1.2l0-5.3l2.3,0
|
||||
l0,5.3c0,1-0.3,1.8-0.8,2.3c-0.5,0.5-1.3,0.8-2.3,0.8C40.2,52.2,39.4,51.6,38.8,50.8z"/>
|
||||
<path id="XMLID_3028_" style="fill:#FFFFFF;" d="M45.3,43.8l6.7,0v1.9l-4.4,0V47l4,0l0,1.8l-4,0l0,1.3l4.5,0l0,2l-6.7,0
|
||||
L45.3,43.8z"/>
|
||||
<path id="XMLID_3026_" style="fill:#FFFFFF;" d="M55,45.8l-2.5,0l0-2l7.3,0l0,2l-2.5,0l0,6.3l-2.3,0L55,45.8z"/>
|
||||
<path id="XMLID_3022_" style="fill:#FFFFFF;" d="M39,54l4.3,0c1,0,1.8,0.3,2.3,0.7c0.3,0.3,0.5,0.8,0.5,1.4v0
|
||||
c0,1-0.5,1.5-1.3,1.9c1,0.3,1.6,0.9,1.6,2v0c0,1.4-1.2,2.3-3.1,2.3l-4.3,0L39,54z M43.8,56.6c0-0.5-0.4-0.7-1-0.7l-1.5,0l0,1.5
|
||||
l1.4,0C43.4,57.3,43.8,57.1,43.8,56.6L43.8,56.6z M43,59l-1.8,0l0,1.5H43c0.7,0,1.1-0.3,1.1-0.8v0C44.1,59.2,43.7,59,43,59z"/>
|
||||
<path id="XMLID_3019_" style="fill:#FFFFFF;" d="M46.8,54l3.9,0c1.3,0,2.1,0.3,2.7,0.9c0.5,0.5,0.7,1.1,0.7,1.9v0
|
||||
c0,1.3-0.7,2.1-1.7,2.6l2,2.9l-2.6,0l-1.7-2.5h-1l0,2.5l-2.3,0L46.8,54z M50.6,58c0.8,0,1.2-0.4,1.2-1v0c0-0.7-0.5-1-1.2-1
|
||||
l-1.5,0v2H50.6z"/>
|
||||
<path id="XMLID_3016_" style="fill:#FFFFFF;" d="M56.8,54l2.2,0l3.5,8.4l-2.5,0l-0.6-1.5l-3.2,0l-0.6,1.5l-2.4,0L56.8,54z
|
||||
M58.8,59l-0.9-2.3L57,59L58.8,59z"/>
|
||||
<path id="XMLID_3014_" style="fill:#FFFFFF;" d="M62.8,54l2.3,0l0,8.3l-2.3,0L62.8,54z"/>
|
||||
<path id="XMLID_3012_" style="fill:#FFFFFF;" d="M65.7,54l2.1,0l3.4,4.4l0-4.4l2.3,0l0,8.3l-2,0L68,57.8l0,4.6l-2.3,0L65.7,54z"
|
||||
/>
|
||||
<path id="XMLID_3010_" style="fill:#FFFFFF;" d="M73.7,61.1l1.3-1.5c0.8,0.7,1.7,1,2.7,1c0.6,0,1-0.2,1-0.6v0
|
||||
c0-0.4-0.3-0.5-1.4-0.8c-1.8-0.4-3.1-0.9-3.1-2.6v0c0-1.5,1.2-2.7,3.2-2.7c1.4,0,2.5,0.4,3.4,1.1l-1.2,1.6
|
||||
c-0.8-0.5-1.6-0.8-2.3-0.8c-0.6,0-0.8,0.2-0.8,0.5v0c0,0.4,0.3,0.5,1.4,0.8c1.9,0.4,3.1,1,3.1,2.6v0c0,1.7-1.3,2.7-3.4,2.7
|
||||
C76.1,62.5,74.7,62,73.7,61.1z"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.8 KiB |
@@ -4,16 +4,16 @@
|
||||
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>7.5.0</version>
|
||||
<version>8.0.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<name>ik-analyzer-solr7</name>
|
||||
<name>ik-analyzer-solr</name>
|
||||
<url>http://code.google.com/p/ik-analyzer/</url>
|
||||
<description>IK-Analyzer for solr7.5</description>
|
||||
<description>IK-Analyzer for solr 7-8</description>
|
||||
|
||||
<properties>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<lucene.version>7.5.0</lucene.version>
|
||||
<lucene.version>8.0.0</lucene.version>
|
||||
<javac.src.version>1.8</javac.src.version>
|
||||
<javac.target.version>1.8</javac.target.version>
|
||||
<maven.compiler.plugin.version>3.3</maven.compiler.plugin.version>
|
||||
@@ -55,9 +55,9 @@
|
||||
</licenses>
|
||||
<scm>
|
||||
<tag>master</tag>
|
||||
<url>https://github.com/magese/ik-analyzer-solr7</url>
|
||||
<connection>scm:git:git@github.com:magese/ik-analyzer-solr7.git</connection>
|
||||
<developerConnection>scm:git:git@github.com:magese/ik-analyzer-solr7.git</developerConnection>
|
||||
<url>https://github.com/magese/ik-analyzer-solr</url>
|
||||
<connection>scm:git:git@github.com:magese/ik-analyzer-solr.git</connection>
|
||||
<developerConnection>scm:git:git@github.com:magese/ik-analyzer-solr.git</developerConnection>
|
||||
</scm>
|
||||
<developers>
|
||||
<developer>
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
|
||||
@@ -19,16 +41,16 @@ public class DefaultConfig implements Configuration {
|
||||
/*
|
||||
* 分词器默认字典路径
|
||||
*/
|
||||
private static final String PATH_DIC_MAIN = "org/wltea/analyzer/dic/main2012.dic";
|
||||
private static final String PATH_DIC_QUANTIFIER = "org/wltea/analyzer/dic/quantifier.dic";
|
||||
private static final String PATH_DIC_MAIN = "dict/magese.dic";
|
||||
private static final String PATH_DIC_QUANTIFIER = "dict/quantifier.dic";
|
||||
|
||||
/*
|
||||
* 分词器配置文件路径
|
||||
*/
|
||||
private static final String FILE_NAME = "IKAnalyzer.cfg.xml";
|
||||
//配置属性——扩展字典
|
||||
// 配置属性——扩展字典
|
||||
private static final String EXT_DICT = "ext_dict";
|
||||
//配置属性——扩展停止词典
|
||||
// 配置属性——扩展停止词典
|
||||
private static final String EXT_STOP = "ext_stopwords";
|
||||
|
||||
private Properties props;
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -241,7 +263,7 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
*
|
||||
*/
|
||||
public String toString(){
|
||||
return String.valueOf(this.getBeginPosition()) + "-" + this.getEndPosition() +
|
||||
return this.getBeginPosition() + "-" + this.getEndPosition() +
|
||||
" : " + this.lexemeText + " : \t" +
|
||||
this.getLexemeTypeString();
|
||||
}
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
|
||||
@@ -1,14 +1,33 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.io.*;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
@@ -91,32 +110,30 @@ public class Dictionary {
|
||||
* 重新更新词典
|
||||
* 由于停用词等不经常变也不建议常增加,故这里只修改动态扩展词库
|
||||
*
|
||||
* @param inputStreamList 词典文件IO流集合
|
||||
* @param inputStreamReaderList 词典文件IO流集合
|
||||
*/
|
||||
public static void reloadDic(List<InputStream> inputStreamList) {
|
||||
public static void reloadDic(List<Reader> inputStreamReaderList) {
|
||||
// 如果本类单例尚未实例化,则先进行初始化操作
|
||||
if (singleton == null) {
|
||||
Configuration cfg = DefaultConfig.getInstance();
|
||||
initial(cfg);
|
||||
}
|
||||
// 对词典流集合进行循环读取,将读取到的词语加载到主词典中
|
||||
for (InputStream is : inputStreamList) {
|
||||
for (Reader in : inputStreamReaderList) {
|
||||
try {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
LineNumberReader br = new LineNumberReader(in);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
singleton._MainDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
while ((theWord = br.readLine()) != null) {
|
||||
if (theWord.trim().length() == 0 || theWord.trim().charAt(0) == '#') continue;
|
||||
singleton._MainDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Other Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
} finally {
|
||||
try {
|
||||
if (is != null) {
|
||||
is.close();
|
||||
if (in != null) {
|
||||
in.close();
|
||||
}
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
@@ -35,7 +57,7 @@ public final class IKTokenizer extends Tokenizer {
|
||||
private int endPosition;
|
||||
|
||||
/**
|
||||
* Lucene 7.4 Tokenizer适配器类构造函数
|
||||
* Lucene 7.6 Tokenizer适配器类构造函数
|
||||
*/
|
||||
public IKTokenizer() {
|
||||
this(false);
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
@@ -14,6 +36,11 @@ import org.wltea.analyzer.dic.Dictionary;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.io.Reader;
|
||||
import java.nio.charset.CharsetDecoder;
|
||||
import java.nio.charset.CodingErrorAction;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
@@ -28,7 +55,9 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
public IKTokenizerFactory(Map<String, String> args) {
|
||||
super(args);
|
||||
String useSmartArg = args.get("useSmart");
|
||||
String confArg = args.get("conf");
|
||||
this.setUseSmart(Boolean.parseBoolean(useSmartArg));
|
||||
this.setConf(confArg);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -45,10 +74,10 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
*/
|
||||
@Override
|
||||
public void inform(ResourceLoader resourceLoader) throws IOException {
|
||||
System.out.println(String.format("IKTokenizerFactory "+ this.hashCode() +" inform conf: %s", this.conf));
|
||||
System.out.println(String.format("IKTokenizerFactory " + this.hashCode() + " inform conf: %s", getConf()));
|
||||
this.loader = resourceLoader;
|
||||
update();
|
||||
if ((this.conf != null) && (!this.conf.trim().isEmpty())) {
|
||||
if ((getConf() != null) && (!getConf().trim().isEmpty())) {
|
||||
UpdateThread.getInstance().register(this);
|
||||
}
|
||||
}
|
||||
@@ -60,24 +89,27 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
*/
|
||||
@Override
|
||||
public void update() throws IOException {
|
||||
// 默认UTF-8解码
|
||||
CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
|
||||
.onMalformedInput(CodingErrorAction.REPORT)
|
||||
.onUnmappableCharacter(CodingErrorAction.REPORT);
|
||||
|
||||
// 获取ik.conf配置文件信息
|
||||
Properties p = canUpdate();
|
||||
if (p != null) {
|
||||
// 获取词典表名称集合
|
||||
List<String> dicPaths = SplitFileNames(p.getProperty("files"));
|
||||
// 获取词典文件的IO流
|
||||
List<InputStream> inputStreamList = new ArrayList<>();
|
||||
List<Reader> inputStreamReaderList = new ArrayList<>();
|
||||
for (String path : dicPaths) {
|
||||
if ((path != null) && (!path.isEmpty())) {
|
||||
InputStream is = this.loader.openResource(path);
|
||||
if (is != null) {
|
||||
inputStreamList.add(is);
|
||||
}
|
||||
Reader isr = new InputStreamReader(loader.openResource(path), decoder);
|
||||
inputStreamReaderList.add(isr);
|
||||
}
|
||||
}
|
||||
// 如果IO流集合不为空则执行加载词典
|
||||
if (!inputStreamList.isEmpty())
|
||||
Dictionary.reloadDic(inputStreamList);
|
||||
if (!inputStreamReaderList.isEmpty())
|
||||
Dictionary.reloadDic(inputStreamReaderList);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,10 +118,10 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
*/
|
||||
private Properties canUpdate() {
|
||||
try {
|
||||
if (this.conf == null)
|
||||
if (getConf() == null)
|
||||
return null;
|
||||
Properties p = new Properties();
|
||||
InputStream confStream = this.loader.openResource(this.conf); // 获取配置文件流
|
||||
InputStream confStream = this.loader.openResource(getConf()); // 获取配置文件流
|
||||
p.load(confStream); // 读取配置文件
|
||||
confStream.close(); // 关闭文件流
|
||||
String lastupdate = p.getProperty("lastupdate", "0"); // 获取最后更新数字
|
||||
@@ -100,13 +132,13 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
String paths = p.getProperty("files"); // 获取词典文件名
|
||||
if ((paths == null) || (paths.trim().isEmpty()))
|
||||
return null;
|
||||
System.out.println("loading ik.conf files success.");
|
||||
System.out.println("loading " + getConf() + " files success.");
|
||||
return p;
|
||||
}
|
||||
this.lastUpdateTime = t;
|
||||
return null;
|
||||
} catch (Exception e) {
|
||||
System.err.println("parsing ik.conf NullPointerException!!!" + Arrays.toString(e.getStackTrace()));
|
||||
System.err.println("parsing " + getConf() + " NullPointerException!!!" + Arrays.toString(e.getStackTrace()));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -126,6 +158,7 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
return result;
|
||||
}
|
||||
|
||||
/* getter & setter */
|
||||
private boolean useSmart() {
|
||||
return useSmart;
|
||||
}
|
||||
@@ -133,4 +166,12 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
private void setUseSmart(boolean useSmart) {
|
||||
this.useSmart = useSmart;
|
||||
}
|
||||
|
||||
private String getConf() {
|
||||
return conf;
|
||||
}
|
||||
|
||||
private void setConf(String conf) {
|
||||
this.conf = conf;
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
|
||||
@@ -1,7 +1,29 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* update by Magese(magese@live.cn)
|
||||
* IK 中文分词 版本 8.0.0
|
||||
* IK Analyzer release 8.0.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012,乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.0.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.0.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
@@ -17,10 +39,7 @@ import org.apache.lucene.index.IndexWriterConfig;
|
||||
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
|
||||
import org.apache.lucene.queryparser.classic.ParseException;
|
||||
import org.apache.lucene.queryparser.classic.QueryParser;
|
||||
import org.apache.lucene.search.IndexSearcher;
|
||||
import org.apache.lucene.search.Query;
|
||||
import org.apache.lucene.search.ScoreDoc;
|
||||
import org.apache.lucene.search.TopDocs;
|
||||
import org.apache.lucene.search.*;
|
||||
import org.apache.lucene.store.Directory;
|
||||
import org.apache.lucene.store.RAMDirectory;
|
||||
import org.wltea.analyzer.lucene.IKAnalyzer;
|
||||
@@ -85,10 +104,11 @@ public class LuceneIndexAndSearchDemo {
|
||||
|
||||
//搜索相似度最高的5条记录
|
||||
TopDocs topDocs = isearcher.search(query, 5);
|
||||
System.out.println("命中:" + topDocs.totalHits);
|
||||
long totalHits = topDocs.totalHits.value;
|
||||
System.out.println("命中:" + totalHits);
|
||||
//输出结果
|
||||
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
|
||||
for (int i = 0; i < topDocs.totalHits; i++) {
|
||||
for (int i = 0; i < totalHits; i++) {
|
||||
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
|
||||
System.out.println("内容:" + targetDoc.toString());
|
||||
}
|
||||
|
||||
@@ -1,316 +0,0 @@
|
||||
丈
|
||||
下
|
||||
世
|
||||
世纪
|
||||
两
|
||||
个
|
||||
中
|
||||
串
|
||||
亩
|
||||
人
|
||||
介
|
||||
付
|
||||
代
|
||||
件
|
||||
任
|
||||
份
|
||||
伏
|
||||
伙
|
||||
位
|
||||
位数
|
||||
例
|
||||
倍
|
||||
像素
|
||||
元
|
||||
克
|
||||
克拉
|
||||
公亩
|
||||
公克
|
||||
公分
|
||||
公升
|
||||
公尺
|
||||
公担
|
||||
公斤
|
||||
公里
|
||||
公顷
|
||||
具
|
||||
册
|
||||
出
|
||||
刀
|
||||
分
|
||||
分钟
|
||||
分米
|
||||
划
|
||||
列
|
||||
则
|
||||
刻
|
||||
剂
|
||||
剑
|
||||
副
|
||||
加仑
|
||||
勺
|
||||
包
|
||||
匙
|
||||
匹
|
||||
区
|
||||
千克
|
||||
千米
|
||||
升
|
||||
卷
|
||||
厅
|
||||
厘
|
||||
厘米
|
||||
双
|
||||
发
|
||||
口
|
||||
句
|
||||
只
|
||||
台
|
||||
叶
|
||||
号
|
||||
名
|
||||
吨
|
||||
听
|
||||
员
|
||||
周
|
||||
周年
|
||||
品
|
||||
回
|
||||
团
|
||||
圆
|
||||
圈
|
||||
地
|
||||
场
|
||||
块
|
||||
坪
|
||||
堆
|
||||
声
|
||||
壶
|
||||
处
|
||||
夜
|
||||
大
|
||||
天
|
||||
头
|
||||
套
|
||||
女
|
||||
孔
|
||||
字
|
||||
宗
|
||||
室
|
||||
家
|
||||
寸
|
||||
对
|
||||
封
|
||||
尊
|
||||
小时
|
||||
尺
|
||||
尾
|
||||
局
|
||||
层
|
||||
届
|
||||
岁
|
||||
师
|
||||
帧
|
||||
幅
|
||||
幕
|
||||
幢
|
||||
平方
|
||||
平方公尺
|
||||
平方公里
|
||||
平方分米
|
||||
平方厘米
|
||||
平方码
|
||||
平方米
|
||||
平方英寸
|
||||
平方英尺
|
||||
平方英里
|
||||
平米
|
||||
年
|
||||
年代
|
||||
年级
|
||||
度
|
||||
座
|
||||
式
|
||||
引
|
||||
张
|
||||
成
|
||||
战
|
||||
截
|
||||
户
|
||||
房
|
||||
所
|
||||
扇
|
||||
手
|
||||
打
|
||||
批
|
||||
把
|
||||
折
|
||||
担
|
||||
拍
|
||||
招
|
||||
拨
|
||||
拳
|
||||
指
|
||||
掌
|
||||
排
|
||||
撮
|
||||
支
|
||||
文
|
||||
斗
|
||||
斤
|
||||
方
|
||||
族
|
||||
日
|
||||
时
|
||||
曲
|
||||
月
|
||||
月份
|
||||
期
|
||||
本
|
||||
朵
|
||||
村
|
||||
束
|
||||
条
|
||||
来
|
||||
杯
|
||||
枚
|
||||
枝
|
||||
枪
|
||||
架
|
||||
柄
|
||||
柜
|
||||
栋
|
||||
栏
|
||||
株
|
||||
样
|
||||
根
|
||||
格
|
||||
案
|
||||
桌
|
||||
档
|
||||
桩
|
||||
桶
|
||||
梯
|
||||
棵
|
||||
楼
|
||||
次
|
||||
款
|
||||
步
|
||||
段
|
||||
毛
|
||||
毫
|
||||
毫升
|
||||
毫米
|
||||
毫克
|
||||
池
|
||||
洲
|
||||
派
|
||||
海里
|
||||
滴
|
||||
炮
|
||||
点
|
||||
点钟
|
||||
片
|
||||
版
|
||||
环
|
||||
班
|
||||
瓣
|
||||
瓶
|
||||
生
|
||||
男
|
||||
画
|
||||
界
|
||||
盆
|
||||
盎司
|
||||
盏
|
||||
盒
|
||||
盘
|
||||
相
|
||||
眼
|
||||
石
|
||||
码
|
||||
碗
|
||||
碟
|
||||
磅
|
||||
种
|
||||
科
|
||||
秒
|
||||
秒钟
|
||||
窝
|
||||
立方公尺
|
||||
立方分米
|
||||
立方厘米
|
||||
立方码
|
||||
立方米
|
||||
立方英寸
|
||||
立方英尺
|
||||
站
|
||||
章
|
||||
笔
|
||||
等
|
||||
筐
|
||||
筒
|
||||
箱
|
||||
篇
|
||||
篓
|
||||
篮
|
||||
簇
|
||||
米
|
||||
类
|
||||
粒
|
||||
级
|
||||
组
|
||||
维
|
||||
缕
|
||||
缸
|
||||
罐
|
||||
网
|
||||
群
|
||||
股
|
||||
脚
|
||||
船
|
||||
艇
|
||||
艘
|
||||
色
|
||||
节
|
||||
英亩
|
||||
英寸
|
||||
英尺
|
||||
英里
|
||||
行
|
||||
袋
|
||||
角
|
||||
言
|
||||
课
|
||||
起
|
||||
趟
|
||||
路
|
||||
车
|
||||
转
|
||||
轮
|
||||
辆
|
||||
辈
|
||||
连
|
||||
通
|
||||
遍
|
||||
部
|
||||
里
|
||||
重
|
||||
针
|
||||
钟
|
||||
钱
|
||||
锅
|
||||
门
|
||||
间
|
||||
队
|
||||
阶段
|
||||
隅
|
||||
集
|
||||
页
|
||||
顶
|
||||
顷
|
||||
项
|
||||
顿
|
||||
颗
|
||||
餐
|
||||
首
|
||||