Compare commits
41 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Magese
|
0c6560fe61 | ||
|
Magese
|
bf3df0e58c | ||
|
老王
|
4180900415 | ||
|
gaozhicheng
|
eaf58c0f1d | ||
|
gaozhicheng
|
aba7aeff19 | ||
|
gaozhicheng
|
a234d047cd | ||
|
gaozhicheng
|
8d67193774 | ||
|
gaozhicheng
|
9b731bb002 | ||
|
gaozhicheng
|
8b04070253 | ||
|
gaozhicheng
|
4dd4a86b4d | ||
|
Magese
|
59b86a341f | ||
|
dependabot[bot]
|
a45434162d | ||
|
magese
|
95ba364090 | ||
|
magese
|
0c8992fd80 | ||
|
magese
|
356d9d9ae9 | ||
|
magese
|
cd43d4b9ee | ||
|
magese
|
5068a58e0f | ||
|
magese
|
107b4ced30 | ||
|
magese
|
e6817ecba5 | ||
|
magese
|
0916f3f8bd | ||
|
magese
|
a3705f5753 | ||
|
magese
|
cc9f0d7bea | ||
|
Magese
|
7a2515a134 | ||
|
magese
|
2ebf31fed5 | ||
|
magese
|
fd35153cce | ||
|
magese
|
48a0234bb6 | ||
|
magese
|
a00336d239 | ||
|
magese
|
32d874b1ba | ||
|
magese
|
7b0d009d32 | ||
|
Magese
|
e3695f9192 | ||
|
Magese
|
33ac83d619 | ||
|
magese
|
fa1451042b | ||
|
magese
|
c2e408473a | ||
|
magese
|
712da9e7fc | ||
|
magese
|
76a4499777 | ||
|
magese
|
e6023b1317 | ||
|
magese
|
2b2f551f0e | ||
|
Magese
|
39a4fe434f | ||
|
magese
|
c73e99133a | ||
|
Magese
|
1dc338cbba | ||
|
Magese
|
76c5092fd9 |
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Screenshots**
|
||||
If applicable, add screenshots to help explain your problem.
|
||||
|
||||
**Desktop (please complete the following information):**
|
||||
- OS: [e.g. iOS]
|
||||
- Browser [e.g. chrome, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Smartphone (please complete the following information):**
|
||||
- Device: [e.g. iPhone6]
|
||||
- OS: [e.g. iOS8.1]
|
||||
- Browser [e.g. stock browser, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
+2
-1
@@ -1,3 +1,4 @@
|
||||
language: java
|
||||
|
||||
jdk:
|
||||
- oraclejdk8
|
||||
- openjdk8
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
|
||||
##Solr-Cloud说明
|
||||
## Solr-Cloud说明
|
||||
|
||||
##### 因为`Solr-Cloud`中的配置文件是交由`zookeeper`进行管理的， 所以为了方便更新动态词典， 所以也要将动态词典文件上传至`zookeeper`中，目录与`solr`的配置文件目录一致。
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
```
|
||||
|
||||
4. 配置Solr的`managed-schema`，添加`ik分词器`，示例如下；
|
||||
```console
|
||||
```xml
|
||||
<!-- ik分词器 -->
|
||||
<fieldType name="text_ik" class="solr.TextField">
|
||||
<analyzer type="index">
|
||||
|
||||
@@ -1,20 +1,22 @@
|
||||
# ik-analyzer-solr7
|
||||
ik-analyzer for solr7.x
|
||||
# ik-analyzer-solr
|
||||
ik-analyzer for solr 7.x-8.x
|
||||
|
||||
<!-- Badges section here. -->
|
||||
[](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
[](https://github.com/magese/ik-analyzer-solr7/releases)
|
||||
|
||||
[](https://github.com/magese/ik-analyzer-solr/releases)
|
||||
[](./LICENSE)
|
||||
[](https://travis-ci.org/magese/ik-analyzer-solr7)
|
||||
[](https://travis-ci.org/magese/ik-analyzer-solr)
|
||||
[](http://hits.dwyl.io/magese/ik-analyzer-solr)
|
||||
|
||||
[](https://github.com/magese/ik-analyzer-solr7/fork)
|
||||
[](https://github.com/magese/ik-analyzer-solr7/star)
|
||||
[](https://github.com/magese/ik-analyzer-solr/network/members)
|
||||
[](https://github.com/magese/ik-analyzer-solr/stargazers)
|
||||
<!-- /Badges section end. -->
|
||||
|
||||
## 简介
|
||||
#### 适配最新版本solr7；
|
||||
#### 扩展IK原有词库：
|
||||
**适配最新版本solr 7&8；**
|
||||
|
||||
**扩展IK原有词库：**
|
||||
|
||||
| 分词工具 | 词库中词的数量 | 最后更新时间 |
|
||||
| :------: | :------: | :------: |
|
||||
| ik | 27.5万 | 2012年 |
|
||||
@@ -22,23 +24,27 @@ ik-analyzer for solr7.x
|
||||
| word | 64.2万 | 2014年 |
|
||||
| jieba | 58.4万 | 2012年 |
|
||||
| jcesg | 16.6万 | 2018年 |
|
||||
| sougou词库 | 115.2万 | 2018年 |
|
||||
#### 将以上词库进行整理后约188.5万条词汇；
|
||||
#### 添加动态加载词典表功能，在不需要重启solr服务的情况下加载新增的词典。
|
||||
* IKAnalyzer的原作者为林良益<linliangyi2007@gmail.com>，项目网站为<http://code.google.com/p/ik-analyzer>
|
||||
* 该项目动态加载功能根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改，其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
| sougou词库 | 115.2万 | 2020年 |
|
||||
|
||||
**将以上词库进行整理后约187.1万条词汇；**
|
||||
|
||||
**添加动态加载词典表功能，在不需要重启solr服务的情况下加载新增的词典。**
|
||||
|
||||
> <small>关闭默认主词典请在`IKAnalyzer.cfg.xml`配置文件中设置`use_main_dict`为`false`。</small>
|
||||
> * IKAnalyzer的原作者为林良益<linliangyi2007@gmail.com>，项目网站为<http://code.google.com/p/ik-analyzer>
|
||||
> * 该项目动态加载功能根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改，其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
|
||||
|
||||
## 使用说明
|
||||
* jar包下载地址：[](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/7.6.0/ik-analyzer-7.6.0.jar)
|
||||
* jar包下载地址：[](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/8.5.0/ik-analyzer-8.5.0.jar)
|
||||
* 历史版本：[](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
|
||||
```console
|
||||
```xml
|
||||
<!-- Maven仓库地址 -->
|
||||
<dependency>
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>7.6.0</version>
|
||||
<version>8.5.0</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
@@ -58,7 +64,7 @@ ik-analyzer for solr7.x
|
||||
```
|
||||
|
||||
3. 配置Solr的`managed-schema`，添加`ik分词器`，示例如下；
|
||||
```console
|
||||
```xml
|
||||
<!-- ik分词器 -->
|
||||
<fieldType name="text_ik" class="solr.TextField">
|
||||
<analyzer type="index">
|
||||
@@ -76,8 +82,16 @@ ik-analyzer for solr7.x
|
||||
|
||||
|
||||
|
||||
5. `ik.conf`文件说明：
|
||||
```console
|
||||
5. `IKAnalyzer.cfg.xml`配置文件说明：
|
||||
|
||||
| 名称 | 类型 | 描述 | 默认 |
|
||||
| ------ | ------ | ------ | ------ |
|
||||
| use_main_dict | boolean | 是否使用默认主词典 | true |
|
||||
| ext_dict | String | 扩展词典文件名称，多个用分号隔开 | ext.dic; |
|
||||
| ext_stopwords | String | 停用词典文件名称，多个用分号隔开 | stopword.dic; |
|
||||
|
||||
6. `ik.conf`文件说明：
|
||||
```properties
|
||||
files=dynamicdic.txt
|
||||
lastupdate=0
|
||||
```
|
||||
@@ -85,31 +99,53 @@ ik-analyzer for solr7.x
|
||||
1. `files`为动态词典列表，可以设置多个词典表，用逗号进行分隔，默认动态词典表为`dynamicdic.txt`；
|
||||
2. `lastupdate`默认值为`0`，每次对动态词典表修改后请+1，不然不会将词典表中新的词语添加到内存中。<s>`lastupdate`采用的是`int`类型，不支持时间戳，如果使用时间戳的朋友可以把源码中的`int`改成`long`即可；</s> `2018-08-23` 已将源码中`lastUpdate`改为`long`类型，现可以用时间戳了。
|
||||
|
||||
6. `dynamicdic.txt` 为动态词典
|
||||
7. `dynamicdic.txt` 为动态词典
|
||||
|
||||
在此文件配置的词语不需重启服务即可加载进内存中。
|
||||
以`#`开头的词语视为注释，将不会加载到内存中。
|
||||
|
||||
|
||||
## 更新说明
|
||||
- `2018-12-26:`
|
||||
- **2021-12-23:** 升级lucene版本为`8.5.0`
|
||||
- **2021-03-22:** 升级lucene版本为`8.4.0`
|
||||
- **2020-12-30:**
|
||||
- 升级lucene版本为`8.3.1`
|
||||
- 更新词库
|
||||
- **2019-11-12:**
|
||||
- 升级lucene版本为`8.3.0`
|
||||
- `IKAnalyzer.cfg.xml`增加配置项`use_main_dict`，用于配置是否启用默认主词典
|
||||
- **2019-09-27:** 升级lucene版本为`8.2.0`
|
||||
- **2019-07-11:** 升级lucene版本为`8.1.1`
|
||||
- **2019-05-27:**
|
||||
- 升级lucene版本为`8.1.0`
|
||||
- 优化原词典部分重复词语
|
||||
- 更新搜狗2019最新流行词汇词典，约20k词汇量
|
||||
- **2019-05-15:** 升级lucene版本为`8.0.0`，并支持Solr8使用
|
||||
- **2019-03-01:** 升级lucene版本为`7.7.1`
|
||||
- **2019-02-15:** 升级lucene版本为`7.7.0`
|
||||
- **2018-12-26:**
|
||||
- 升级lucene版本为`7.6.0`
|
||||
- 兼容solr-cloud，动态词典配置文件及动态词典可交由`zookeeper`进行管理
|
||||
- 动态词典增加注释功能，以`#`开头的行将视为注释
|
||||
- `2018-12-04:` 整理更新词库列表`magese.dic`
|
||||
- `2018-10-10:` 升级lucene版本为`7.5.0`
|
||||
- `2018-09-03:` 优化注释与输出信息，取消部分中文输出避免不同字符集乱码，现会打印被调用inform方法的hashcode
|
||||
- `2018-08-23: `
|
||||
- **2018-12-04:** 整理更新词库列表`magese.dic`
|
||||
- **2018-10-10:** 升级lucene版本为`7.5.0`
|
||||
- **2018-09-03:** 优化注释与输出信息，取消部分中文输出避免不同字符集乱码，现会打印被调用inform方法的hashcode
|
||||
- **2018-08-23:**
|
||||
- 完善了动态更新词库代码注释；
|
||||
- 将ik.conf配置文件中的lastUpdate属性改为long类型，现已支持时间戳形式
|
||||
- `2018-08-13:` 更新maven仓库地址
|
||||
- `2018-08-01:` 移除默认的扩展词与停用词
|
||||
- `2018-07-23:` 升级lucene版本为`7.4.0`
|
||||
- **2018-08-13:** 更新maven仓库地址
|
||||
- **2018-08-01:** 移除默认的扩展词与停用词
|
||||
- **2018-07-23:** 升级lucene版本为`7.4.0`
|
||||
|
||||
|
||||
## 感谢 Thanks
|
||||
|
||||
[](https://www.jetbrains.com/?from=ik-analyzer-solr)
|
||||
|
||||
[](https://www.java.com)
|
||||
|
||||
|
||||
## BUG & 疑问 & 其它
|
||||
如果您在使用过程中遇到了BUG，或者有不清楚的地方，请挂ISSUE或者联系作者：<magese@live.cn>
|
||||
|
||||
如果您觉得该项目对您有帮助，请别忘记给这个项目一个`star`
|
||||
|
||||
[](https://www.java.com)
|
||||
@@ -0,0 +1,66 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!-- Generator: Adobe Illustrator 19.1.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
|
||||
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
|
||||
width="120.1px" height="130.2px" viewBox="0 0 120.1 130.2" style="enable-background:new 0 0 120.1 130.2;" xml:space="preserve"
|
||||
>
|
||||
<g>
|
||||
<linearGradient id="XMLID_2_" gradientUnits="userSpaceOnUse" x1="31.8412" y1="120.5578" x2="110.2402" y2="73.24">
|
||||
<stop offset="0" style="stop-color:#FCEE39"/>
|
||||
<stop offset="1" style="stop-color:#F37B3D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3041_" style="fill:url(#XMLID_2_);" d="M118.6,71.8c0.9-0.8,1.4-1.9,1.5-3.2c0.1-2.6-1.8-4.7-4.4-4.9
|
||||
c-1.2-0.1-2.4,0.4-3.3,1.1l0,0l-83.8,45.9c-1.9,0.8-3.6,2.2-4.7,4.1c-2.9,4.8-1.3,11,3.6,13.9c3.4,2,7.5,1.8,10.7-0.2l0,0l0,0
|
||||
c0.2-0.2,0.5-0.3,0.7-0.5l78-54.8C117.3,72.9,118.4,72.1,118.6,71.8L118.6,71.8L118.6,71.8z"/>
|
||||
<linearGradient id="XMLID_3_" gradientUnits="userSpaceOnUse" x1="48.3607" y1="6.9083" x2="119.9179" y2="69.5546">
|
||||
<stop offset="0" style="stop-color:#EF5A6B"/>
|
||||
<stop offset="0.57" style="stop-color:#F26F4E"/>
|
||||
<stop offset="1" style="stop-color:#F37B3D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3049_" style="fill:url(#XMLID_3_);" d="M118.8,65.1L118.8,65.1L55,2.5C53.6,1,51.6,0,49.3,0
|
||||
c-4.3,0-7.7,3.5-7.7,7.7v0c0,2.1,0.8,3.9,2.1,5.3l0,0l0,0c0.4,0.4,0.8,0.7,1.2,1l67.4,57.7l0,0c0.8,0.7,1.8,1.2,3,1.3
|
||||
c2.6,0.1,4.7-1.8,4.9-4.4C120.2,67.3,119.7,66,118.8,65.1z"/>
|
||||
<linearGradient id="XMLID_4_" gradientUnits="userSpaceOnUse" x1="52.9467" y1="63.6407" x2="10.5379" y2="37.1562">
|
||||
<stop offset="0" style="stop-color:#7C59A4"/>
|
||||
<stop offset="0.3852" style="stop-color:#AF4C92"/>
|
||||
<stop offset="0.7654" style="stop-color:#DC4183"/>
|
||||
<stop offset="0.957" style="stop-color:#ED3D7D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3042_" style="fill:url(#XMLID_4_);" d="M57.1,59.5C57,59.5,17.7,28.5,16.9,28l0,0l0,0c-0.6-0.3-1.2-0.6-1.8-0.9
|
||||
c-5.8-2.2-12.2,0.8-14.4,6.6c-1.9,5.1,0.2,10.7,4.6,13.4l0,0l0,0C6,47.5,6.6,47.8,7.3,48c0.4,0.2,45.4,18.8,45.4,18.8l0,0
|
||||
c1.8,0.8,3.9,0.3,5.1-1.2C59.3,63.7,59,61,57.1,59.5z"/>
|
||||
<linearGradient id="XMLID_5_" gradientUnits="userSpaceOnUse" x1="52.1736" y1="3.7019" x2="10.7706" y2="37.8971">
|
||||
<stop offset="0" style="stop-color:#EF5A6B"/>
|
||||
<stop offset="0.364" style="stop-color:#EE4E72"/>
|
||||
<stop offset="1" style="stop-color:#ED3D7D"/>
|
||||
</linearGradient>
|
||||
<path id="XMLID_3057_" style="fill:url(#XMLID_5_);" d="M49.3,0c-1.7,0-3.3,0.6-4.6,1.5L4.9,28.3c-0.1,0.1-0.2,0.1-0.2,0.2l-0.1,0
|
||||
l0,0c-1.7,1.2-3.1,3-3.9,5.1C-1.5,39.4,1.5,45.9,7.3,48c3.6,1.4,7.5,0.7,10.4-1.4l0,0l0,0c0.7-0.5,1.3-1,1.8-1.6l34.6-31.2l0,0
|
||||
c1.8-1.4,3-3.6,3-6.1v0C57.1,3.5,53.6,0,49.3,0z"/>
|
||||
<g id="XMLID_3008_">
|
||||
<rect id="XMLID_3033_" x="34.6" y="37.4" style="fill:#000000;" width="51" height="51"/>
|
||||
<rect id="XMLID_3032_" x="39" y="78.8" style="fill:#FFFFFF;" width="19.1" height="3.2"/>
|
||||
<g id="XMLID_3009_">
|
||||
<path id="XMLID_3030_" style="fill:#FFFFFF;" d="M38.8,50.8l1.5-1.4c0.4,0.5,0.8,0.8,1.3,0.8c0.6,0,0.9-0.4,0.9-1.2l0-5.3l2.3,0
|
||||
l0,5.3c0,1-0.3,1.8-0.8,2.3c-0.5,0.5-1.3,0.8-2.3,0.8C40.2,52.2,39.4,51.6,38.8,50.8z"/>
|
||||
<path id="XMLID_3028_" style="fill:#FFFFFF;" d="M45.3,43.8l6.7,0v1.9l-4.4,0V47l4,0l0,1.8l-4,0l0,1.3l4.5,0l0,2l-6.7,0
|
||||
L45.3,43.8z"/>
|
||||
<path id="XMLID_3026_" style="fill:#FFFFFF;" d="M55,45.8l-2.5,0l0-2l7.3,0l0,2l-2.5,0l0,6.3l-2.3,0L55,45.8z"/>
|
||||
<path id="XMLID_3022_" style="fill:#FFFFFF;" d="M39,54l4.3,0c1,0,1.8,0.3,2.3,0.7c0.3,0.3,0.5,0.8,0.5,1.4v0
|
||||
c0,1-0.5,1.5-1.3,1.9c1,0.3,1.6,0.9,1.6,2v0c0,1.4-1.2,2.3-3.1,2.3l-4.3,0L39,54z M43.8,56.6c0-0.5-0.4-0.7-1-0.7l-1.5,0l0,1.5
|
||||
l1.4,0C43.4,57.3,43.8,57.1,43.8,56.6L43.8,56.6z M43,59l-1.8,0l0,1.5H43c0.7,0,1.1-0.3,1.1-0.8v0C44.1,59.2,43.7,59,43,59z"/>
|
||||
<path id="XMLID_3019_" style="fill:#FFFFFF;" d="M46.8,54l3.9,0c1.3,0,2.1,0.3,2.7,0.9c0.5,0.5,0.7,1.1,0.7,1.9v0
|
||||
c0,1.3-0.7,2.1-1.7,2.6l2,2.9l-2.6,0l-1.7-2.5h-1l0,2.5l-2.3,0L46.8,54z M50.6,58c0.8,0,1.2-0.4,1.2-1v0c0-0.7-0.5-1-1.2-1
|
||||
l-1.5,0v2H50.6z"/>
|
||||
<path id="XMLID_3016_" style="fill:#FFFFFF;" d="M56.8,54l2.2,0l3.5,8.4l-2.5,0l-0.6-1.5l-3.2,0l-0.6,1.5l-2.4,0L56.8,54z
|
||||
M58.8,59l-0.9-2.3L57,59L58.8,59z"/>
|
||||
<path id="XMLID_3014_" style="fill:#FFFFFF;" d="M62.8,54l2.3,0l0,8.3l-2.3,0L62.8,54z"/>
|
||||
<path id="XMLID_3012_" style="fill:#FFFFFF;" d="M65.7,54l2.1,0l3.4,4.4l0-4.4l2.3,0l0,8.3l-2,0L68,57.8l0,4.6l-2.3,0L65.7,54z"
|
||||
/>
|
||||
<path id="XMLID_3010_" style="fill:#FFFFFF;" d="M73.7,61.1l1.3-1.5c0.8,0.7,1.7,1,2.7,1c0.6,0,1-0.2,1-0.6v0
|
||||
c0-0.4-0.3-0.5-1.4-0.8c-1.8-0.4-3.1-0.9-3.1-2.6v0c0-1.5,1.2-2.7,3.2-2.7c1.4,0,2.5,0.4,3.4,1.1l-1.2,1.6
|
||||
c-0.8-0.5-1.6-0.8-2.3-0.8c-0.6,0-0.8,0.2-0.8,0.5v0c0,0.4,0.3,0.5,1.4,0.8c1.9,0.4,3.1,1,3.1,2.6v0c0,1.7-1.3,2.7-3.4,2.7
|
||||
C76.1,62.5,74.7,62,73.7,61.1z"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.8 KiB |
@@ -4,29 +4,22 @@
|
||||
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>7.6.0</version>
|
||||
<version>8.5.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<name>ik-analyzer-solr7</name>
|
||||
<name>ik-analyzer-solr</name>
|
||||
<url>http://code.google.com/p/ik-analyzer/</url>
|
||||
<description>IK-Analyzer for solr7.6</description>
|
||||
<description>IK-Analyzer for solr 7-8</description>
|
||||
|
||||
<properties>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<lucene.version>7.6.0</lucene.version>
|
||||
<lucene.version>8.5.0</lucene.version>
|
||||
<javac.src.version>1.8</javac.src.version>
|
||||
<javac.target.version>1.8</javac.target.version>
|
||||
<maven.compiler.plugin.version>3.3</maven.compiler.plugin.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<version>4.11</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.lucene</groupId>
|
||||
<artifactId>lucene-core</artifactId>
|
||||
@@ -55,9 +48,9 @@
|
||||
</licenses>
|
||||
<scm>
|
||||
<tag>master</tag>
|
||||
<url>https://github.com/magese/ik-analyzer-solr7</url>
|
||||
<connection>scm:git:git@github.com:magese/ik-analyzer-solr7.git</connection>
|
||||
<developerConnection>scm:git:git@github.com:magese/ik-analyzer-solr7.git</developerConnection>
|
||||
<url>https://github.com/magese/ik-analyzer-solr</url>
|
||||
<connection>scm:git:git@github.com:magese/ik-analyzer-solr.git</connection>
|
||||
<developerConnection>scm:git:git@github.com:magese/ik-analyzer-solr.git</developerConnection>
|
||||
</scm>
|
||||
<developers>
|
||||
<developer>
|
||||
@@ -152,4 +145,3 @@
|
||||
</profile>
|
||||
</profiles>
|
||||
</project>
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
@@ -50,6 +50,19 @@ public interface Configuration {
|
||||
*/
|
||||
void setUseSmart(boolean useSmart);
|
||||
|
||||
/**
|
||||
* 获取是否使用主词典
|
||||
*
|
||||
* @return = true 默认加载主词典， = false 不加载主词典
|
||||
*/
|
||||
boolean useMainDict();
|
||||
|
||||
/**
|
||||
* 设置是否使用主词典
|
||||
*
|
||||
* @param useMainDic = true 默认加载主词典， = false 不加载主词典
|
||||
*/
|
||||
void setUseMainDict(boolean useMainDic);
|
||||
|
||||
/**
|
||||
* 获取主词典路径
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
@@ -41,24 +41,28 @@ public class DefaultConfig implements Configuration {
|
||||
/*
|
||||
* 分词器默认字典路径
|
||||
*/
|
||||
private static final String PATH_DIC_MAIN = "dict/magese.dic";
|
||||
private static final String PATH_DIC_MAIN = "dict/main_dic_2020.dic";
|
||||
private static final String PATH_DIC_QUANTIFIER = "dict/quantifier.dic";
|
||||
|
||||
/*
|
||||
* 分词器配置文件路径
|
||||
*/
|
||||
private static final String FILE_NAME = "IKAnalyzer.cfg.xml";
|
||||
// 配置属性——是否使用主词典
|
||||
private static final String USE_MAIN = "use_main_dict";
|
||||
// 配置属性——扩展字典
|
||||
private static final String EXT_DICT = "ext_dict";
|
||||
// 配置属性——扩展停止词典
|
||||
private static final String EXT_STOP = "ext_stopwords";
|
||||
|
||||
private Properties props;
|
||||
/*
|
||||
* 是否使用smart方式分词
|
||||
*/
|
||||
private final Properties props;
|
||||
|
||||
// 是否使用smart方式分词
|
||||
private boolean useSmart;
|
||||
|
||||
// 是否加载主词典
|
||||
private boolean useMainDict = true;
|
||||
|
||||
/**
|
||||
* 返回单例
|
||||
*
|
||||
@@ -100,10 +104,33 @@ public class DefaultConfig implements Configuration {
|
||||
*
|
||||
* @param useSmart =true ，分词器使用智能切分策略， =false则使用细粒度切分
|
||||
*/
|
||||
@Override
|
||||
public void setUseSmart(boolean useSmart) {
|
||||
this.useSmart = useSmart;
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取是否使用主词典
|
||||
*
|
||||
* @return = true 默认加载主词典， = false 不加载主词典
|
||||
*/
|
||||
public boolean useMainDict() {
|
||||
String useMainDictCfg = props.getProperty(USE_MAIN);
|
||||
if (useMainDictCfg != null && useMainDictCfg.trim().length() > 0)
|
||||
setUseMainDict(Boolean.parseBoolean(useMainDictCfg));
|
||||
return useMainDict;
|
||||
}
|
||||
|
||||
/**
|
||||
* 设置是否使用主词典
|
||||
*
|
||||
* @param useMainDict = true 默认加载主词典， = false 不加载主词典
|
||||
*/
|
||||
@Override
|
||||
public void setUseMainDict(boolean useMainDict) {
|
||||
this.useMainDict = useMainDict;
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取主词典路径
|
||||
*
|
||||
@@ -142,7 +169,6 @@ public class DefaultConfig implements Configuration {
|
||||
return extDictFiles;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 获取扩展停止词典配置路径
|
||||
*
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,23 +21,19 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Reader;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.LinkedList;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import org.wltea.analyzer.cfg.Configuration;
|
||||
import org.wltea.analyzer.dic.Dictionary;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Reader;
|
||||
import java.util.*;
|
||||
|
||||
/**
|
||||
* 分词器上下文状态
|
||||
*/
|
||||
@@ -66,17 +62,17 @@ class AnalyzeContext {
|
||||
|
||||
//子分词器锁
|
||||
//该集合非空，说明有子分词器在占用segmentBuff
|
||||
private Set<String> buffLocker;
|
||||
private final Set<String> buffLocker;
|
||||
|
||||
//原始分词结果集合，未经歧义处理
|
||||
private QuickSortSet orgLexemes;
|
||||
//LexemePath位置索引表
|
||||
private Map<Integer, LexemePath> pathMap;
|
||||
private final Map<Integer, LexemePath> pathMap;
|
||||
//最终分词结果集
|
||||
private LinkedList<Lexeme> results;
|
||||
private final LinkedList<Lexeme> results;
|
||||
|
||||
//分词器配置项
|
||||
private Configuration cfg;
|
||||
private final Configuration cfg;
|
||||
|
||||
AnalyzeContext(Configuration cfg) {
|
||||
this.cfg = cfg;
|
||||
@@ -254,7 +250,7 @@ class AnalyzeContext {
|
||||
*/
|
||||
void outputToResult() {
|
||||
int index = 0;
|
||||
for (; index <= this.cursor; ) {
|
||||
while (index <= this.cursor) {
|
||||
//跳过非CJK字符
|
||||
if (CharacterUtil.CHAR_USELESS == this.charTypes[index]) {
|
||||
index++;
|
||||
@@ -353,12 +349,14 @@ class AnalyzeContext {
|
||||
if (Lexeme.TYPE_ARABIC == result.getLexemeType()) {
|
||||
Lexeme nextLexeme = this.results.peekFirst();
|
||||
boolean appendOk = false;
|
||||
if (Lexeme.TYPE_CNUM == nextLexeme.getLexemeType()) {
|
||||
//合并英文数词+中文数词
|
||||
appendOk = result.append(nextLexeme, Lexeme.TYPE_CNUM);
|
||||
} else if (Lexeme.TYPE_COUNT == nextLexeme.getLexemeType()) {
|
||||
//合并英文数词+中文量词
|
||||
appendOk = result.append(nextLexeme, Lexeme.TYPE_CQUAN);
|
||||
if (nextLexeme != null) {
|
||||
if (Lexeme.TYPE_CNUM == nextLexeme.getLexemeType()) {
|
||||
//合并英文数词+中文数词
|
||||
appendOk = result.append(nextLexeme, Lexeme.TYPE_CNUM);
|
||||
} else if (Lexeme.TYPE_COUNT == nextLexeme.getLexemeType()) {
|
||||
//合并英文数词+中文量词
|
||||
appendOk = result.append(nextLexeme, Lexeme.TYPE_CQUAN);
|
||||
}
|
||||
}
|
||||
if (appendOk) {
|
||||
//弹出
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
@@ -38,13 +38,13 @@ import org.wltea.analyzer.dic.Hit;
|
||||
* 中文-日韩文子分词器
|
||||
*/
|
||||
class CJKSegmenter implements ISegmenter {
|
||||
|
||||
|
||||
//子分词器标签
|
||||
private static final String SEGMENTER_NAME = "CJK_SEGMENTER";
|
||||
//待处理的分词hit队列
|
||||
private List<Hit> tmpHits;
|
||||
|
||||
|
||||
|
||||
|
||||
CJKSegmenter(){
|
||||
this.tmpHits = new LinkedList<>();
|
||||
}
|
||||
@@ -54,7 +54,7 @@ class CJKSegmenter implements ISegmenter {
|
||||
*/
|
||||
public void analyze(AnalyzeContext context) {
|
||||
if(CharacterUtil.CHAR_USELESS != context.getCurrentCharType()){
|
||||
|
||||
|
||||
//优先处理tmpHits中的hit
|
||||
if(!this.tmpHits.isEmpty()){
|
||||
//处理词段队列
|
||||
@@ -65,18 +65,18 @@ class CJKSegmenter implements ISegmenter {
|
||||
//输出当前的词
|
||||
Lexeme newLexeme = new Lexeme(context.getBufferOffset() , hit.getBegin() , context.getCursor() - hit.getBegin() + 1 , Lexeme.TYPE_CNWORD);
|
||||
context.addLexeme(newLexeme);
|
||||
|
||||
|
||||
if(!hit.isPrefix()){//不是词前缀，hit不需要继续匹配，移除
|
||||
this.tmpHits.remove(hit);
|
||||
}
|
||||
|
||||
|
||||
}else if(hit.isUnmatch()){
|
||||
//hit不是词，移除
|
||||
this.tmpHits.remove(hit);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
//*********************************
|
||||
//再对当前指针位置的字符进行单字匹配
|
||||
Hit singleCharHit = Dictionary.getSingleton().matchInMainDict(context.getSegmentBuff(), context.getCursor(), 1);
|
||||
@@ -94,24 +94,24 @@ class CJKSegmenter implements ISegmenter {
|
||||
//前缀匹配则放入hit列表
|
||||
this.tmpHits.add(singleCharHit);
|
||||
}
|
||||
|
||||
|
||||
|
||||
}else{
|
||||
//遇到CHAR_USELESS字符
|
||||
//清空队列
|
||||
this.tmpHits.clear();
|
||||
}
|
||||
|
||||
|
||||
//判断缓冲区是否已经读完
|
||||
if(context.isBufferConsumed()){
|
||||
//清空队列
|
||||
this.tmpHits.clear();
|
||||
}
|
||||
|
||||
|
||||
//判断是否锁定缓冲区
|
||||
if(this.tmpHits.size() == 0){
|
||||
context.unlockBuffer(SEGMENTER_NAME);
|
||||
|
||||
|
||||
}else{
|
||||
context.lockBuffer(SEGMENTER_NAME);
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
@@ -36,11 +36,11 @@ import java.util.List;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
* 中文数量词子分词器
|
||||
*/
|
||||
class CN_QuantifierSegmenter implements ISegmenter{
|
||||
|
||||
|
||||
//子分词器标签
|
||||
private static final String SEGMENTER_NAME = "QUAN_SEGMENTER";
|
||||
|
||||
@@ -54,7 +54,7 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
ChnNumberChars.add(nChar);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* 词元的开始位置，
|
||||
* 同时作为子分词器状态标识
|
||||
@@ -69,14 +69,14 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
|
||||
//待处理的量词hit队列
|
||||
private List<Hit> countHits;
|
||||
|
||||
|
||||
|
||||
|
||||
CN_QuantifierSegmenter(){
|
||||
nStart = -1;
|
||||
nEnd = -1;
|
||||
this.countHits = new LinkedList<>();
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 分词
|
||||
*/
|
||||
@@ -85,7 +85,7 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
this.processCNumber(context);
|
||||
//处理中文量词
|
||||
this.processCount(context);
|
||||
|
||||
|
||||
//判断是否锁定缓冲区
|
||||
if(this.nStart == -1 && this.nEnd == -1 && countHits.isEmpty()){
|
||||
//对缓冲区解锁
|
||||
@@ -94,7 +94,7 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
context.lockBuffer(SEGMENTER_NAME);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* 重置子分词器状态
|
||||
@@ -104,20 +104,20 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
nEnd = -1;
|
||||
countHits.clear();
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 处理数词
|
||||
*/
|
||||
private void processCNumber(AnalyzeContext context){
|
||||
if(nStart == -1 && nEnd == -1){//初始状态
|
||||
if(CharacterUtil.CHAR_CHINESE == context.getCurrentCharType()
|
||||
if(CharacterUtil.CHAR_CHINESE == context.getCurrentCharType()
|
||||
&& ChnNumberChars.contains(context.getCurrentChar())){
|
||||
//记录数词的起始、结束位置
|
||||
nStart = context.getCursor();
|
||||
nEnd = context.getCursor();
|
||||
}
|
||||
}else{//正在处理状态
|
||||
if(CharacterUtil.CHAR_CHINESE == context.getCurrentCharType()
|
||||
if(CharacterUtil.CHAR_CHINESE == context.getCurrentCharType()
|
||||
&& ChnNumberChars.contains(context.getCurrentChar())){
|
||||
//记录数词的结束位置
|
||||
nEnd = context.getCursor();
|
||||
@@ -129,7 +129,7 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
nEnd = -1;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
//缓冲区已经用完，还有尚未输出的数词
|
||||
if(context.isBufferConsumed()){
|
||||
if(nStart != -1 && nEnd != -1){
|
||||
@@ -139,9 +139,9 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
nStart = -1;
|
||||
nEnd = -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 处理中文量词
|
||||
* @param context 需要处理的内容
|
||||
@@ -151,9 +151,9 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
if(!this.needCountScan(context)){
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
if(CharacterUtil.CHAR_CHINESE == context.getCurrentCharType()){
|
||||
|
||||
|
||||
//优先处理countHits中的hit
|
||||
if(!this.countHits.isEmpty()){
|
||||
//处理词段队列
|
||||
@@ -164,17 +164,17 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
//输出当前的词
|
||||
Lexeme newLexeme = new Lexeme(context.getBufferOffset() , hit.getBegin() , context.getCursor() - hit.getBegin() + 1 , Lexeme.TYPE_COUNT);
|
||||
context.addLexeme(newLexeme);
|
||||
|
||||
|
||||
if(!hit.isPrefix()){//不是词前缀，hit不需要继续匹配，移除
|
||||
this.countHits.remove(hit);
|
||||
}
|
||||
|
||||
|
||||
}else if(hit.isUnmatch()){
|
||||
//hit不是词，移除
|
||||
this.countHits.remove(hit);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//*********************************
|
||||
//对当前指针位置的字符进行单字匹配
|
||||
@@ -193,21 +193,21 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
//前缀匹配则放入hit列表
|
||||
this.countHits.add(singleCharHit);
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
}else{
|
||||
//输入的不是中文字符
|
||||
//清空未成形的量词
|
||||
this.countHits.clear();
|
||||
}
|
||||
|
||||
|
||||
//缓冲区数据已经读完，还有尚未输出的量词
|
||||
if(context.isBufferConsumed()){
|
||||
//清空未成形的量词
|
||||
this.countHits.clear();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 判断是否需要扫描量词
|
||||
*/
|
||||
@@ -226,7 +226,7 @@ class CN_QuantifierSegmenter implements ISegmenter{
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 添加数词词元到结果集
|
||||
* @param context 需要添加的词元
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
@@ -32,18 +32,18 @@ package org.wltea.analyzer.core;
|
||||
* 字符集识别工具类
|
||||
*/
|
||||
class CharacterUtil {
|
||||
|
||||
|
||||
static final int CHAR_USELESS = 0;
|
||||
|
||||
|
||||
static final int CHAR_ARABIC = 0X00000001;
|
||||
|
||||
|
||||
static final int CHAR_ENGLISH = 0X00000002;
|
||||
|
||||
|
||||
static final int CHAR_CHINESE = 0X00000004;
|
||||
|
||||
|
||||
static final int CHAR_OTHER_CJK = 0X00000008;
|
||||
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* 识别字符类型
|
||||
* @param input 需要识别的字符
|
||||
@@ -52,23 +52,23 @@ class CharacterUtil {
|
||||
static int identifyCharType(char input){
|
||||
if(input >= '0' && input <= '9'){
|
||||
return CHAR_ARABIC;
|
||||
|
||||
|
||||
}else if((input >= 'a' && input <= 'z')
|
||||
|| (input >= 'A' && input <= 'Z')){
|
||||
return CHAR_ENGLISH;
|
||||
|
||||
|
||||
}else {
|
||||
Character.UnicodeBlock ub = Character.UnicodeBlock.of(input);
|
||||
|
||||
if(ub == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS
|
||||
|| ub == Character.UnicodeBlock.CJK_COMPATIBILITY_IDEOGRAPHS
|
||||
|
||||
if(ub == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS
|
||||
|| ub == Character.UnicodeBlock.CJK_COMPATIBILITY_IDEOGRAPHS
|
||||
|| ub == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_A){
|
||||
//目前已知的中文字符UTF-8集合
|
||||
return CHAR_CHINESE;
|
||||
|
||||
|
||||
}else if(ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS //全角数字字符和日韩字符
|
||||
//韩文字符集
|
||||
|| ub == Character.UnicodeBlock.HANGUL_SYLLABLES
|
||||
|| ub == Character.UnicodeBlock.HANGUL_SYLLABLES
|
||||
|| ub == Character.UnicodeBlock.HANGUL_JAMO
|
||||
|| ub == Character.UnicodeBlock.HANGUL_COMPATIBILITY_JAMO
|
||||
//日文字符集
|
||||
@@ -76,13 +76,13 @@ class CharacterUtil {
|
||||
|| ub == Character.UnicodeBlock.KATAKANA //片假名
|
||||
|| ub == Character.UnicodeBlock.KATAKANA_PHONETIC_EXTENSIONS){
|
||||
return CHAR_OTHER_CJK;
|
||||
|
||||
|
||||
}
|
||||
}
|
||||
//其他的不做处理的字符
|
||||
return CHAR_USELESS;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 进行字符规格化（全角转半角，大写转小写处理）
|
||||
* @param input 需要转换的字符
|
||||
@@ -91,14 +91,14 @@ class CharacterUtil {
|
||||
static char regularize(char input){
|
||||
if (input == 12288) {
|
||||
input = (char) 32;
|
||||
|
||||
|
||||
}else if (input > 65280 && input < 65375) {
|
||||
input = (char) (input - 65248);
|
||||
|
||||
|
||||
}else if (input >= 'A' && input <= 'Z') {
|
||||
input += 32;
|
||||
}
|
||||
|
||||
|
||||
return input;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,21 +21,21 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
import org.wltea.analyzer.cfg.Configuration;
|
||||
import org.wltea.analyzer.cfg.DefaultConfig;
|
||||
import org.wltea.analyzer.dic.Dictionary;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Reader;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import org.wltea.analyzer.cfg.Configuration;
|
||||
import org.wltea.analyzer.cfg.DefaultConfig;
|
||||
import org.wltea.analyzer.dic.Dictionary;
|
||||
|
||||
/**
|
||||
* IK分词器主类
|
||||
*/
|
||||
@@ -58,7 +58,6 @@ public final class IKSegmenter {
|
||||
*
|
||||
* @param input 读取流
|
||||
* @param useSmart 为true，使用智能分词策略
|
||||
* <p>
|
||||
* 非智能分词：细粒度输出所有可能的切分结果
|
||||
* 智能分词： 合并数词和量词，对分词结果进行歧义判断
|
||||
*/
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,26 +21,26 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
* 子分词器接口
|
||||
*/
|
||||
interface ISegmenter {
|
||||
|
||||
|
||||
/**
|
||||
* 从分析器读取下一个可能分解的词元对象
|
||||
* @param context 分词算法上下文
|
||||
*/
|
||||
void analyze(AnalyzeContext context);
|
||||
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* 重置子分析器状态
|
||||
*/
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,14 +21,14 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
/**
|
||||
* IK词元对象
|
||||
* IK词元对象
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
public class Lexeme implements Comparable<Lexeme>{
|
||||
@@ -50,7 +50,7 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
static final int TYPE_COUNT = 32;
|
||||
//中文数量词
|
||||
static final int TYPE_CQUAN = 48;
|
||||
|
||||
|
||||
//词元的起始位移
|
||||
private int offset;
|
||||
//词元的相对起始位置
|
||||
@@ -61,8 +61,8 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
private String lexemeText;
|
||||
//词元类型
|
||||
private int lexemeType;
|
||||
|
||||
|
||||
|
||||
|
||||
public Lexeme(int offset , int begin , int length , int lexemeType){
|
||||
this.offset = offset;
|
||||
this.begin = begin;
|
||||
@@ -72,7 +72,7 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
this.length = length;
|
||||
this.lexemeType = lexemeType;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* 判断词元相等算法
|
||||
* 起始位置偏移、起始位置、终止位置相同
|
||||
@@ -82,21 +82,21 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
if(o == null){
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
if(this == o){
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
if(o instanceof Lexeme){
|
||||
Lexeme other = (Lexeme)o;
|
||||
return this.offset == other.getOffset()
|
||||
&& this.begin == other.getBegin()
|
||||
&& this.length == other.getLength();
|
||||
}else{
|
||||
}else{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* 词元哈希编码算法
|
||||
* @see java.lang.Object#hashCode()
|
||||
@@ -106,7 +106,7 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
int absEnd = getEndPosition();
|
||||
return (absBegin * 37) + (absEnd * 31) + ((absBegin * absEnd) % getLength()) * 11;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* 词元在排序集合中的比较算法
|
||||
* @see java.lang.Comparable#compareTo(java.lang.Object)
|
||||
@@ -119,12 +119,12 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
//词元长度优先
|
||||
//this.length < other.getLength()
|
||||
return Integer.compare(other.getLength(), this.length);
|
||||
|
||||
|
||||
}else{//this.begin > other.getBegin()
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
private int getOffset() {
|
||||
return offset;
|
||||
}
|
||||
@@ -155,22 +155,22 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
public int getEndPosition(){
|
||||
return offset + begin + length;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 获取词元的字符长度
|
||||
* @return int
|
||||
*/
|
||||
public int getLength(){
|
||||
return this.length;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
public void setLength(int length) {
|
||||
if(this.length < 0){
|
||||
throw new IllegalArgumentException("length < 0");
|
||||
}
|
||||
this.length = length;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 获取词元的文本内容
|
||||
* @return String
|
||||
@@ -199,7 +199,7 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
int getLexemeType() {
|
||||
return lexemeType;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 获取词元类型标示字符串
|
||||
* @return String
|
||||
@@ -209,41 +209,41 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
|
||||
case TYPE_ENGLISH :
|
||||
return "ENGLISH";
|
||||
|
||||
|
||||
case TYPE_ARABIC :
|
||||
return "ARABIC";
|
||||
|
||||
|
||||
case TYPE_LETTER :
|
||||
return "LETTER";
|
||||
|
||||
case TYPE_CNWORD :
|
||||
|
||||
case TYPE_CNWORD :
|
||||
return "CN_WORD";
|
||||
|
||||
case TYPE_CNCHAR :
|
||||
|
||||
case TYPE_CNCHAR :
|
||||
return "CN_CHAR";
|
||||
|
||||
|
||||
case TYPE_OTHER_CJK :
|
||||
return "OTHER_CJK";
|
||||
|
||||
|
||||
case TYPE_COUNT :
|
||||
return "COUNT";
|
||||
|
||||
|
||||
case TYPE_CNUM :
|
||||
return "TYPE_CNUM";
|
||||
|
||||
case TYPE_CQUAN:
|
||||
|
||||
case TYPE_CQUAN:
|
||||
return "TYPE_CQUAN";
|
||||
|
||||
|
||||
default :
|
||||
return "UNKONW";
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
public void setLexemeType(int lexemeType) {
|
||||
this.lexemeType = lexemeType;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* 合并两个相邻的词元
|
||||
* @return boolean 词元是否成功合并
|
||||
@@ -257,16 +257,16 @@ public class Lexeme implements Comparable<Lexeme>{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
*/
|
||||
public String toString(){
|
||||
return this.getBeginPosition() + "-" + this.getEndPosition() +
|
||||
" : " + this.lexemeText + " : \t" +
|
||||
this.getLexemeTypeString();
|
||||
}
|
||||
|
||||
|
||||
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.2.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.2.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
@@ -31,178 +31,182 @@ package org.wltea.analyzer.core;
|
||||
* IK分词器专用的Lexem快速排序集合
|
||||
*/
|
||||
class QuickSortSet {
|
||||
//链表头
|
||||
private Cell head;
|
||||
//链表尾
|
||||
private Cell tail;
|
||||
//链表的实际大小
|
||||
private int size;
|
||||
|
||||
QuickSortSet(){
|
||||
this.size = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* 向链表集合添加词元
|
||||
*/
|
||||
void addLexeme(Lexeme lexeme){
|
||||
Cell newCell = new Cell(lexeme);
|
||||
if(this.size == 0){
|
||||
this.head = newCell;
|
||||
this.tail = newCell;
|
||||
this.size++;
|
||||
//链表头
|
||||
private Cell head;
|
||||
//链表尾
|
||||
private Cell tail;
|
||||
//链表的实际大小
|
||||
private int size;
|
||||
|
||||
}else{
|
||||
QuickSortSet() {
|
||||
this.size = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* 向链表集合添加词元
|
||||
*/
|
||||
void addLexeme(Lexeme lexeme) {
|
||||
Cell newCell = new Cell(lexeme);
|
||||
if (this.size == 0) {
|
||||
this.head = newCell;
|
||||
this.tail = newCell;
|
||||
this.size++;
|
||||
|
||||
} else {
|
||||
/*if(this.tail.compareTo(newCell) == 0){//词元与尾部词元相同，不放入集合
|
||||
|
||||
}else */if(this.tail.compareTo(newCell) < 0){//词元接入链表尾部
|
||||
this.tail.next = newCell;
|
||||
newCell.prev = this.tail;
|
||||
this.tail = newCell;
|
||||
this.size++;
|
||||
}else */
|
||||
if (this.tail.compareTo(newCell) < 0) {//词元接入链表尾部
|
||||
this.tail.next = newCell;
|
||||
newCell.prev = this.tail;
|
||||
this.tail = newCell;
|
||||
this.size++;
|
||||
|
||||
}else if(this.head.compareTo(newCell) > 0){//词元接入链表头部
|
||||
this.head.prev = newCell;
|
||||
newCell.next = this.head;
|
||||
this.head = newCell;
|
||||
this.size++;
|
||||
} else if (this.head.compareTo(newCell) > 0) {//词元接入链表头部
|
||||
this.head.prev = newCell;
|
||||
newCell.next = this.head;
|
||||
this.head = newCell;
|
||||
this.size++;
|
||||
|
||||
}else{
|
||||
//从尾部上逆
|
||||
Cell index = this.tail;
|
||||
while(index != null && index.compareTo(newCell) > 0){
|
||||
index = index.prev;
|
||||
}
|
||||
} else {
|
||||
//从尾部上逆
|
||||
Cell index = this.tail;
|
||||
while (index != null && index.compareTo(newCell) > 0) {
|
||||
index = index.prev;
|
||||
}
|
||||
/*if(index.compareTo(newCell) == 0){//词元与集合中的词元重复，不放入集合
|
||||
|
||||
}else */if((index != null ? index.compareTo(newCell) : 1) < 0){//词元插入链表中的某个位置
|
||||
newCell.prev = index;
|
||||
newCell.next = index.next;
|
||||
index.next.prev = newCell;
|
||||
index.next = newCell;
|
||||
this.size++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回链表头部元素
|
||||
*/
|
||||
Lexeme peekFirst(){
|
||||
if(this.head != null){
|
||||
return this.head.lexeme;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 取出链表集合的第一个元素
|
||||
* @return Lexeme
|
||||
*/
|
||||
Lexeme pollFirst(){
|
||||
if(this.size == 1){
|
||||
Lexeme first = this.head.lexeme;
|
||||
this.head = null;
|
||||
this.tail = null;
|
||||
this.size--;
|
||||
return first;
|
||||
}else if(this.size > 1){
|
||||
Lexeme first = this.head.lexeme;
|
||||
this.head = this.head.next;
|
||||
this.size --;
|
||||
return first;
|
||||
}else{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回链表尾部元素
|
||||
*/
|
||||
Lexeme peekLast(){
|
||||
if(this.tail != null){
|
||||
return this.tail.lexeme;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 取出链表集合的最后一个元素
|
||||
* @return Lexeme
|
||||
*/
|
||||
Lexeme pollLast(){
|
||||
if(this.size == 1){
|
||||
Lexeme last = this.head.lexeme;
|
||||
this.head = null;
|
||||
this.tail = null;
|
||||
this.size--;
|
||||
return last;
|
||||
|
||||
}else if(this.size > 1){
|
||||
Lexeme last = this.tail.lexeme;
|
||||
this.tail = this.tail.prev;
|
||||
this.size--;
|
||||
return last;
|
||||
|
||||
}else{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回集合大小
|
||||
*/
|
||||
int size(){
|
||||
return this.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* 判断集合是否为空
|
||||
*/
|
||||
boolean isEmpty(){
|
||||
return this.size == 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回lexeme链的头部
|
||||
*/
|
||||
Cell getHead(){
|
||||
return this.head;
|
||||
}
|
||||
}else */
|
||||
if ((index != null ? index.compareTo(newCell) : 1) < 0) {//词元插入链表中的某个位置
|
||||
newCell.prev = index;
|
||||
newCell.next = index.next;
|
||||
index.next.prev = newCell;
|
||||
index.next = newCell;
|
||||
this.size++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* IK 中文分词 版本 7.0
|
||||
* IK Analyzer release 7.0
|
||||
* update by Magese(magese@live.cn)
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
class Cell implements Comparable<Cell>{
|
||||
private Cell prev;
|
||||
private Cell next;
|
||||
private Lexeme lexeme;
|
||||
|
||||
Cell(Lexeme lexeme){
|
||||
if(lexeme == null){
|
||||
throw new IllegalArgumentException("lexeme must not be null");
|
||||
}
|
||||
this.lexeme = lexeme;
|
||||
}
|
||||
/**
|
||||
* 返回链表头部元素
|
||||
*/
|
||||
Lexeme peekFirst() {
|
||||
if (this.head != null) {
|
||||
return this.head.lexeme;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
public int compareTo(Cell o) {
|
||||
return this.lexeme.compareTo(o.lexeme);
|
||||
}
|
||||
/**
|
||||
* 取出链表集合的第一个元素
|
||||
*
|
||||
* @return Lexeme
|
||||
*/
|
||||
Lexeme pollFirst() {
|
||||
if (this.size == 1) {
|
||||
Lexeme first = this.head.lexeme;
|
||||
this.head = null;
|
||||
this.tail = null;
|
||||
this.size--;
|
||||
return first;
|
||||
} else if (this.size > 1) {
|
||||
Lexeme first = this.head.lexeme;
|
||||
this.head = this.head.next;
|
||||
this.size--;
|
||||
return first;
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public Cell getPrev(){
|
||||
return this.prev;
|
||||
}
|
||||
|
||||
Cell getNext(){
|
||||
return this.next;
|
||||
}
|
||||
|
||||
public Lexeme getLexeme(){
|
||||
return this.lexeme;
|
||||
}
|
||||
}
|
||||
/**
|
||||
* 返回链表尾部元素
|
||||
*/
|
||||
Lexeme peekLast() {
|
||||
if (this.tail != null) {
|
||||
return this.tail.lexeme;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 取出链表集合的最后一个元素
|
||||
*
|
||||
* @return Lexeme
|
||||
*/
|
||||
Lexeme pollLast() {
|
||||
if (this.size == 1) {
|
||||
Lexeme last = this.head.lexeme;
|
||||
this.head = null;
|
||||
this.tail = null;
|
||||
this.size--;
|
||||
return last;
|
||||
|
||||
} else if (this.size > 1) {
|
||||
Lexeme last = this.tail.lexeme;
|
||||
this.tail = this.tail.prev;
|
||||
this.size--;
|
||||
return last;
|
||||
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回集合大小
|
||||
*/
|
||||
int size() {
|
||||
return this.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* 判断集合是否为空
|
||||
*/
|
||||
boolean isEmpty() {
|
||||
return this.size == 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* 返回lexeme链的头部
|
||||
*/
|
||||
Cell getHead() {
|
||||
return this.head;
|
||||
}
|
||||
|
||||
/*
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
* update by Magese(magese@live.cn)
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
static class Cell implements Comparable<Cell> {
|
||||
private Cell prev;
|
||||
private Cell next;
|
||||
private final Lexeme lexeme;
|
||||
|
||||
Cell(Lexeme lexeme) {
|
||||
if (lexeme == null) {
|
||||
throw new IllegalArgumentException("lexeme must not be null");
|
||||
}
|
||||
this.lexeme = lexeme;
|
||||
}
|
||||
|
||||
public int compareTo(Cell o) {
|
||||
return this.lexeme.compareTo(o.lexeme);
|
||||
}
|
||||
|
||||
public Cell getPrev() {
|
||||
return this.prev;
|
||||
}
|
||||
|
||||
Cell getNext() {
|
||||
return this.next;
|
||||
}
|
||||
|
||||
public Lexeme getLexeme() {
|
||||
return this.lexeme;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
@@ -226,31 +226,25 @@ public class Dictionary {
|
||||
private void loadMainDict() {
|
||||
// 建立一个主词典实例
|
||||
_MainDict = new DictSegment((char) 0);
|
||||
// 读取主词典文件
|
||||
InputStream is = this.getClass().getClassLoader().getResourceAsStream(cfg.getMainDictionary());
|
||||
if (is == null) {
|
||||
throw new RuntimeException("Main Dictionary not found!!!");
|
||||
}
|
||||
|
||||
try {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
_MainDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Main Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
|
||||
} finally {
|
||||
// 获取是否加载主词典
|
||||
if (cfg.useMainDict()) {
|
||||
// 读取主词典文件
|
||||
InputStream is = this.getClass().getClassLoader().getResourceAsStream(cfg.getMainDictionary());
|
||||
if (is == null) {
|
||||
throw new RuntimeException("Main Dictionary not found!!!");
|
||||
}
|
||||
try {
|
||||
is.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
readDict(is, _MainDict);
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Main Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
|
||||
} finally {
|
||||
try {
|
||||
is.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
}
|
||||
// 加载扩展词典
|
||||
@@ -274,17 +268,7 @@ public class Dictionary {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
// 加载扩展词典数据到主内存词典中
|
||||
// System.out.println(theWord);
|
||||
_MainDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
|
||||
readDict(is, _MainDict);
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Extension Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
@@ -319,17 +303,7 @@ public class Dictionary {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
// System.out.println(theWord);
|
||||
// 加载扩展停止词典数据到内存中
|
||||
_StopWordDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
|
||||
readDict(is, _StopWordDict);
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Extension Stop word Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
@@ -357,15 +331,7 @@ public class Dictionary {
|
||||
throw new RuntimeException("Quantifier Dictionary not found!!!");
|
||||
}
|
||||
try {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
_QuantifierDict.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
|
||||
readDict(is, _QuantifierDict);
|
||||
} catch (IOException ioe) {
|
||||
System.err.println("Quantifier Dictionary loading exception.");
|
||||
ioe.printStackTrace();
|
||||
@@ -379,4 +345,21 @@ public class Dictionary {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 读取词典文件到词典树中
|
||||
*
|
||||
* @param is 文件输入流
|
||||
* @param dictSegment 词典树分段
|
||||
* @throws IOException 读取异常
|
||||
*/
|
||||
private void readDict(InputStream is, DictSegment dictSegment) throws IOException {
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8), 512);
|
||||
String theWord;
|
||||
do {
|
||||
theWord = br.readLine();
|
||||
if (theWord != null && !"".equals(theWord.trim())) {
|
||||
dictSegment.fillSegment(theWord.trim().toLowerCase().toCharArray());
|
||||
}
|
||||
} while (theWord != null);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
@@ -38,13 +38,13 @@ public class Hit {
|
||||
private static final int MATCH = 0x00000001;
|
||||
//Hit前缀匹配
|
||||
private static final int PREFIX = 0x00000010;
|
||||
|
||||
|
||||
|
||||
|
||||
//该HIT当前状态，默认未匹配
|
||||
private int hitState = UNMATCH;
|
||||
|
||||
|
||||
//记录词典匹配过程中，当前匹配到的词典分支节点
|
||||
private DictSegment matchedDictSegment;
|
||||
private DictSegment matchedDictSegment;
|
||||
/*
|
||||
* 词段开始位置
|
||||
*/
|
||||
@@ -53,8 +53,8 @@ public class Hit {
|
||||
* 词段的结束位置
|
||||
*/
|
||||
private int end;
|
||||
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* 判断是否完全匹配
|
||||
*/
|
||||
@@ -62,7 +62,7 @@ public class Hit {
|
||||
return (this.hitState & MATCH) > 0;
|
||||
}
|
||||
/**
|
||||
*
|
||||
*
|
||||
*/
|
||||
void setMatch() {
|
||||
this.hitState = this.hitState | MATCH;
|
||||
@@ -75,7 +75,7 @@ public class Hit {
|
||||
return (this.hitState & PREFIX) > 0;
|
||||
}
|
||||
/**
|
||||
*
|
||||
*
|
||||
*/
|
||||
void setPrefix() {
|
||||
this.hitState = this.hitState | PREFIX;
|
||||
@@ -87,34 +87,34 @@ public class Hit {
|
||||
return this.hitState == UNMATCH ;
|
||||
}
|
||||
/**
|
||||
*
|
||||
*
|
||||
*/
|
||||
void setUnmatch() {
|
||||
this.hitState = UNMATCH;
|
||||
}
|
||||
|
||||
|
||||
DictSegment getMatchedDictSegment() {
|
||||
return matchedDictSegment;
|
||||
}
|
||||
|
||||
|
||||
void setMatchedDictSegment(DictSegment matchedDictSegment) {
|
||||
this.matchedDictSegment = matchedDictSegment;
|
||||
}
|
||||
|
||||
|
||||
public int getBegin() {
|
||||
return begin;
|
||||
}
|
||||
|
||||
|
||||
void setBegin(int begin) {
|
||||
this.begin = begin;
|
||||
}
|
||||
|
||||
|
||||
public int getEnd() {
|
||||
return end;
|
||||
}
|
||||
|
||||
|
||||
void setEnd(int end) {
|
||||
this.end = end;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
@@ -34,44 +34,40 @@ import org.apache.lucene.analysis.Tokenizer;
|
||||
* IK分词器，Lucene Analyzer接口实现
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
public final class IKAnalyzer extends Analyzer{
|
||||
|
||||
private boolean useSmart;
|
||||
|
||||
private boolean useSmart() {
|
||||
return useSmart;
|
||||
}
|
||||
public final class IKAnalyzer extends Analyzer {
|
||||
|
||||
public void setUseSmart(boolean useSmart) {
|
||||
this.useSmart = useSmart;
|
||||
}
|
||||
private boolean useSmart;
|
||||
|
||||
/**
|
||||
* IK分词器Lucene Analyzer接口实现类
|
||||
*
|
||||
* 默认细粒度切分算法
|
||||
*/
|
||||
public IKAnalyzer(){
|
||||
this(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* IK分词器Lucene Analyzer接口实现类
|
||||
*
|
||||
* @param useSmart 当为true时，分词器进行智能切分
|
||||
*/
|
||||
public IKAnalyzer(boolean useSmart){
|
||||
super();
|
||||
this.useSmart = useSmart;
|
||||
}
|
||||
private boolean useSmart() {
|
||||
return useSmart;
|
||||
}
|
||||
|
||||
/**
|
||||
* 重载Analyzer接口，构造分词组件
|
||||
*/
|
||||
@Override
|
||||
protected TokenStreamComponents createComponents(String fieldName) {
|
||||
Tokenizer _IKTokenizer = new IKTokenizer(this.useSmart());
|
||||
return new TokenStreamComponents(_IKTokenizer);
|
||||
}
|
||||
|
||||
/**
|
||||
* IK分词器Lucene Analyzer接口实现类
|
||||
* 默认细粒度切分算法
|
||||
*/
|
||||
public IKAnalyzer() {
|
||||
this(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* IK分词器Lucene Analyzer接口实现类
|
||||
*
|
||||
* @param useSmart 当为true时，分词器进行智能切分
|
||||
*/
|
||||
public IKAnalyzer(boolean useSmart) {
|
||||
super();
|
||||
this.useSmart = useSmart;
|
||||
}
|
||||
|
||||
/**
|
||||
* 重载Analyzer接口，构造分词组件
|
||||
*/
|
||||
@Override
|
||||
protected TokenStreamComponents createComponents(String fieldName) {
|
||||
Tokenizer _IKTokenizer = new IKTokenizer(this.useSmart());
|
||||
return new TokenStreamComponents(_IKTokenizer);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
@@ -43,88 +43,88 @@ import java.io.IOException;
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
public final class IKTokenizer extends Tokenizer {
|
||||
|
||||
//IK分词器实现
|
||||
private IKSegmenter _IKImplement;
|
||||
|
||||
//词元文本属性
|
||||
private CharTermAttribute termAtt;
|
||||
//词元位移属性
|
||||
private OffsetAttribute offsetAtt;
|
||||
//词元分类属性（该属性分类参考org.wltea.analyzer.core.Lexeme中的分类常量）
|
||||
private TypeAttribute typeAtt;
|
||||
//记录最后一个词元的结束位置
|
||||
private int endPosition;
|
||||
|
||||
/**
|
||||
* Lucene 7.6 Tokenizer适配器类构造函数
|
||||
*/
|
||||
public IKTokenizer() {
|
||||
this(false);
|
||||
}
|
||||
|
||||
IKTokenizer(boolean useSmart) {
|
||||
super();
|
||||
init(useSmart);
|
||||
}
|
||||
//IK分词器实现
|
||||
private IKSegmenter _IKImplement;
|
||||
|
||||
public IKTokenizer(AttributeFactory factory) {
|
||||
this(factory, false);
|
||||
}
|
||||
//词元文本属性
|
||||
private CharTermAttribute termAtt;
|
||||
//词元位移属性
|
||||
private OffsetAttribute offsetAtt;
|
||||
//词元分类属性（该属性分类参考org.wltea.analyzer.core.Lexeme中的分类常量）
|
||||
private TypeAttribute typeAtt;
|
||||
//记录最后一个词元的结束位置
|
||||
private int endPosition;
|
||||
|
||||
IKTokenizer(AttributeFactory factory, boolean useSmart) {
|
||||
super(factory);
|
||||
init(useSmart);
|
||||
}
|
||||
/**
|
||||
* Lucene 7.6 Tokenizer适配器类构造函数
|
||||
*/
|
||||
public IKTokenizer() {
|
||||
this(false);
|
||||
}
|
||||
|
||||
private void init(boolean useSmart) {
|
||||
offsetAtt = addAttribute(OffsetAttribute.class);
|
||||
termAtt = addAttribute(CharTermAttribute.class);
|
||||
typeAtt = addAttribute(TypeAttribute.class);
|
||||
_IKImplement = new IKSegmenter(input , useSmart);
|
||||
}
|
||||
IKTokenizer(boolean useSmart) {
|
||||
super();
|
||||
init(useSmart);
|
||||
}
|
||||
|
||||
/* (non-Javadoc)
|
||||
* @see org.apache.lucene.analysis.TokenStream#incrementToken()
|
||||
*/
|
||||
@Override
|
||||
public boolean incrementToken() throws IOException {
|
||||
//清除所有的词元属性
|
||||
clearAttributes();
|
||||
Lexeme nextLexeme = _IKImplement.next();
|
||||
if(nextLexeme != null){
|
||||
//将Lexeme转成Attributes
|
||||
//设置词元文本
|
||||
termAtt.append(nextLexeme.getLexemeText());
|
||||
//设置词元长度
|
||||
termAtt.setLength(nextLexeme.getLength());
|
||||
//设置词元位移
|
||||
offsetAtt.setOffset(nextLexeme.getBeginPosition(), nextLexeme.getEndPosition());
|
||||
//记录分词的最后位置
|
||||
endPosition = nextLexeme.getEndPosition();
|
||||
//记录词元分类
|
||||
typeAtt.setType(nextLexeme.getLexemeTypeString());
|
||||
//返会true告知还有下个词元
|
||||
return true;
|
||||
}
|
||||
//返会false告知词元输出完毕
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* (non-Javadoc)
|
||||
* @see org.apache.lucene.analysis.Tokenizer#reset(java.io.Reader)
|
||||
*/
|
||||
@Override
|
||||
public void reset() throws IOException {
|
||||
super.reset();
|
||||
_IKImplement.reset(input);
|
||||
}
|
||||
|
||||
@Override
|
||||
public final void end() {
|
||||
// set final offset
|
||||
int finalOffset = correctOffset(this.endPosition);
|
||||
offsetAtt.setOffset(finalOffset, finalOffset);
|
||||
}
|
||||
public IKTokenizer(AttributeFactory factory) {
|
||||
this(factory, false);
|
||||
}
|
||||
|
||||
IKTokenizer(AttributeFactory factory, boolean useSmart) {
|
||||
super(factory);
|
||||
init(useSmart);
|
||||
}
|
||||
|
||||
private void init(boolean useSmart) {
|
||||
offsetAtt = addAttribute(OffsetAttribute.class);
|
||||
termAtt = addAttribute(CharTermAttribute.class);
|
||||
typeAtt = addAttribute(TypeAttribute.class);
|
||||
_IKImplement = new IKSegmenter(input, useSmart);
|
||||
}
|
||||
|
||||
/* (non-Javadoc)
|
||||
* @see org.apache.lucene.analysis.TokenStream#incrementToken()
|
||||
*/
|
||||
@Override
|
||||
public boolean incrementToken() throws IOException {
|
||||
//清除所有的词元属性
|
||||
clearAttributes();
|
||||
Lexeme nextLexeme = _IKImplement.next();
|
||||
if (nextLexeme != null) {
|
||||
//将Lexeme转成Attributes
|
||||
//设置词元文本
|
||||
termAtt.append(nextLexeme.getLexemeText());
|
||||
//设置词元长度
|
||||
termAtt.setLength(nextLexeme.getLength());
|
||||
//设置词元位移
|
||||
offsetAtt.setOffset(nextLexeme.getBeginPosition(), nextLexeme.getEndPosition());
|
||||
//记录分词的最后位置
|
||||
endPosition = nextLexeme.getEndPosition();
|
||||
//记录词元分类
|
||||
typeAtt.setType(nextLexeme.getLexemeTypeString());
|
||||
//返会true告知还有下个词元
|
||||
return true;
|
||||
}
|
||||
//返会false告知词元输出完毕
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* (non-Javadoc)
|
||||
* @see org.apache.lucene.analysis.Tokenizer#reset(java.io.Reader)
|
||||
*/
|
||||
@Override
|
||||
public void reset() throws IOException {
|
||||
super.reset();
|
||||
_IKImplement.reset(input);
|
||||
}
|
||||
|
||||
@Override
|
||||
public final void end() {
|
||||
// set final offset
|
||||
int finalOffset = correctOffset(this.endPosition);
|
||||
offsetAtt.setOffset(finalOffset, finalOffset);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
@@ -74,7 +74,7 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
*/
|
||||
@Override
|
||||
public void inform(ResourceLoader resourceLoader) throws IOException {
|
||||
System.out.println(String.format("IKTokenizerFactory " + this.hashCode() + " inform conf: %s", getConf()));
|
||||
System.out.printf("IKTokenizerFactory " + this.hashCode() + " inform conf: %s%n", getConf());
|
||||
this.loader = resourceLoader;
|
||||
update();
|
||||
if ((getConf() != null) && (!getConf().trim().isEmpty())) {
|
||||
@@ -174,4 +174,4 @@ public class IKTokenizerFactory extends TokenizerFactory implements ResourceLoad
|
||||
private void setConf(String conf) {
|
||||
this.conf = conf;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
@@ -35,7 +35,7 @@ import java.util.Vector;
|
||||
*/
|
||||
public class UpdateThread implements Runnable {
|
||||
private static final long INTERVAL = 30000L; // 循环等待时间
|
||||
private Vector<UpdateJob> filterFactorys; // 更新任务集合
|
||||
private final Vector<UpdateJob> filterFactorys; // 更新任务集合
|
||||
|
||||
/**
|
||||
* 私有化构造器，阻止外部进行实例化
|
||||
@@ -51,7 +51,7 @@ public class UpdateThread implements Runnable {
|
||||
* 静态内部类，实现线程安全单例模式
|
||||
*/
|
||||
private static class Builder {
|
||||
private static UpdateThread singleton = new UpdateThread();
|
||||
private static final UpdateThread singleton = new UpdateThread();
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -81,6 +81,7 @@ public class UpdateThread implements Runnable {
|
||||
//noinspection InfiniteLoopStatement
|
||||
while (true) {
|
||||
try {
|
||||
//noinspection BusyWait
|
||||
Thread.sleep(INTERVAL);
|
||||
} catch (InterruptedException e) {
|
||||
e.printStackTrace();
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
@@ -1,86 +0,0 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
import org.apache.lucene.analysis.Analyzer;
|
||||
import org.apache.lucene.analysis.TokenStream;
|
||||
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
|
||||
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
|
||||
import org.apache.lucene.analysis.tokenattributes.TypeAttribute;
|
||||
import org.wltea.analyzer.lucene.IKAnalyzer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.StringReader;
|
||||
|
||||
/**
|
||||
* 使用IKAnalyzer进行分词的演示
|
||||
* 2012-10-22
|
||||
*/
|
||||
public class IKAnalzyerDemo {
|
||||
|
||||
public static void main(String[] args) {
|
||||
//构建IK分词器，使用smart分词模式
|
||||
Analyzer analyzer = new IKAnalyzer(true);
|
||||
|
||||
//获取Lucene的TokenStream对象
|
||||
TokenStream ts = null;
|
||||
try {
|
||||
ts = analyzer.tokenStream("myfield", new StringReader("这是一个中文分词的例子，你可以直接运行它！IKAnalyer can analysis english text too"));
|
||||
//获取词元位置属性
|
||||
OffsetAttribute offset = ts.addAttribute(OffsetAttribute.class);
|
||||
//获取词元文本属性
|
||||
CharTermAttribute term = ts.addAttribute(CharTermAttribute.class);
|
||||
//获取词元文本属性
|
||||
TypeAttribute type = ts.addAttribute(TypeAttribute.class);
|
||||
|
||||
|
||||
//重置TokenStream（重置StringReader）
|
||||
ts.reset();
|
||||
//迭代获取分词结果
|
||||
while (ts.incrementToken()) {
|
||||
System.out.println(offset.startOffset() + " - " + offset.endOffset() + " : " + term.toString() + " | " + type.type());
|
||||
}
|
||||
//关闭TokenStream（关闭StringReader）
|
||||
ts.end(); // Perform end-of-stream operations, e.g. set the final offset.
|
||||
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
} finally {
|
||||
//释放TokenStream的所有资源
|
||||
if (ts != null) {
|
||||
try {
|
||||
ts.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@@ -1,137 +0,0 @@
|
||||
/*
|
||||
* IK 中文分词 版本 7.5
|
||||
* IK Analyzer release 7.5
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 7.6版本 由 Magese (magese@live.cn) 更新
|
||||
* release 7.6 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
import org.apache.lucene.analysis.Analyzer;
|
||||
import org.apache.lucene.document.Document;
|
||||
import org.apache.lucene.document.Field;
|
||||
import org.apache.lucene.document.StringField;
|
||||
import org.apache.lucene.document.TextField;
|
||||
import org.apache.lucene.index.DirectoryReader;
|
||||
import org.apache.lucene.index.IndexReader;
|
||||
import org.apache.lucene.index.IndexWriter;
|
||||
import org.apache.lucene.index.IndexWriterConfig;
|
||||
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
|
||||
import org.apache.lucene.queryparser.classic.ParseException;
|
||||
import org.apache.lucene.queryparser.classic.QueryParser;
|
||||
import org.apache.lucene.search.IndexSearcher;
|
||||
import org.apache.lucene.search.Query;
|
||||
import org.apache.lucene.search.ScoreDoc;
|
||||
import org.apache.lucene.search.TopDocs;
|
||||
import org.apache.lucene.store.Directory;
|
||||
import org.apache.lucene.store.RAMDirectory;
|
||||
import org.wltea.analyzer.lucene.IKAnalyzer;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
|
||||
/**
|
||||
* 使用IKAnalyzer进行Lucene索引和查询的演示
|
||||
* 2012-3-2
|
||||
* <p>
|
||||
* 以下是结合Lucene4.0 API的写法
|
||||
*/
|
||||
public class LuceneIndexAndSearchDemo {
|
||||
|
||||
|
||||
/**
|
||||
* 模拟：
|
||||
* 创建一个单条记录的索引，并对其进行搜索
|
||||
*
|
||||
*/
|
||||
public static void main(String[] args) {
|
||||
//Lucene Document的域名
|
||||
String fieldName = "text";
|
||||
//检索内容
|
||||
String text = "IK Analyzer是一个结合词典分词和文法分词的中文分词开源工具包。它使用了全新的正向迭代最细粒度切分算法。";
|
||||
|
||||
//实例化IKAnalyzer分词器
|
||||
Analyzer analyzer = new IKAnalyzer(true);
|
||||
|
||||
Directory directory = null;
|
||||
IndexWriter iwriter;
|
||||
IndexReader ireader = null;
|
||||
IndexSearcher isearcher;
|
||||
try {
|
||||
//建立内存索引对象
|
||||
directory = new RAMDirectory();
|
||||
|
||||
//配置IndexWriterConfig
|
||||
IndexWriterConfig iwConfig = new IndexWriterConfig(analyzer);
|
||||
iwConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);
|
||||
iwriter = new IndexWriter(directory, iwConfig);
|
||||
//写入索引
|
||||
Document doc = new Document();
|
||||
doc.add(new StringField("ID", "10000", Field.Store.YES));
|
||||
doc.add(new TextField(fieldName, text, Field.Store.YES));
|
||||
iwriter.addDocument(doc);
|
||||
iwriter.close();
|
||||
|
||||
|
||||
//搜索过程**********************************
|
||||
//实例化搜索器
|
||||
ireader = DirectoryReader.open(directory);
|
||||
isearcher = new IndexSearcher(ireader);
|
||||
|
||||
String keyword = "中文分词工具包";
|
||||
//使用QueryParser查询分析器构造Query对象
|
||||
QueryParser qp = new QueryParser(fieldName, analyzer);
|
||||
qp.setDefaultOperator(QueryParser.AND_OPERATOR);
|
||||
Query query = qp.parse(keyword);
|
||||
System.out.println("Query = " + query);
|
||||
|
||||
//搜索相似度最高的5条记录
|
||||
TopDocs topDocs = isearcher.search(query, 5);
|
||||
System.out.println("命中：" + topDocs.totalHits);
|
||||
//输出结果
|
||||
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
|
||||
for (int i = 0; i < topDocs.totalHits; i++) {
|
||||
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
|
||||
System.out.println("内容：" + targetDoc.toString());
|
||||
}
|
||||
|
||||
} catch (ParseException | IOException e) {
|
||||
e.printStackTrace();
|
||||
} finally {
|
||||
if (ireader != null) {
|
||||
try {
|
||||
ireader.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
if (directory != null) {
|
||||
try {
|
||||
directory.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,11 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
|
||||
<properties>
|
||||
<comment>IK Analyzer 扩展配置</comment>
|
||||
<!--用户可以在这里配置自己的扩展字典 -->
|
||||
<entry key="ext_dict">ext.dic;</entry>
|
||||
|
||||
<!--用户可以在这里配置自己的扩展停止词字典-->
|
||||
<entry key="ext_stopwords">stopword.dic;</entry>
|
||||
|
||||
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
|
||||
<properties>
|
||||
<comment>IK Analyzer 扩展配置</comment>
|
||||
<!-- 配置是否加载默认词典 -->
|
||||
<entry key="use_main_dict">true</entry>
|
||||
<!-- 配置自己的扩展字典，多个用分号分隔 -->
|
||||
<entry key="ext_dict">ext.dic;</entry>
|
||||
<!-- 配置自己的扩展停止词字典，多个用分号分隔 -->
|
||||
<entry key="ext_stopwords">stopword.dic;</entry>
|
||||
</properties>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,3 +1,3 @@
|
||||
Wed Aug 01 11:21:30 CST 2018
|
||||
Wed Aug 01 00:00:00 CST 2021
|
||||
files=dynamicdic.txt
|
||||
lastupdate=0
|
||||
|
||||
Reference in New Issue
Block a user