Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Magese
|
0c6560fe61 | ||
|
Magese
|
bf3df0e58c | ||
|
老王
|
4180900415 | ||
|
gaozhicheng
|
eaf58c0f1d | ||
|
gaozhicheng
|
aba7aeff19 | ||
|
gaozhicheng
|
a234d047cd | ||
|
gaozhicheng
|
8d67193774 |
@@ -13,8 +13,10 @@ ik-analyzer for solr 7.x-8.x
|
||||
<!-- /Badges section end. -->
|
||||
|
||||
## 简介
|
||||
#### 适配最新版本solr 7&8；
|
||||
#### 扩展IK原有词库：
|
||||
**适配最新版本solr 7&8；**
|
||||
|
||||
**扩展IK原有词库：**
|
||||
|
||||
| 分词工具 | 词库中词的数量 | 最后更新时间 |
|
||||
| :------: | :------: | :------: |
|
||||
| ik | 27.5万 | 2012年 |
|
||||
@@ -23,15 +25,18 @@ ik-analyzer for solr 7.x-8.x
|
||||
| jieba | 58.4万 | 2012年 |
|
||||
| jcesg | 16.6万 | 2018年 |
|
||||
| sougou词库 | 115.2万 | 2020年 |
|
||||
#### 将以上词库进行整理后约187.1万条词汇；
|
||||
#### 添加动态加载词典表功能，在不需要重启solr服务的情况下加载新增的词典。
|
||||
|
||||
**将以上词库进行整理后约187.1万条词汇；**
|
||||
|
||||
**添加动态加载词典表功能，在不需要重启solr服务的情况下加载新增的词典。**
|
||||
|
||||
> <small>关闭默认主词典请在`IKAnalyzer.cfg.xml`配置文件中设置`use_main_dict`为`false`。</small>
|
||||
* IKAnalyzer的原作者为林良益<linliangyi2007@gmail.com>，项目网站为<http://code.google.com/p/ik-analyzer>
|
||||
* 该项目动态加载功能根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改，其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
> * IKAnalyzer的原作者为林良益<linliangyi2007@gmail.com>，项目网站为<http://code.google.com/p/ik-analyzer>
|
||||
> * 该项目动态加载功能根据博主[@星火燎原智勇](http://www.cnblogs.com/liang1101/articles/6395016.html)的博客进行修改，其GITHUB地址为[@liang68](https://github.com/liang68)
|
||||
|
||||
|
||||
## 使用说明
|
||||
* jar包下载地址：[](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/8.3.1/ik-analyzer-8.3.1.jar)
|
||||
* jar包下载地址：[](https://search.maven.org/remotecontent?filepath=com/github/magese/ik-analyzer/8.5.0/ik-analyzer-8.5.0.jar)
|
||||
* 历史版本：[](https://search.maven.org/search?q=g:com.github.magese%20AND%20a:ik-analyzer&core=gav)
|
||||
|
||||
```xml
|
||||
@@ -39,7 +44,7 @@ ik-analyzer for solr 7.x-8.x
|
||||
<dependency>
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>8.3.1</version>
|
||||
<version>8.5.0</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
@@ -101,34 +106,36 @@ ik-analyzer for solr 7.x-8.x
|
||||
|
||||
|
||||
## 更新说明
|
||||
- `2020-12-30:`
|
||||
- **2021-12-23:** 升级lucene版本为`8.5.0`
|
||||
- **2021-03-22:** 升级lucene版本为`8.4.0`
|
||||
- **2020-12-30:**
|
||||
- 升级lucene版本为`8.3.1`
|
||||
- 更新词库
|
||||
- `2019-11-12:`
|
||||
- **2019-11-12:**
|
||||
- 升级lucene版本为`8.3.0`
|
||||
- `IKAnalyzer.cfg.xml`增加配置项`use_main_dict`，用于配置是否启用默认主词典
|
||||
- `2019-09-27:` 升级lucene版本为`8.2.0`
|
||||
- `2019-07-11:` 升级lucene版本为`8.1.1`
|
||||
- `2019-05-27:`
|
||||
- **2019-09-27:** 升级lucene版本为`8.2.0`
|
||||
- **2019-07-11:** 升级lucene版本为`8.1.1`
|
||||
- **2019-05-27:**
|
||||
- 升级lucene版本为`8.1.0`
|
||||
- 优化原词典部分重复词语
|
||||
- 更新搜狗2019最新流行词汇词典，约20k词汇量
|
||||
- `2019-05-15:` 升级lucene版本为`8.0.0`，并支持Solr8使用
|
||||
- `2019-03-01:` 升级lucene版本为`7.7.1`
|
||||
- `2019-02-15:` 升级lucene版本为`7.7.0`
|
||||
- `2018-12-26:`
|
||||
- **2019-05-15:** 升级lucene版本为`8.0.0`，并支持Solr8使用
|
||||
- **2019-03-01:** 升级lucene版本为`7.7.1`
|
||||
- **2019-02-15:** 升级lucene版本为`7.7.0`
|
||||
- **2018-12-26:**
|
||||
- 升级lucene版本为`7.6.0`
|
||||
- 兼容solr-cloud，动态词典配置文件及动态词典可交由`zookeeper`进行管理
|
||||
- 动态词典增加注释功能，以`#`开头的行将视为注释
|
||||
- `2018-12-04:` 整理更新词库列表`magese.dic`
|
||||
- `2018-10-10:` 升级lucene版本为`7.5.0`
|
||||
- `2018-09-03:` 优化注释与输出信息，取消部分中文输出避免不同字符集乱码，现会打印被调用inform方法的hashcode
|
||||
- `2018-08-23: `
|
||||
- **2018-12-04:** 整理更新词库列表`magese.dic`
|
||||
- **2018-10-10:** 升级lucene版本为`7.5.0`
|
||||
- **2018-09-03:** 优化注释与输出信息，取消部分中文输出避免不同字符集乱码，现会打印被调用inform方法的hashcode
|
||||
- **2018-08-23:**
|
||||
- 完善了动态更新词库代码注释；
|
||||
- 将ik.conf配置文件中的lastUpdate属性改为long类型，现已支持时间戳形式
|
||||
- `2018-08-13:` 更新maven仓库地址
|
||||
- `2018-08-01:` 移除默认的扩展词与停用词
|
||||
- `2018-07-23:` 升级lucene版本为`7.4.0`
|
||||
- **2018-08-13:** 更新maven仓库地址
|
||||
- **2018-08-01:** 移除默认的扩展词与停用词
|
||||
- **2018-07-23:** 升级lucene版本为`7.4.0`
|
||||
|
||||
|
||||
## 感谢 Thanks
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
<groupId>com.github.magese</groupId>
|
||||
<artifactId>ik-analyzer</artifactId>
|
||||
<version>8.3.1</version>
|
||||
<version>8.5.0</version>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<name>ik-analyzer-solr</name>
|
||||
@@ -13,20 +13,13 @@
|
||||
|
||||
<properties>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
<lucene.version>8.3.1</lucene.version>
|
||||
<lucene.version>8.5.0</lucene.version>
|
||||
<javac.src.version>1.8</javac.src.version>
|
||||
<javac.target.version>1.8</javac.target.version>
|
||||
<maven.compiler.plugin.version>3.3</maven.compiler.plugin.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<version>4.13.1</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.lucene</groupId>
|
||||
<artifactId>lucene-core</artifactId>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.cfg;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,7 +21,7 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.core;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -176,8 +176,8 @@ class QuickSortSet {
|
||||
}
|
||||
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
* update by Magese(magese@live.cn)
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.dic;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.lucene;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
* IK 中文分词 版本 8.5.0
|
||||
* IK Analyzer release 8.5.0
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
@@ -21,8 +21,8 @@
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
* 8.5.0版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.5.0 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.query;
|
||||
|
||||
@@ -1,86 +0,0 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
import org.apache.lucene.analysis.Analyzer;
|
||||
import org.apache.lucene.analysis.TokenStream;
|
||||
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
|
||||
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
|
||||
import org.apache.lucene.analysis.tokenattributes.TypeAttribute;
|
||||
import org.wltea.analyzer.lucene.IKAnalyzer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.StringReader;
|
||||
|
||||
/**
|
||||
* 使用IKAnalyzer进行分词的演示
|
||||
* 2012-10-22
|
||||
*/
|
||||
public class IKAnalzyerDemo {
|
||||
|
||||
public static void main(String[] args) {
|
||||
//构建IK分词器，使用smart分词模式
|
||||
Analyzer analyzer = new IKAnalyzer(true);
|
||||
|
||||
//获取Lucene的TokenStream对象
|
||||
TokenStream ts = null;
|
||||
try {
|
||||
ts = analyzer.tokenStream("myfield", new StringReader("这是一个中文分词的例子，你可以直接运行它！IKAnalyer can analysis english text too"));
|
||||
//获取词元位置属性
|
||||
OffsetAttribute offset = ts.addAttribute(OffsetAttribute.class);
|
||||
//获取词元文本属性
|
||||
CharTermAttribute term = ts.addAttribute(CharTermAttribute.class);
|
||||
//获取词元文本属性
|
||||
TypeAttribute type = ts.addAttribute(TypeAttribute.class);
|
||||
|
||||
|
||||
//重置TokenStream（重置StringReader）
|
||||
ts.reset();
|
||||
//迭代获取分词结果
|
||||
while (ts.incrementToken()) {
|
||||
System.out.println(offset.startOffset() + " - " + offset.endOffset() + " : " + term.toString() + " | " + type.type());
|
||||
}
|
||||
//关闭TokenStream（关闭StringReader）
|
||||
ts.end(); // Perform end-of-stream operations, e.g. set the final offset.
|
||||
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
} finally {
|
||||
//释放TokenStream的所有资源
|
||||
if (ts != null) {
|
||||
try {
|
||||
ts.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@@ -1,136 +0,0 @@
|
||||
/*
|
||||
* IK 中文分词 版本 8.3.1
|
||||
* IK Analyzer release 8.3.1
|
||||
*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*
|
||||
* 源代码由林良益(linliangyi2005@gmail.com)提供
|
||||
* 版权声明 2012，乌龙茶工作室
|
||||
* provided by Linliangyi and copyright 2012 by Oolong studio
|
||||
*
|
||||
* 8.3.1版本 由 Magese (magese@live.cn) 更新
|
||||
* release 8.3.1 update by Magese(magese@live.cn)
|
||||
*
|
||||
*/
|
||||
package org.wltea.analyzer.sample;
|
||||
|
||||
import org.apache.lucene.analysis.Analyzer;
|
||||
import org.apache.lucene.document.Document;
|
||||
import org.apache.lucene.document.Field;
|
||||
import org.apache.lucene.document.StringField;
|
||||
import org.apache.lucene.document.TextField;
|
||||
import org.apache.lucene.index.DirectoryReader;
|
||||
import org.apache.lucene.index.IndexReader;
|
||||
import org.apache.lucene.index.IndexWriter;
|
||||
import org.apache.lucene.index.IndexWriterConfig;
|
||||
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
|
||||
import org.apache.lucene.queryparser.classic.ParseException;
|
||||
import org.apache.lucene.queryparser.classic.QueryParser;
|
||||
import org.apache.lucene.search.*;
|
||||
import org.apache.lucene.store.Directory;
|
||||
import org.apache.lucene.store.RAMDirectory;
|
||||
import org.wltea.analyzer.lucene.IKAnalyzer;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
|
||||
/**
|
||||
* 使用IKAnalyzer进行Lucene索引和查询的演示
|
||||
* 2012-3-2
|
||||
* <p>
|
||||
* 以下是结合Lucene4.0 API的写法
|
||||
*/
|
||||
public class LuceneIndexAndSearchDemo {
|
||||
|
||||
|
||||
/**
|
||||
* 模拟：
|
||||
* 创建一个单条记录的索引，并对其进行搜索
|
||||
*
|
||||
*/
|
||||
public static void main(String[] args) {
|
||||
//Lucene Document的域名
|
||||
String fieldName = "text";
|
||||
//检索内容
|
||||
String text = "IK Analyzer是一个结合词典分词和文法分词的中文分词开源工具包。它使用了全新的正向迭代最细粒度切分算法。";
|
||||
|
||||
//实例化IKAnalyzer分词器
|
||||
Analyzer analyzer = new IKAnalyzer(true);
|
||||
|
||||
Directory directory = null;
|
||||
IndexWriter iwriter;
|
||||
IndexReader ireader = null;
|
||||
IndexSearcher isearcher;
|
||||
try {
|
||||
//建立内存索引对象
|
||||
//noinspection deprecation
|
||||
directory = new RAMDirectory();
|
||||
|
||||
//配置IndexWriterConfig
|
||||
IndexWriterConfig iwConfig = new IndexWriterConfig(analyzer);
|
||||
iwConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);
|
||||
iwriter = new IndexWriter(directory, iwConfig);
|
||||
//写入索引
|
||||
Document doc = new Document();
|
||||
doc.add(new StringField("ID", "10000", Field.Store.YES));
|
||||
doc.add(new TextField(fieldName, text, Field.Store.YES));
|
||||
iwriter.addDocument(doc);
|
||||
iwriter.close();
|
||||
|
||||
|
||||
//搜索过程**********************************
|
||||
//实例化搜索器
|
||||
ireader = DirectoryReader.open(directory);
|
||||
isearcher = new IndexSearcher(ireader);
|
||||
|
||||
String keyword = "中文分词工具包";
|
||||
//使用QueryParser查询分析器构造Query对象
|
||||
QueryParser qp = new QueryParser(fieldName, analyzer);
|
||||
qp.setDefaultOperator(QueryParser.AND_OPERATOR);
|
||||
Query query = qp.parse(keyword);
|
||||
System.out.println("Query = " + query);
|
||||
|
||||
//搜索相似度最高的5条记录
|
||||
TopDocs topDocs = isearcher.search(query, 5);
|
||||
long totalHits = topDocs.totalHits.value;
|
||||
System.out.println("命中：" + totalHits);
|
||||
//输出结果
|
||||
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
|
||||
for (int i = 0; i < totalHits; i++) {
|
||||
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
|
||||
System.out.println("内容：" + targetDoc.toString());
|
||||
}
|
||||
|
||||
} catch (ParseException | IOException e) {
|
||||
e.printStackTrace();
|
||||
} finally {
|
||||
if (ireader != null) {
|
||||
try {
|
||||
ireader.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
if (directory != null) {
|
||||
try {
|
||||
directory.close();
|
||||
} catch (IOException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,3 @@
|
||||
Wed Aug 01 11:21:30 CST 2018
|
||||
Wed Aug 01 00:00:00 CST 2021
|
||||
files=dynamicdic.txt
|
||||
lastupdate=0
|
||||
|
||||
Reference in New Issue
Block a user