add pretrained model description and saving disk description (#5383)
* add description and saving disk Signed-off-by: xinyual <[email protected]> * update Signed-off-by: xinyual <[email protected]> * update Signed-off-by: xinyual <[email protected]> * Update _ml-commons-plugin/pretrained-models.md Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: xinyual <[email protected]> * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: xinyual <[email protected]> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: xinyual <[email protected]> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: xinyual <[email protected]> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: xinyual <[email protected]> * Moved recommended model choice options outside the table Signed-off-by: Fanit Kolchina <[email protected]> * Add a link for more info Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: xinyual <[email protected]> Signed-off-by: xinyual <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
This commit is contained in:
xinyual
co-authored by
kolchfa-aws
Nathan Bower
Fanit Kolchina
kolchfa-aws
Nathan Bower
Fanit Kolchina
parent
49889b44f0
commit
242c35db59
@@ -80,11 +80,17 @@ The following table provides a list of sentence transformer models and artifact
|
||||
|
||||
Sparse encoding models transfer text into a sparse vector and convert the vector to a list of `<token: weight>` pairs representing the text entry and its corresponding weight in the sparse vector. You can use these models for use cases such as clustering or sparse neural search.
|
||||
|
||||
We recommend the following models for optimal performance:
|
||||
|
||||
- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` model during both ingestion and search.
|
||||
- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` model during ingestion and the
|
||||
`amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` model during search.
|
||||
|
||||
The following table provides a list of sparse encoding models and artifact links you can use to download them.
|
||||
|
||||
| Model name | Auto-truncation | TorchScript artifact |
|
||||
|---|---|
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-v1-1.0.0-torch_script.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.0/torch_script/config.json) |
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-doc-v1-1.0.0-torch_script.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/config.json) |
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.0/torch_script/opensearch-neural-sparse-tokenizer-v1-1.0.0.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.0/torch_script/config.json) |
|
||||
| Model name | Auto-truncation | TorchScript artifact | Description |
|
||||
|---|---|---|
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-v1-1.0.0-torch_script.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.0/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `<entry, weight>` pairs, where each entry corresponds to a non-zero element index. |
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/opensearch-neural-sparse-encoding-doc-v1-1.0.0-torch_script.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.0/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `<entry, weight>` pairs, where each entry corresponds to a non-zero element index. |
|
||||
| `amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.0/torch_script/opensearch-neural-sparse-tokenizer-v1-1.0.0.zip)<br>- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.0/torch_script/config.json) | A neural sparse tokenizer model. The model tokenizes text into tokens and assigns each token a predefined weight, which is the token's IDF (if the IDF file is not provided, the weight defaults to 1). For more information, see [Uploading your own model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/#uploading-your-own-model). |
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ Field | Data type | Required/Optional | Description
|
||||
:--- | :--- | :---
|
||||
`query_text` | String | Required | The query text from which to generate vector embeddings.
|
||||
`model_id` | String | Required | The ID of the sparse encoding model or tokenizer model that will be used to generate vector embeddings from the query text. The model must be deployed in OpenSearch before it can be used in sparse neural search. For more information, see [Using custom models within OpenSearch]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/) and [Semantic search]({{site.url}}{{site.baseurl}}/ml-commons-plugin/semantic-search/).
|
||||
`max_token_score` | Float | Optional | The theoretical upper bound of the score for all tokens in the vocabulary (required for performance optimization).
|
||||
`max_token_score` | Float | Optional | The theoretical upper bound of the score for all tokens in the vocabulary (required for performance optimization). For OpenSearch-provided [pretrained sparse embedding models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sparse-encoding-models), we recommend setting `max_token_score` to 2 for `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` and to 3.5 for `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1`.
|
||||
|
||||
#### Example request
|
||||
|
||||
|
||||
@@ -83,6 +83,38 @@ PUT /my-nlp-index
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
To save disk space, you can exclude the embedding vector from the source as follows:
|
||||
|
||||
```json
|
||||
PUT /my-nlp-index
|
||||
{
|
||||
"settings": {
|
||||
"default_pipeline": "nlp-ingest-pipeline-sparse"
|
||||
},
|
||||
"mappings": {
|
||||
"_source": {
|
||||
"excludes": [
|
||||
"passage_embedding"
|
||||
]
|
||||
},
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "text"
|
||||
},
|
||||
"passage_embedding": {
|
||||
"type": "rank_features"
|
||||
},
|
||||
"passage_text": {
|
||||
"type": "text"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
Once the `<token, weight>` pairs are excluded from the source, they cannot be recovered. Before applying this optimization, make sure you don't need the `<token, weight>` pairs for your application.
|
||||
{: .important}
|
||||
|
||||
## Step 3: Ingest documents into the index
|
||||
|
||||
|
||||
Reference in New Issue
Block a user