Compare commits
30 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 87d14aedf0 | |||
| 7424f51f82 | |||
| c769eea533 | |||
| 4693e3ba27 | |||
| 4d3db0bbd4 | |||
| 6569051dc8 | |||
| e4219fe761 | |||
| 268a85ba0c | |||
| 1835b3ec37 | |||
| 041fd93005 | |||
| 676314ceaf | |||
| 4edfffe18c | |||
| 475f1227ff | |||
| 13f7a847c1 | |||
| 40c6f07871 | |||
| 82fce438bc | |||
| 0094f38d4d | |||
| 4cd5ceb312 | |||
| e347a869bc | |||
| a179929bdb | |||
| 119b9e79ec | |||
| 4ca66d11c3 | |||
| f47b41b984 | |||
| c0ddecacd8 | |||
| 07a22dff69 | |||
| de06d8b143 | |||
| d997df6164 | |||
| e7cc6db397 | |||
| 85d9de759c | |||
| df471fb310 |
@@ -15,6 +15,19 @@ As an intermediate solution, we are adding a [version value](https://github.com/
|
||||
For a longer term solution, we plan to create an OpenSearch output plugin for Logstash. This plugin *does not exist yet*, but we've included it in the compatibility matrices below based on its expected behavior.
|
||||
|
||||
|
||||
## Downloads
|
||||
|
||||
These versions of Logstash and Beats offer the best compatibility with OpenSearch. For more information, see the [compatibility matrices](#compatibility-matrices).
|
||||
|
||||
- [Logstash OSS 7.12.1](https://www.elastic.co/downloads/past-releases/logstash-oss-7-12-1)
|
||||
- [Filebeat OSS 7.12.1](https://www.elastic.co/downloads/past-releases/filebeat-oss-7-12-1)
|
||||
- [Metricbeat OSS 7.12.1](https://www.elastic.co/downloads/past-releases/metricbeat-oss-7-12-1)
|
||||
- [Packetbeat OSS 7.12.1](https://www.elastic.co/downloads/past-releases/packetbeat-oss-7-12-1)
|
||||
- [Heartbeat OSS 7.12.1](https://elastic.co/downloads/past-releases/heartbeat-oss-7-12-1)
|
||||
- [Winlogbeat OSS 7.12.1](https://www.elastic.co/downloads/past-releases/winlogbeat-oss-7-12-1)
|
||||
- [Auditbeat OSS 7.12.1](https://elastic.co/downloads/past-releases/auditbeat-oss-7-12-1)
|
||||
|
||||
|
||||
## Compatibility Matrices
|
||||
|
||||
*Italicized* cells are untested, but indicate what a value theoretically should be based on existing information.
|
||||
|
||||
@@ -9,6 +9,9 @@ nav_order: 20
|
||||
|
||||
You can use OpenSearch Dashboards to create PNG, PDF, and CSV reports. To create reports, you must have the correct permissions. For a summary of the predefined roles and the permissions they grant, see the [security plugin]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles#predefined-roles).
|
||||
|
||||
CSV reports have a non-configurable 10,000 row limit. They have no explicit size limit (e.g. in MB), but extremely large documents could cause report generation to fail with an out of memory error from the V8 JavaScript engine.
|
||||
{: .tip }
|
||||
|
||||
|
||||
## Create reports from Discovery, Visualize, or Dashboard
|
||||
|
||||
|
||||
@@ -912,6 +912,27 @@ POST _plugins/_alerting/destinations
|
||||
"host": "hooks.chime.aws"
|
||||
}
|
||||
}
|
||||
|
||||
POST _plugins/_alerting/destinations
|
||||
{
|
||||
"type": "email",
|
||||
"name": "my-email-destination",
|
||||
"email": {
|
||||
"email_account_id": "YjY7mXMBx015759_IcfW",
|
||||
"recipients": [
|
||||
{
|
||||
"type": "email_group",
|
||||
"email_group_id": "YzY-mXMBx015759_dscs"
|
||||
},
|
||||
{
|
||||
"type": "email",
|
||||
"email": "example@email.com"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
// The email_account_id and email_group_id will be the document IDs of the email_account and email_group you have created.
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
+220
-15
@@ -8,16 +8,7 @@ nav_order: 14
|
||||
|
||||
Index templates let you initialize new indices with predefined mappings and settings. For example, if you continuously index log data, you can define an index template so that all of these indices have the same number of shards and replicas.
|
||||
|
||||
---
|
||||
|
||||
#### Table of contents
|
||||
1. TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Create a template
|
||||
### Create a template
|
||||
|
||||
To create an index template, use a POST request:
|
||||
|
||||
@@ -106,7 +97,7 @@ GET logs-2020-01-01
|
||||
|
||||
Any additional indices that match this pattern---`logs-2020-01-02`, `logs-2020-01-03`, and so on---will inherit the same mappings and settings.
|
||||
|
||||
## Retrieve a template
|
||||
### Retrieve a template
|
||||
|
||||
To list all index templates:
|
||||
|
||||
@@ -138,7 +129,7 @@ To check if a specific template exists:
|
||||
HEAD _index_template/<name>
|
||||
```
|
||||
|
||||
## Configure multiple templates
|
||||
### Configure multiple templates
|
||||
|
||||
You can create multiple index templates for your indices. If the index name matches more than one template, OpenSearch merges all mappings and settings from all matching templates and applies them to the index.
|
||||
|
||||
@@ -184,7 +175,7 @@ PUT _index_template/template-02
|
||||
|
||||
Because `template-02` has a higher `priority` value, it takes precedence over `template-01` . The `logs-2020-01-02` index would have the `number_of_shards` value as 3.
|
||||
|
||||
## Delete a template
|
||||
### Delete a template
|
||||
|
||||
You can delete an index template using its name:
|
||||
|
||||
@@ -192,11 +183,225 @@ You can delete an index template using its name:
|
||||
DELETE _index_template/daily_logs
|
||||
```
|
||||
|
||||
## Composable index templates
|
||||
|
||||
Managing multiple index templates has the following challenges:
|
||||
|
||||
- If you have duplication between index templates, storing these index templates results in a bigger cluster state.
|
||||
- If you want to make a change across all your index templates, you have to manually make the change for each template.
|
||||
- If an index matches multiple templates, OpenSearch might merge the templates in an unexpected way that you discover only after an index is created.
|
||||
|
||||
You can use composable index templates to overcome these challenges. Composable index templates let you abstract common settings, mappings, and aliases into a reusable building block called a component template.
|
||||
|
||||
You can combine component templates to compose an index template.
|
||||
|
||||
Settings and mappings that you specify directly in the [create index]({{site.url}}{{site.baseurl}}/opensearch/rest-api/create-index/) request override any settings or mappings specified in an index template and its component templates.
|
||||
{: .note }
|
||||
|
||||
### Create a component template
|
||||
|
||||
Let's define two component templates---`component_template_1` and `component_template_2`:
|
||||
|
||||
#### Component template 1
|
||||
|
||||
```json
|
||||
PUT _component_template/component_template_1
|
||||
{
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Component template 2
|
||||
|
||||
```json
|
||||
PUT _component_template/component_template_2
|
||||
{
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"ip_address": {
|
||||
"type": "ip"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Use component templates to create an index template
|
||||
|
||||
When creating index templates, you need to include the component templates in a `composed_of` list.
|
||||
|
||||
OpenSearch applies the component templates in the order in which you specify them within the index template. The settings, mappings, and aliases that you specify inside the index template are applied last.
|
||||
|
||||
```json
|
||||
PUT _index_template/daily_logs
|
||||
{
|
||||
"index_patterns": [
|
||||
"logs-2020-01-*"
|
||||
],
|
||||
"template": {
|
||||
"aliases": {
|
||||
"my_logs": {}
|
||||
},
|
||||
"settings": {
|
||||
"number_of_shards": 2,
|
||||
"number_of_replicas": 1
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"timestamp": {
|
||||
"type": "date",
|
||||
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
|
||||
},
|
||||
"value": {
|
||||
"type": "double"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"priority": 200,
|
||||
"composed_of": [
|
||||
"component_template_1",
|
||||
"component_template_2"
|
||||
],
|
||||
"version": 3,
|
||||
"_meta": {
|
||||
"description": "using component templates"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If you create an index named `logs-2020-01-01`, you can see that it derives its mappings and settings from both the component templates:
|
||||
|
||||
```json
|
||||
PUT logs-2020-01-01
|
||||
GET logs-2020-01-01
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
```json
|
||||
{
|
||||
"logs-2020-01-01": {
|
||||
"aliases": {
|
||||
"my_logs": {}
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
},
|
||||
"ip_address": {
|
||||
"type": "ip"
|
||||
},
|
||||
"timestamp": {
|
||||
"type": "date",
|
||||
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
|
||||
},
|
||||
"value": {
|
||||
"type": "double"
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"index": {
|
||||
"creation_date": "1625382479459",
|
||||
"number_of_shards": "2",
|
||||
"number_of_replicas": "1",
|
||||
"uuid": "rYUlpOXDSUSuZifQLPfa5A",
|
||||
"version": {
|
||||
"created": "7100299"
|
||||
},
|
||||
"provided_name": "logs-2020-01-01"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Simulate multi-component templates
|
||||
|
||||
For index templates composed of multiple component templates, you can simulate applying a new template to verify whether the settings are applied as you expect.
|
||||
|
||||
To simulate the settings that would be applied to a specific index name:
|
||||
|
||||
```json
|
||||
POST _index_template/_simulate_index/<index_name>
|
||||
```
|
||||
|
||||
To simulate the settings that would be applied from an existing template:
|
||||
|
||||
```json
|
||||
POST _index_template/_simulate/<index_template>
|
||||
```
|
||||
|
||||
You can also specify a template definition in the simulate request:
|
||||
|
||||
```json
|
||||
POST _index_template/_simulate
|
||||
{
|
||||
"index_patterns": [
|
||||
"logs-2020-01-*"
|
||||
],
|
||||
"template": {
|
||||
"settings" : {
|
||||
"index.number_of_shards" : 3
|
||||
}
|
||||
},
|
||||
"composed_of": ["component_template_1", "component_template_2"]
|
||||
}
|
||||
```
|
||||
|
||||
The `_simulate` API returns the final settings, mappings, and aliases that will be applied to indices that match the index pattern. You can also see any overlapping templates whose configuration is superseded by the simulated template body or higher priority templates:
|
||||
|
||||
```json
|
||||
{
|
||||
"template" : {
|
||||
"settings" : {
|
||||
"index" : {
|
||||
"number_of_shards" : "3"
|
||||
}
|
||||
},
|
||||
"mappings" : {
|
||||
"properties" : {
|
||||
"@timestamp" : {
|
||||
"type" : "date"
|
||||
},
|
||||
"ip_address" : {
|
||||
"type" : "ip"
|
||||
}
|
||||
}
|
||||
},
|
||||
"aliases" : { }
|
||||
},
|
||||
"overlapping" : [
|
||||
{
|
||||
"name" : "daily_logs",
|
||||
"index_patterns" : [
|
||||
"logs-2020-01-*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Index template options
|
||||
|
||||
You can specify the following template options:
|
||||
|
||||
Option | Type | Description | Required
|
||||
:--- | :--- | :--- | :---
|
||||
`priority` | `Number` | The priority of the index template. | No
|
||||
`create` | `Boolean` | Whether this index template should replace an existing one. | No
|
||||
`template` | `Object` | Specify index settings, mappings, and aliases. | No
|
||||
`priority` | `Integer` | The priority of the index template. | No
|
||||
`composed_of` | `String array` | The names of component templates applied on a new index together with the current template. | No
|
||||
`version` | `Integer` | Specify a version number to simplify template management. Default is `null`. | No
|
||||
`_meta ` | `Object` | Specify meta information about the template. | No
|
||||
|
||||
@@ -7,10 +7,11 @@ nav_order: 5
|
||||
|
||||
# Docker security configuration
|
||||
|
||||
Before deploying to a production environment, you should replace the demo security certificates and configuration YAML files with your own. With the tarball, you have direct access to the file system, but the Docker image requires modifying the Docker storage volumes include the replacement files.
|
||||
Before deploying to a production environment, you should replace the demo security certificates and configuration YAML files with your own. With the tarball, you have direct access to the file system, but the Docker image requires modifying the Docker storage volumes to include the replacement files.
|
||||
|
||||
Additionally, you can set the Docker environment variable `DISABLE_INSTALL_DEMO_CONFIG` to `true`. This change completely disables the demo installer.
|
||||
|
||||
|
||||
## Sample Docker Compose file
|
||||
|
||||
```yml
|
||||
@@ -141,6 +142,7 @@ If you encounter any `File /usr/share/opensearch/config/opensearch.yml has insec
|
||||
|
||||
Finally, you can reach OpenSearch Dashboards at http://localhost:5601, sign in, and use the **Security** panel to perform other management tasks.
|
||||
|
||||
|
||||
## Using certificates with Docker
|
||||
|
||||
To use your own certificates in your configuration, add all of the necessary certificates to the volumes section of the Docker Compose file:
|
||||
|
||||
@@ -150,7 +150,7 @@ networks:
|
||||
opensearch-net:
|
||||
```
|
||||
|
||||
If you override `opensearch_dashboards.yml` settings using environment variables, as seen above, use all uppercase letters and periods in place of underscores (e.g. for `opensearch.url`, specify `OPENSEARCH_URL`).
|
||||
If you override `opensearch_dashboards.yml` settings using environment variables, as seen above, use all uppercase letters and periods in place of underscores (e.g. for `opensearch.hosts`, use `OPENSEARCH_HOSTS`).
|
||||
{: .note}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
layout: default
|
||||
title: Alias
|
||||
parent: REST API reference
|
||||
grand_parent: OpenSearch
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
# Alias
|
||||
|
||||
An alias is a virtual pointer that you can use to reference one or more indices. Creating and updating aliases are atomic operations, so you can reindex your data and point an alias at it without any downtime.
|
||||
|
||||
|
||||
## Example
|
||||
|
||||
```json
|
||||
POST _aliases
|
||||
{
|
||||
"actions": [
|
||||
{
|
||||
"add": {
|
||||
"index": "movies",
|
||||
"alias": "movies-alias1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"remove": {
|
||||
"index": "old-index",
|
||||
"alias": "old-index-alias"
|
||||
}
|
||||
}
|
||||
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Path and HTTP methods
|
||||
|
||||
```
|
||||
POST _aliases
|
||||
```
|
||||
|
||||
## URL parameters
|
||||
|
||||
All alias parameters are optional.
|
||||
|
||||
Parameter | Data Type | Description
|
||||
:--- | :--- | :---
|
||||
master_timeout | Time | The amount of time to wait for a response from the master node. Default is `30s`.
|
||||
timeout | Time | The amount of time to wait for a response from the cluster. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
||||
In your request body, you need to specify what action to take, the alias name, and the index you want to associate with the alias. Other fields are optional.
|
||||
|
||||
Field | Data Type | Description | Required
|
||||
:--- | :--- | :--- | :---
|
||||
actions | Array | Set of actions you want to perform on the index. Valid options are: `add`, `remove`, and `remove_index`. You must have at least one action in the array. | Yes
|
||||
add | N/A | Adds an alias to the specified index. | No
|
||||
remove | N/A | Removes an alias from the specified index. | No
|
||||
remove_index | N/A | Deletes an index. | No
|
||||
index | String | Name of the index you want to associate with the alias. Supports wildcard expressions. | Yes if you don't supply an `indices` field in the body.
|
||||
indices | Array | Array of index names you want to associate with the alias. | Yes if you don't supply an `index` field in the body.
|
||||
alias | String | The name of the alias. | Yes if you don't supply an `aliases` field in the body.
|
||||
aliases | Array | Array of alias names. | Yes if you don't supply an `alias` field in the body.
|
||||
filter | Object | A filter to use with the alias, so the alias points to a filtered part of the index. | No
|
||||
is_hidden | Boolean | Specifies whether the alias should be hidden from results that include wildcard expressions | No
|
||||
must_exist | Boolean | Specifies whether the alias to remove must exist. | No
|
||||
is_write_index | Boolean | Specifies whether the index should be a write index. An alias can only have one write index at a time. | No
|
||||
routing | String | Used to assign a custom value to a shard for specific operations. | No
|
||||
index_routing | String | Assigns a custom value to a shard only for index operations. | No
|
||||
search_routing | String | Assigns a custom value to a shard only for search operations. | No
|
||||
|
||||
## Response
|
||||
|
||||
```json
|
||||
{
|
||||
"acknowledged": true
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
layout: default
|
||||
title: Create index
|
||||
parent: REST API reference
|
||||
grand_parent: OpenSearch
|
||||
nav_order: 7
|
||||
---
|
||||
|
||||
# Create index
|
||||
|
||||
While you can create an index by using a document as a base, you can also just create an empty index for use later.
|
||||
|
||||
## Example
|
||||
|
||||
The following example demonstrates how to create an index with a non-default number of primary and replica shards, specifies that `age` is of type `integer`, and assigns a `sample-alias1` alias to the index.
|
||||
|
||||
```json
|
||||
PUT /sample-index1
|
||||
|
||||
{
|
||||
"settings": {
|
||||
"index": {
|
||||
"number_of_shards": 2,
|
||||
"number_of_replicas": 1
|
||||
}
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"age": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"aliases": {
|
||||
"sample-alias1": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Path and HTTP methods
|
||||
|
||||
```
|
||||
PUT <index-name>
|
||||
```
|
||||
|
||||
## Index naming restrictions
|
||||
|
||||
OpenSearch indices have the following naming restrictions:
|
||||
|
||||
- All letters must be lowercase.
|
||||
- Index names can't begin with underscores (`_`) or hyphens (`-`).
|
||||
- Index names can't contain spaces, commas, or the following characters:
|
||||
|
||||
`:`, `"`, `*`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<`
|
||||
|
||||
## URL parameters
|
||||
|
||||
You can include the following URL parameters in your request. All parameters are optional.
|
||||
|
||||
Parameter | Type | Description
|
||||
:--- | :--- | :---
|
||||
include_type_name | Boolean | If `true`, the request expects a type in the body of mappings. Because OpenSearch indices all have a type of `_doc`, we recommend that this parameter is left as the default of `false.`
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to `all` or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
||||
As part of your request, you can supply parameters in your request's body that specify index settings, mappings, and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index. The following sections provide more information about index settings and mappings.
|
||||
|
||||
|
||||
### Index settings
|
||||
|
||||
Index settings are separated into two varieties: static index settings and dynamic index settings. Static index settings are settings that you specify at index creation and can't change later. You can change dynamic settings at any time, including at index creation.
|
||||
|
||||
#### Static index settings
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_shards | The number of primary shards in the index. Default is 1.
|
||||
index.number_of_routing_shards | The number of routing shards used to split an index.
|
||||
index.shard.check_on_startup | Whether the index's shards should be checked for corruption. Available options are `false` (do not check for corruption), `checksum` (check for physical corruption), and `true` (check for both physical and logical corruption). Default is `false`.
|
||||
index.codec | The compression type to use to compress stored data. Available values are `best_compression` and `default`.
|
||||
index.routing_partition_size | The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than just a single shard. To enable, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1.
|
||||
index.soft_deletes_enabled | Whether to retain a history of operations performed on the index.
|
||||
index.soft_deletes.retention_lease.period | The maximum amount of time to retain a shard's history of operations. Default is `12h`.
|
||||
index.load_fixed_bitset_filters_eagerly | Indicates whether cached filters should be pre-loaded for nested queries. Available options are `true` and `false`. Default is `true`.
|
||||
index.hidden | Whether the index should be hidden. Hidden indices are not returned as part of queries that have wildcards. Available options are `true` and `false`. Default is `false`.
|
||||
|
||||
#### Dynamic index Settings
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_replicas | The number of replica shards each primary shard should have. For example, if you have 4 primary shards and set `index.number_of_replicas` to 3, the index has 12 replica shards. Default is 1.
|
||||
index.auto_expand_replicas | Automatically expands the number of replicas based on the amount of data nodes in the cluster. Specify a lower bound and upper limit (for example, 0-9), or `all` for the upper limit. For example, if you have 5 data nodes and set `index.auto_expand_replicas` to 0-3, then the cluster does not autoamtically add another replica shard. However, if you set this value to `0-all` and add 2 more nodes for a total of 7, the cluster will expand to now have 6 replica shards. Default is disabled.
|
||||
index.search.idle.after | Amount of time a shard should wait for a search or get request until it goes idle. Default is `30s`.
|
||||
index.refresh_interval | How often the index should refresh, which publishes its most recent changes and makes them available for searching. Can be set to `-1` to disable refreshing. Default is `1s`.
|
||||
index.max_result_window | The maximum value of `from` + `size` for searches to the index. `from` is the starting index to search from, and `size` is the amount of results to return. Default: 10000.
|
||||
index.max_inner_result_window | The maximum value of `from` + `size` for inner hits definitions and top hits aggregations to the index. `from` is the starting index to search from, and `size` is the amount of top hits to return. Default is 100.
|
||||
index.max_rescore_window | The maximum value of `window_size` for rescore requests to the index. Rescore requests reorder the index's documents and return a new score, which can be more precise. Default is the same as index.max_inner_result_window or 10000 by default.
|
||||
index.max_docvalue_fields_search | Maximum amount of `docvalue_fields` allowed in a query. Default is 100.
|
||||
index.max_script_fields | Maximum amount of `script_fields` allowed in a query. Default is 32.
|
||||
index.max_ngram_diff | Maximum allowed difference between `min_gram` and `max_gram` values for `NGramTokenizer` and `NGramTokenFilter`. Default is 1.
|
||||
index.max_shingle_diff | Maximum allowed difference between `max_shingle_size` and `min_shingle_size` for the `shingle` token filter. Default is 3.
|
||||
index.max_refresh_listeners | Maximum amount of refresh listeners available on the index's shards.
|
||||
index.analyze.max_token_count | Maximum amount of tokens that can return from the `_analyze` API operation. Default is 10000.
|
||||
index.highlight.max_analyzed_offset | The maximum amount of characters that will be analyzed in a highlight request. Default is 1000000.
|
||||
index.max_terms_count | The maximum amount of terms that can be used in a terms query. Default is 65536.
|
||||
index.max_regex_length | The maximum character length of regex that can be in a regexp query. Default is 1000.
|
||||
index.query.default_field | A field or list of fields that OpenSearch uses in queries in case a field isn't specified in the parameters.
|
||||
index.routing.allocation.enable | Specifies options for the index's shard allocation. Available options are `all` (allow shard allocation for all shards), `primaries` (allow shard allocation only for primary shards), `new_primaries` (allow shard allocaiton only for new primary shards), and `none` (do not allow shard allocation). Default is `all`.
|
||||
index.routing.rebalance.enable | Enables shard rebalancing for the index. Available options are `all` (allow shard rebalancing for all shards), `primaries`, (allow shard rebalancing only for primary shards), `replicas` (allow shard rebalancing only for replicas), and `none` (do not allow shard rebalancing). Default is `all`.
|
||||
index.gc_deletes | Amount of time to retain a deleted document's version number. Default is `60s`.
|
||||
index.default_pipeline | The default ingest node pipeline for the index. If the default pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
index.final_pipeline | The final ingest node pipeline for the index. If the final pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
|
||||
### Mappings
|
||||
|
||||
Mappings define how a documents and its fields are stored and indexed. If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and their fields. However, if you know exactly what types your data fall under and want to enforce that standard, then you can use explicit mappings.
|
||||
|
||||
For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. Using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers.
|
||||
|
||||
#### Dynamic mapping types
|
||||
|
||||
Type | Description
|
||||
:--- | :---
|
||||
null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values.
|
||||
boolean | OpenSearch accepts `true` and `false` as boolean values. An empty string is equal to `false.`
|
||||
float | A single-precision 32-bit floating point number.
|
||||
double | A double-precision 64-bit floating point number.
|
||||
integer | A signed 32-bit number.
|
||||
object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
|
||||
array | Arrays in OpenSearch can only store values of one type, such as an array of just integers or strings. Empty arrays are treated as though they are fields with no values.
|
||||
text | A string sequence of characters that represent full-text values.
|
||||
keyword | A string sequence of structured characters, such as an email or ZIP code.
|
||||
date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date.
|
||||
numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, `unsigned_long`. Default is disabled.
|
||||
|
||||
#### Explicit mapping
|
||||
|
||||
If you know exactly what your data's typings need to be, you can specify them in your request body when creating your index.
|
||||
|
||||
```json
|
||||
{
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"year": { "type" : "text" },
|
||||
"age": { "type" : "integer" },
|
||||
"director":{ "type" : "text" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Response
|
||||
```json
|
||||
{
|
||||
"acknowledged": true,
|
||||
"shards_acknowledged": true,
|
||||
"index": "sample-index1"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
layout: default
|
||||
title: Scroll
|
||||
parent: REST API reference
|
||||
nav_order: 120
|
||||
---
|
||||
|
||||
# Scroll
|
||||
|
||||
You can use the `scroll` operation to retrieve a large number of results. For example, for machine learning jobs, you can request an unlimited number of results in batches.
|
||||
|
||||
To use the `scroll` operation, add a `scroll` parameter to the request header with a search context to tell OpenSearch how long you need to keep scrolling. This search context needs to be long enough to process a single batch of results.
|
||||
|
||||
Because search contexts consume a lot of memory, we suggest you don't use the `scroll` operation for frequent user queries. Instead, use the `sort` parameter with the `search_after` parameter to scroll responses for user queries.
|
||||
{: .note }
|
||||
|
||||
## Example
|
||||
|
||||
To set the number of results that you want returned for each batch, use the `size` parameter:
|
||||
|
||||
```json
|
||||
GET shakespeare/_search?scroll=10m
|
||||
{
|
||||
"size": 10000
|
||||
}
|
||||
```
|
||||
|
||||
OpenSearch caches the results and returns a scroll ID to access them in batches:
|
||||
|
||||
```json
|
||||
"_scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ=="
|
||||
```
|
||||
|
||||
Pass this scroll ID to the `scroll` operation to get back the next batch of results:
|
||||
|
||||
```json
|
||||
GET _search/scroll
|
||||
{
|
||||
"scroll": "10m",
|
||||
"scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ=="
|
||||
}
|
||||
```
|
||||
|
||||
Using this scroll ID, you get results in batches of 10,000 as long as the search context is still open. Typically, the scroll ID does not change between requests, but it *can* change, so make sure to always use the latest scroll ID. If you don't send the next scroll request within the set search context, the `scroll` operation does not return any results.
|
||||
|
||||
If you expect billions of results, use a sliced scroll. Slicing allows you to perform multiple scroll operations for the same request, but in parallel.
|
||||
Set the ID and the maximum number of slices for the scroll:
|
||||
|
||||
```json
|
||||
GET shakespeare/_search?scroll=10m
|
||||
{
|
||||
"slice": {
|
||||
"id": 0,
|
||||
"max": 10
|
||||
},
|
||||
"query": {
|
||||
"match_all": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With a single scroll ID, you get back 10 results.
|
||||
You can have up to 10 IDs.
|
||||
|
||||
Close the search context when you’re done scrolling, because the `scroll` operation continues to consume computing resources until the timeout:
|
||||
|
||||
```json
|
||||
DELETE _search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAcWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ==
|
||||
```
|
||||
|
||||
To close all open scroll contexts:
|
||||
|
||||
```json
|
||||
DELETE _search/scroll/_all
|
||||
```
|
||||
|
||||
The `scroll` operation corresponds to a specific timestamp. It doesn't consider documents added after that timestamp as potential results.
|
||||
|
||||
|
||||
## Path and HTTP methods
|
||||
|
||||
```
|
||||
GET _search/scroll
|
||||
POST _search/scroll
|
||||
```
|
||||
```
|
||||
GET _search/scroll/<scroll-id>
|
||||
POST _search/scroll/<scroll-id>
|
||||
```
|
||||
|
||||
## URL parameters
|
||||
|
||||
All scroll parameters are optional.
|
||||
|
||||
Parameter | Type | Description
|
||||
:--- | :--- | :---
|
||||
scroll | Time | Specifies the amount of time the search context is maintained.
|
||||
scroll_id | String | The scroll ID for the search.
|
||||
rest_total_hits_as_int | Boolean | Whether the `hits.total` property is returned as an integer or an object. Default is false.
|
||||
|
||||
## Response
|
||||
|
||||
```json
|
||||
{
|
||||
"succeeded": true,
|
||||
"num_freed": 1
|
||||
}
|
||||
```
|
||||
@@ -16,10 +16,10 @@ Asynchronous search in OpenSearch lets you send search requests that run in the
|
||||
|
||||
## REST API
|
||||
|
||||
To perform an asynchronous search, send requests to `_opensearch/_asynchronous_search`, with your query in the request body:
|
||||
To perform an asynchronous search, send requests to `_plugins/_asynchronous_search`, with your query in the request body:
|
||||
|
||||
```json
|
||||
POST _opensearch/_asynchronous_search
|
||||
POST _plugins/_asynchronous_search
|
||||
```
|
||||
|
||||
You can specify the following options.
|
||||
@@ -33,7 +33,7 @@ Options | Description | Default value | Required
|
||||
#### Sample request
|
||||
|
||||
```json
|
||||
POST _opensearch/_asynchronous_search/?pretty&size=10&wait_for_completion_timeout=1ms&keep_on_completion=true&request_cache=false
|
||||
POST _plugins/_asynchronous_search/?pretty&size=10&wait_for_completion_timeout=1ms&keep_on_completion=true&request_cache=false
|
||||
{
|
||||
"aggs": {
|
||||
"city": {
|
||||
@@ -116,7 +116,7 @@ Options | Description
|
||||
After you submit an asynchronous search request, you can request partial responses with the ID that you see in the asynchronous search response.
|
||||
|
||||
```json
|
||||
GET _opensearch/_asynchronous_search/<ID>?pretty
|
||||
GET _plugins/_asynchronous_search/<ID>?pretty
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
@@ -192,7 +192,7 @@ For asynchronous searches with `keep_on_completion` as `true` and a sufficiently
|
||||
You can use the DELETE API operation to delete any ongoing asynchronous search by its ID. If the search is still running, it’s canceled. If the search is complete, the saved search results are deleted.
|
||||
|
||||
```json
|
||||
DELETE _opensearch/_asynchronous_search/<ID>?pretty
|
||||
DELETE _plugins/_asynchronous_search/<ID>?pretty
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
@@ -208,7 +208,7 @@ DELETE _opensearch/_asynchronous_search/<ID>?pretty
|
||||
You can use the stats API operation to monitor asynchronous searches that are running, completed, and/or persisted.
|
||||
|
||||
```json
|
||||
GET _opensearch/_asynchronous_search/stats
|
||||
GET _plugins/_asynchronous_search/stats
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
@@ -21,7 +21,7 @@ PUT _cluster/settings
|
||||
}
|
||||
```
|
||||
|
||||
Similarly, you can also update the settings by sending the request to the plugin setting endpoing `_plugins/_query/setting`:
|
||||
Similarly, you can also update the settings by sending the request to the plugin setting endpoint `_plugins/_query/setting`:
|
||||
```json
|
||||
PUT _plugins/_query/settings
|
||||
{
|
||||
@@ -37,4 +37,4 @@ Setting | Default | Description
|
||||
`plugins.sql.slowlog` | 2 seconds | Configure the time limit (in seconds) for slow queries. The plugin logs slow queries as `Slow query: elapsed=xxx (ms)` in `opensearch.log`.
|
||||
`plugins.sql.cursor.keep_alive` | 1 minute | This value configures how long the cursor context is kept open. Cursor contexts are resource heavy, so we recommend a low value.
|
||||
`plugins.query.memory_limit` | 85% | This setting configures the heap memory usage limit for the circuit breaker of the query engine.
|
||||
`plugins.query.size_limit` | 200 | The setting sets the default size of index that the query engine fetches from OpenSearch.
|
||||
`plugins.query.size_limit` | 200 | The setting sets the default size of index that the query engine fetches from OpenSearch.
|
||||
|
||||
Reference in New Issue
Block a user