Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a9bde64480 | |||
| 01df7f0b53 | |||
| 76e3b2710e | |||
| 37cab900a5 | |||
| 6c9a46c130 | |||
| 15a5f18fb7 | |||
| 4453173942 | |||
| 23981ecc33 | |||
| a99458c41b | |||
| b93af9a94b | |||
| 25421e60e0 |
@@ -1,3 +1,5 @@
|
|||||||
|
<img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">
|
||||||
|
|
||||||
# OpenSearch documentation
|
# OpenSearch documentation
|
||||||
|
|
||||||
This repository contains the documentation for OpenSearch, the search, analytics, and visualization suite with advanced security, alerting, SQL support, automated index management, deep performance analysis, and more. You can find the rendered documentation at [opensearch.org/docs](https://opensearch.org/docs).
|
This repository contains the documentation for OpenSearch, the search, analytics, and visualization suite with advanced security, alerting, SQL support, automated index management, deep performance analysis, and more. You can find the rendered documentation at [opensearch.org/docs](https://opensearch.org/docs).
|
||||||
|
|||||||
@@ -160,6 +160,10 @@ POST _plugins/_ism/add/index_1
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
If you use a wildcard `*` while adding a policy to an index, the ISM plugin interprets `*` as all indices, including system indices like `.opendistro-security`, which stores users, roles, and tenants. A delete action in your policy might accidentally delete all user roles and tenants in your cluster.
|
||||||
|
Don't use the broad `*` wildcard, and instead add a prefix, such as `my-logs*`, when specifying indices with the `_ism/add` API.
|
||||||
|
{: .warning }
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -12,16 +12,7 @@ If you're ingesting continuously generated time-series data such as logs, events
|
|||||||
- You don’t need to update older documents.
|
- You don’t need to update older documents.
|
||||||
- Your searches generally target the newer documents.
|
- Your searches generally target the newer documents.
|
||||||
|
|
||||||
A typical workflow to manage time-series data is as follows:
|
A typical workflow to manage time-series data consists of setting up an alias, configuring a rollover operation, defining a write index, and creating common mapping fields in an index template. Data streams simplifies this process.
|
||||||
|
|
||||||
- To split your data into an index for each day, use the rollover operation.
|
|
||||||
- To perform searches on a virtual index name that gets expanded to the underlying indices, create an [index alias]({{site.url}}{{site.baseurl}}/opensearch/index-alias/).
|
|
||||||
- To perform a write operation on an index alias, configure the latest index as the write index.
|
|
||||||
- To configure new indices, extract common mappings and settings into an [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/).
|
|
||||||
|
|
||||||
Even after you perform all these operations, you’re still not enforcing the best practices when dealing with time-series data. For example, you can modify the indices directly. You’re able to ingest documents without a timestamp field, which might result in slower queries.
|
|
||||||
|
|
||||||
Data streams abstract the complexity and enforce the best practices for managing time-series data.
|
|
||||||
|
|
||||||
With data streams, you can store append-only time-series data across multiple indices with a single endpoint for ingesting and searching data. We recommend using data streams in place of index aliases for time-series data.
|
With data streams, you can store append-only time-series data across multiple indices with a single endpoint for ingesting and searching data. We recommend using data streams in place of index aliases for time-series data.
|
||||||
|
|
||||||
@@ -276,7 +267,7 @@ To manage data streams from OpenSearch Dashboards, open **OpenSearch Dashboards*
|
|||||||
You see a toggle switch for data streams that you can use to show or hide indices belonging to a data stream.
|
You see a toggle switch for data streams that you can use to show or hide indices belonging to a data stream.
|
||||||
|
|
||||||
When you enable this switch, you see a data stream multi-select dropdown menu that you can use for filtering data streams.
|
When you enable this switch, you see a data stream multi-select dropdown menu that you can use for filtering data streams.
|
||||||
You also see a data stream column that shows you the name of the parent data stream the index is contained in.
|
You also see a data stream column that shows you the name of the data stream the index is contained in.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Date math support for index names
|
||||||
|
nav_order: 92
|
||||||
|
---
|
||||||
|
|
||||||
|
# Date math support for index names
|
||||||
|
|
||||||
|
Date math is shorthand arithmetic for finding relative dates.
|
||||||
|
|
||||||
|
If you're indexing time-series data with the dates mapped in the index names, you can use date math in your queries to filter index names and limit the number of searched indices.
|
||||||
|
|
||||||
|
## Date math syntax
|
||||||
|
|
||||||
|
The date math syntax for an index name is as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
<static_name{date_math_expr{date_format|time_zone}}>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `static_name`: The unchanged or static portion of the index name. To use the characters `{` and `}` in the static part of an index name, escape them with a backslash `\`.
|
||||||
|
- `date_math_expr`: The changing or dynamic portion of the index name that’s computed by the date math expression. For example, `now+1h` adds one hour, `now-1d` subtracts one hour, and `now/d` rounds down to the nearest day, where `now` represents the current timestamp.
|
||||||
|
- `date_format`: (Optional) Specify the format for the computed date. The default value is `YYYY.MM.dd`. Make sure that you’re using the correct small or capital letters in the date format. For example, `mm` denotes minute of hour, while `MM` denotes month of year. Similarly, `hh` denotes the hour in the `1-12` range in combination with AM/PM, while `HH` denotes the hour in the `0-23` 24-hour range.
|
||||||
|
- `time_zone`: (Optional) Specify the timezone offset. The default value is UTC. For example, the UTC time offset for PST is `-08:00`.
|
||||||
|
|
||||||
|
## Date math example
|
||||||
|
|
||||||
|
You must enclose date math index names within angle brackets.
|
||||||
|
|
||||||
|
If today is 22nd March, 2024:
|
||||||
|
|
||||||
|
- `<logstash-{now/d}>` resolves to `logstash-2024.03.22`
|
||||||
|
- `<logstash-{now/M}>` resolves to `logstash-2024.03.01`
|
||||||
|
- `<logstash-{now/M{YYYY.MM}}>` resolves to `logstash-2024.03`
|
||||||
|
- `<logstash-{now/M-1M{YYYY.MM}}>` resolves to `logstash-2024.02`
|
||||||
|
- `<logstash-{now/d{yyyy.MM.dd|+12:00}}>` resolves to `logstash-2024.03.23`
|
||||||
|
|
||||||
|
You need to encode all special characters in URI format:
|
||||||
|
|
||||||
|
Special characters | URI format
|
||||||
|
:--- | :---
|
||||||
|
`<` | %3C
|
||||||
|
`>` | %3E
|
||||||
|
`/` | %2F
|
||||||
|
`{` | %7B
|
||||||
|
`}` | %7D
|
||||||
|
`|` | %7C
|
||||||
|
`+` | %2B
|
||||||
|
`:` | %3A
|
||||||
|
`,` | %2C
|
||||||
|
`\` | %5C
|
||||||
|
|
||||||
|
If you are searching for errors in your daily logs with the default Logstash index name format `logstash-YYYY.MM.dd`, you can use date math to restrict the search to indices of the past three days:
|
||||||
|
|
||||||
|
```
|
||||||
|
# GET <logstash-{now/d-2d}>,<logstash-{now/d-1d}>,<logstash-{now/d}>/_search
|
||||||
|
GET %3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogstash-%7Bnow%2Fd%7D%3E/_search
|
||||||
|
```
|
||||||
|
|
||||||
|
This date math expression is evaluated at runtime.
|
||||||
@@ -7,13 +7,13 @@ nav_order: 70
|
|||||||
|
|
||||||
# Important settings
|
# Important settings
|
||||||
|
|
||||||
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. On the OpenSearch Docker image, this setting is the default. To check, start a Bash session in the container and run:
|
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. Even if you use the Docker image, set this value on the *host machine*. To check the current value, run this command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cat /proc/sys/vm/max_map_count
|
cat /proc/sys/vm/max_map_count
|
||||||
```
|
```
|
||||||
|
|
||||||
To increase this value, you have to modify the Docker image. For other install types, add this setting to the host machine's `/etc/sysctl.conf` file with the following line:
|
To increase the value, add the following line to `/etc/sysctl.conf`:
|
||||||
|
|
||||||
```
|
```
|
||||||
vm.max_map_count=262144
|
vm.max_map_count=262144
|
||||||
|
|||||||
Reference in New Issue
Block a user