Compare commits

...

9 Commits

Author SHA1 Message Date
ashwinkumar12345 a9bde64480 incorporated feedback 2021-07-28 00:27:37 -07:00
ashwinkumar12345 01df7f0b53 added date math for index names 2021-07-26 11:30:02 -07:00
aetter 76e3b2710e Update important-settings.md 2021-07-23 12:34:17 -07:00
Miki 37cab900a5 Merge pull request #110 from AMoo-Miki/add-readme-logo
Add themed logo to README
2021-07-21 10:19:06 -07:00
Andrew Etter 6c9a46c130 Merge pull request #108 from opensearch-project/data_stream_feedback
incorporated data streams feedback
2021-07-21 10:02:17 -07:00
Miki 15a5f18fb7 Add themed logo to README
Signed-off-by: Miki <mehranb@amazon.com>
2021-07-20 10:28:25 -07:00
ashwinkumar12345 4453173942 minor fix 2021-07-20 00:42:16 -07:00
ashwinkumar12345 23981ecc33 incorporated feedback 2021-07-19 23:31:33 -07:00
Andrew Etter a99458c41b Merge pull request #107 from opensearch-project/ism_warning
added ISM warning
2021-07-19 13:17:29 -07:00
4 changed files with 66 additions and 13 deletions
+2
View File
@@ -1,3 +1,5 @@
<img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">
# OpenSearch documentation
This repository contains the documentation for OpenSearch, the search, analytics, and visualization suite with advanced security, alerting, SQL support, automated index management, deep performance analysis, and more. You can find the rendered documentation at [opensearch.org/docs](https://opensearch.org/docs).
+2 -11
View File
@@ -12,16 +12,7 @@ If you're ingesting continuously generated time-series data such as logs, events
- You dont need to update older documents.
- Your searches generally target the newer documents.
A typical workflow to manage time-series data is as follows:
- To split your data into an index for each day, use the rollover operation.
- To perform searches on a virtual index name that gets expanded to the underlying indices, create an [index alias]({{site.url}}{{site.baseurl}}/opensearch/index-alias/).
- To perform a write operation on an index alias, configure the latest index as the write index.
- To configure new indices, extract common mappings and settings into an [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/).
Even after you perform all these operations, youre still not enforcing the best practices when dealing with time-series data. For example, you can modify the indices directly. Youre able to ingest documents without a timestamp field, which might result in slower queries.
Data streams abstract the complexity and enforce the best practices for managing time-series data.
A typical workflow to manage time-series data consists of setting up an alias, configuring a rollover operation, defining a write index, and creating common mapping fields in an index template. Data streams simplifies this process.
With data streams, you can store append-only time-series data across multiple indices with a single endpoint for ingesting and searching data. We recommend using data streams in place of index aliases for time-series data.
@@ -276,7 +267,7 @@ To manage data streams from OpenSearch Dashboards, open **OpenSearch Dashboards*
You see a toggle switch for data streams that you can use to show or hide indices belonging to a data stream.
When you enable this switch, you see a data stream multi-select dropdown menu that you can use for filtering data streams.
You also see a data stream column that shows you the name of the parent data stream the index is contained in.
You also see a data stream column that shows you the name of the data stream the index is contained in.
![data stream toggle]({{site.url}}{{site.baseurl}}/images/data_streams_toggle.png)
+60
View File
@@ -0,0 +1,60 @@
---
layout: default
title: Date math support for index names
nav_order: 92
---
# Date math support for index names
Date math is shorthand arithmetic for finding relative dates.
If you're indexing time-series data with the dates mapped in the index names, you can use date math in your queries to filter index names and limit the number of searched indices.
## Date math syntax
The date math syntax for an index name is as follows:
```
<static_name{date_math_expr{date_format|time_zone}}>
```
- `static_name`: The unchanged or static portion of the index name. To use the characters `{` and `}` in the static part of an index name, escape them with a backslash `\`.
- `date_math_expr`: The changing or dynamic portion of the index name thats computed by the date math expression. For example, `now+1h` adds one hour, `now-1d` subtracts one hour, and `now/d` rounds down to the nearest day, where `now` represents the current timestamp.
- `date_format`: (Optional) Specify the format for the computed date. The default value is `YYYY.MM.dd`. Make sure that youre using the correct small or capital letters in the date format. For example, `mm` denotes minute of hour, while `MM` denotes month of year. Similarly, `hh` denotes the hour in the `1-12` range in combination with AM/PM, while `HH` denotes the hour in the `0-23` 24-hour range.
- `time_zone`: (Optional) Specify the timezone offset. The default value is UTC. For example, the UTC time offset for PST is `-08:00`.
## Date math example
You must enclose date math index names within angle brackets.
If today is 22nd March, 2024:
- `<logstash-{now/d}>` resolves to `logstash-2024.03.22`
- `<logstash-{now/M}>` resolves to `logstash-2024.03.01`
- `<logstash-{now/M{YYYY.MM}}>` resolves to `logstash-2024.03`
- `<logstash-{now/M-1M{YYYY.MM}}>` resolves to `logstash-2024.02`
- `<logstash-{now/d{yyyy.MM.dd|+12:00}}>` resolves to `logstash-2024.03.23`
You need to encode all special characters in URI format:
Special characters | URI format
:--- | :---
`<` | %3C
`>` | %3E
`/` | %2F
`{` | %7B
`}` | %7D
`|` | %7C
`+` | %2B
`:` | %3A
`,` | %2C
`\` | %5C
If you are searching for errors in your daily logs with the default Logstash index name format `logstash-YYYY.MM.dd`, you can use date math to restrict the search to indices of the past three days:
```
# GET <logstash-{now/d-2d}>,<logstash-{now/d-1d}>,<logstash-{now/d}>/_search
GET %3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogstash-%7Bnow%2Fd%7D%3E/_search
```
This date math expression is evaluated at runtime.
+2 -2
View File
@@ -7,13 +7,13 @@ nav_order: 70
# Important settings
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. On the OpenSearch Docker image, this setting is the default. To check, start a Bash session in the container and run:
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. Even if you use the Docker image, set this value on the *host machine*. To check the current value, run this command:
```bash
cat /proc/sys/vm/max_map_count
```
To increase this value, you have to modify the Docker image. For other install types, add this setting to the host machine's `/etc/sysctl.conf` file with the following line:
To increase the value, add the following line to `/etc/sysctl.conf`:
```
vm.max_map_count=262144