Compare commits

..

2 Commits

Author SHA1 Message Date
keithhc2 dc2f349d7e Addressed comment and added note about users and roles 2021-07-28 13:21:25 -07:00
keithhc2 30df98e078 Added whitelist.yml 2021-07-28 11:41:08 -07:00
6 changed files with 66 additions and 72 deletions
-2
View File
@@ -1,5 +1,3 @@
<img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">
# OpenSearch documentation
This repository contains the documentation for OpenSearch, the search, analytics, and visualization suite with advanced security, alerting, SQL support, automated index management, deep performance analysis, and more. You can find the rendered documentation at [opensearch.org/docs](https://opensearch.org/docs).
-4
View File
@@ -160,10 +160,6 @@ POST _plugins/_ism/add/index_1
}
```
If you use a wildcard `*` while adding a policy to an index, the ISM plugin interprets `*` as all indices, including system indices like `.opendistro-security`, which stores users, roles, and tenants. A delete action in your policy might accidentally delete all user roles and tenants in your cluster.
Don't use the broad `*` wildcard, and instead add a prefix, such as `my-logs*`, when specifying indices with the `_ism/add` API.
{: .warning }
---
+11 -2
View File
@@ -12,7 +12,16 @@ If you're ingesting continuously generated time-series data such as logs, events
- You dont need to update older documents.
- Your searches generally target the newer documents.
A typical workflow to manage time-series data consists of setting up an alias, configuring a rollover operation, defining a write index, and creating common mapping fields in an index template. Data streams simplifies this process.
A typical workflow to manage time-series data is as follows:
- To split your data into an index for each day, use the rollover operation.
- To perform searches on a virtual index name that gets expanded to the underlying indices, create an [index alias]({{site.url}}{{site.baseurl}}/opensearch/index-alias/).
- To perform a write operation on an index alias, configure the latest index as the write index.
- To configure new indices, extract common mappings and settings into an [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/).
Even after you perform all these operations, youre still not enforcing the best practices when dealing with time-series data. For example, you can modify the indices directly. Youre able to ingest documents without a timestamp field, which might result in slower queries.
Data streams abstract the complexity and enforce the best practices for managing time-series data.
With data streams, you can store append-only time-series data across multiple indices with a single endpoint for ingesting and searching data. We recommend using data streams in place of index aliases for time-series data.
@@ -267,7 +276,7 @@ To manage data streams from OpenSearch Dashboards, open **OpenSearch Dashboards*
You see a toggle switch for data streams that you can use to show or hide indices belonging to a data stream.
When you enable this switch, you see a data stream multi-select dropdown menu that you can use for filtering data streams.
You also see a data stream column that shows you the name of the data stream the index is contained in.
You also see a data stream column that shows you the name of the parent data stream the index is contained in.
![data stream toggle]({{site.url}}{{site.baseurl}}/images/data_streams_toggle.png)
-60
View File
@@ -1,60 +0,0 @@
---
layout: default
title: Date math support for index names
nav_order: 92
---
# Date math support for index names
Date math is shorthand arithmetic for finding relative dates.
If you're indexing time-series data with the dates mapped in the index names, you can use date math in your queries to filter index names and limit the number of searched indices.
## Date math syntax
The date math syntax for an index name is as follows:
```
<static_name{date_math_expr{date_format|time_zone}}>
```
- `static_name`: The unchanged or static portion of the index name. To use the characters `{` and `}` in the static part of an index name, escape them with a backslash `\`.
- `date_math_expr`: The changing or dynamic portion of the index name thats computed by the date math expression. For example, `now+1h` adds one hour, `now-1d` subtracts one hour, and `now/d` rounds down to the nearest day, where `now` represents the current timestamp.
- `date_format`: (Optional) Specify the format for the computed date. The default value is `YYYY.MM.dd`. Make sure that youre using the correct small or capital letters in the date format. For example, `mm` denotes minute of hour, while `MM` denotes month of year. Similarly, `hh` denotes the hour in the `1-12` range in combination with AM/PM, while `HH` denotes the hour in the `0-23` 24-hour range.
- `time_zone`: (Optional) Specify the timezone offset. The default value is UTC. For example, the UTC time offset for PST is `-08:00`.
## Date math example
You must enclose date math index names within angle brackets.
If today is 22nd March, 2024:
- `<logstash-{now/d}>` resolves to `logstash-2024.03.22`
- `<logstash-{now/M}>` resolves to `logstash-2024.03.01`
- `<logstash-{now/M{YYYY.MM}}>` resolves to `logstash-2024.03`
- `<logstash-{now/M-1M{YYYY.MM}}>` resolves to `logstash-2024.02`
- `<logstash-{now/d{yyyy.MM.dd|+12:00}}>` resolves to `logstash-2024.03.23`
You need to encode all special characters in URI format:
Special characters | URI format
:--- | :---
`<` | %3C
`>` | %3E
`/` | %2F
`{` | %7B
`}` | %7D
`|` | %7C
`+` | %2B
`:` | %3A
`,` | %2C
`\` | %5C
If you are searching for errors in your daily logs with the default Logstash index name format `logstash-YYYY.MM.dd`, you can use date math to restrict the search to indices of the past three days:
```
# GET <logstash-{now/d-2d}>,<logstash-{now/d-1d}>,<logstash-{now/d}>/_search
GET %3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogstash-%7Bnow%2Fd%7D%3E/_search
```
This date math expression is evaluated at runtime.
+2 -2
View File
@@ -7,13 +7,13 @@ nav_order: 70
# Important settings
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. Even if you use the Docker image, set this value on the *host machine*. To check the current value, run this command:
For production workloads, make sure the [Linux setting](https://www.kernel.org/doc/Documentation/sysctl/vm.txt) `vm.max_map_count` is set to at least 262144. On the OpenSearch Docker image, this setting is the default. To check, start a Bash session in the container and run:
```bash
cat /proc/sys/vm/max_map_count
```
To increase the value, add the following line to `/etc/sysctl.conf`:
To increase this value, you have to modify the Docker image. For other install types, add this setting to the host machine's `/etc/sysctl.conf` file with the following line:
```
vm.max_map_count=262144
+53 -2
View File
@@ -121,8 +121,59 @@ If you want to run your users' passwords against some validation, specify a regu
Note that OpenSearch validates only users and passwords created through OpenSearch Dashboards or the REST API.
```yml
plugins.restapi.password_validation_regex: '(?=.*[A-Z])(?=.*[^a-zA-Z\d])(?=.*[0-9])(?=.*[a-z]).{8,}'
plugins.restapi.password_validation_error_message: "Password must be minimum 8 characters long and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character."
plugins.security.restapi.password_validation_regex: '(?=.*[A-Z])(?=.*[^a-zA-Z\d])(?=.*[0-9])(?=.*[a-z]).{8,}'
plugins.security.restapi.password_validation_error_message: "Password must be minimum 8 characters long and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character."
```
## whitelist.yml
You can use `whitelist.yml` to allow list any endpoints and HTTP requests. If enabled, all users except the SuperAdmin are allowed access to only the specified endpoints and HTTP requests, and all other HTTP requests associated with the endpoint are denied. For example, if GET `_cluster/settings` is allow listed, users cannot submit PUT requests to `_cluster/settings` to update cluster settings.
Note that while you can configure access to endpoints this way, for most cases, it is still best to configure permissions using the security plugin's users and roles, which have more granular settings.
```yml
---
_meta:
type: "whitelist"
config_version: 2
# Description:
# enabled - feature flag.
# if enabled is false, all endpoints are accessible.
# if enabled is true, all users except the SuperAdmin can only submit the allowed requests to the specified endpoints.
# SuperAdmin can access all APIs.
# SuperAdmin is defined by the SuperAdmin certificate, which is configured with the opensearch.yml setting plugins.security.authcz.admin_dn:
# Refer to the example setting in opensearch.yml to learn more about configuring SuperAdmin.
#
# requests - map of allow listed endpoints and HTTP requests
#this name must be config
config:
enabled: true
requests:
/_cluster/settings:
- GET
/_cat/nodes:
- GET
```
To enable PUT requests to cluster settings, add PUT to the list of allowed operations under `/_cluster/settings`.
```yml
requests:
/_cluster/settings:
- GET
- PUT
```
You can also allow list custom indices. `whitelist.yml` doesn't support wildcards, so you must manually specify all of the indices you want to allow list.
```yml
requests: # Only allow GET requests to /sample-index1/_doc/1 and /sample-index2/_doc/1
/sample-index1/_doc/1:
- GET
/sample-index2/_doc/1:
- GET
```