diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 00000000..ee246a32 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,2 @@ +# This should match the owning team set up in https://github.com/orgs/opensearch-project/teams +* @opensearch-project/docs \ No newline at end of file diff --git a/.github/workflows/dco.yml b/.github/workflows/dco.yml new file mode 100644 index 00000000..d4c9ff2c --- /dev/null +++ b/.github/workflows/dco.yml @@ -0,0 +1,18 @@ +name: Developer Certificate of Origin Check + +on: [pull_request] + +jobs: + check: + runs-on: ubuntu-latest + + steps: + - name: Get PR Commits + id: 'get-pr-commits' + uses: tim-actions/get-pr-commits@v1.1.0 + with: + token: ${{ secrets.GITHUB_TOKEN }} + - name: DCO Check + uses: tim-actions/dco@v1.1.0 + with: + commits: ${{ steps.get-pr-commits.outputs.commits }} \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c4b6a1c5..7b13633c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ # Contributing Guidelines -Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional +Thank you for your interest in contributing to the OpenSource documentation! Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary @@ -9,7 +9,7 @@ information to effectively respond to your bug report or contribution. ## Reporting Bugs/Feature Requests -We welcome you to use the GitHub issue tracker to report bugs or suggest features. +Use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: diff --git a/Gemfile b/Gemfile index 53d29eb7..2dc06281 100644 --- a/Gemfile +++ b/Gemfile @@ -29,4 +29,4 @@ end gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw, :jruby] # Performance-booster for watching directories on Windows -gem "wdm", "~> 0.1.0" if Gem.win_platform? +gem "wdm", "~> 0.1.0" if Gem.win_platform? \ No newline at end of file diff --git a/README.md b/README.md index a9bd14e2..7dcff830 100644 --- a/README.md +++ b/README.md @@ -26,10 +26,9 @@ Community contributions remain essential in keeping this documentation comprehen If you encounter problems or have questions when contributing to the documentation, these people can help: -- [aetter](https://github.com/aetter) -- [ashwinkumar12345](https://github.com/ashwinkumar12345) - [keithhc2](https://github.com/keithhc2) -- [snyder114](https://github.com/snyder114) +- [Naarcha-AWS](https://github.com/Naarcha-AWS) +- [lizsnyder](https://github.com/lizsnyder) ## How the website works @@ -77,6 +76,8 @@ If you just need to fix a typo or add a sentence, this web-based method works we 1. Make your changes. +1. Sign off on the commit by including the text "Signed-off by: " in the optional description. Be sure to use an email that's added to your GitHub account. + 1. Choose **Create a new branch for this commit and start a pull request** and **Commit changes**. @@ -94,7 +95,7 @@ If you want to add a few paragraphs across multiple files and are comfortable wi 1. Edit the Markdown files in `/docs`. -1. Commit, push your changes to your fork, and submit a pull request. +1. Commit, [sign off](https://github.com/src-d/guide/blob/9171d013c648236c39faabcad8598be3c0cf8f56/developer-community/fix-DCO.md#how-to-prevent-missing-sign-offs-in-the-future), push your changes to your fork, and submit a pull request. ### Major changes @@ -143,7 +144,7 @@ If you're making major changes to the documentation and need to see the rendered 1. When you save a file, marvel as Jekyll automatically rebuilds the site and refreshes your web browser. This process can take anywhere from 10-30 seconds. -1. When you're happy with how everything looks, commit, push your changes to your fork, and submit a pull request. +1. When you're happy with how everything looks, commit, [sign off](https://github.com/src-d/guide/blob/9171d013c648236c39faabcad8598be3c0cf8f56/developer-community/fix-DCO.md#how-to-prevent-missing-sign-offs-in-the-future), push your changes to your fork, and submit a pull request. ## Writing tips diff --git a/_clients/agents-and-ingestion-tools/index.md b/_clients/agents-and-ingestion-tools/index.md index 04adfb1e..4eab146a 100644 --- a/_clients/agents-and-ingestion-tools/index.md +++ b/_clients/agents-and-ingestion-tools/index.md @@ -1,7 +1,7 @@ --- layout: default title: Agents and ingestion tools -nav_order: 100 +nav_order: 140 has_children: false has_toc: false redirect_from: @@ -12,9 +12,16 @@ redirect_from: Historically, many multiple popular agents and ingestion tools have worked with Elasticsearch OSS, such as Beats, Logstash, Fluentd, FluentBit, and OpenTelemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility. -As an intermediate compatibility solution, OpenSearch has a setting that instructs the cluster to return version 7.10.2 rather than its actual version. +Previously, an intermediate compatibility solution was available. OpenSearch had a setting that instructed the cluster to return version 7.10.2 rather than its actual version. -If you use clients that include a version check, such as recent versions of Logstash OSS or Filebeat OSS, enable the setting: +The override main response setting `compatibility.override_main_response_version` is deprecated from OpenSearch version 1.x and removed from OpenSearch 2.0.0. This setting is no longer supported for compatibility with legacy clients. +{: .note} + + +Logstash OSS 8.0 introduces a breaking change where all plugins run in ECS compatibility mode by default. If you use a compatible [OSS client](#compatibility-matrices) you must override the default value to maintain legacy behavior: +```yml +ecs_compatibility => disabled +``` ## Downloads @@ -58,13 +70,13 @@ Some users report compatibility issues with ingest pipelines on these versions o ### Compatibility Matrix for Logstash -| | Logstash OSS 7.x to 7.11.x | Logstash OSS 7.12.x\* | Logstash 7.13.x without OpenSearch output plugin | Logstash 7.13.x with OpenSearch output plugin | -| :---| :--- | :--- | :--- | :--- | -| Elasticsearch OSS 7.x to 7.9.x | *Yes* | *Yes* | *No* | *Yes* | -| Elasticsearch OSS 7.10.2 | *Yes* | *Yes* | *No* | *Yes* | -| ODFE 1.x to 1.12 | *Yes* | *Yes* | *No* | *Yes* | -| ODFE 1.13 | *Yes* | *Yes* | *No* | *Yes* | -| OpenSearch 1.0 | Yes via version setting | Yes via version setting | *No* | *Yes* | +| | Logstash OSS 7.x to 7.11.x | Logstash OSS 7.12.x\* | Logstash 7.13.x-7.16.x without OpenSearch output plugin | Logstash 7.13.x-7.16.x with OpenSearch output plugin | Logstash 8.x+ with OpenSearch output plugin +| :---| :--- | :--- | :--- | :--- | :--- | +| Elasticsearch OSS 7.x to 7.9.x | *Yes* | *Yes* | *No* | *Yes* | *Yes* | +| Elasticsearch OSS 7.10.2 | *Yes* | *Yes* | *No* | *Yes* | *Yes* | +| ODFE 1.x to 1.12 | *Yes* | *Yes* | *No* | *Yes* | *Yes* | +| ODFE 1.13 | *Yes* | *Yes* | *No* | *Yes* | *Yes* | +| OpenSearch 1.x | Yes via version setting | Yes via version setting | *No* | *Yes* | Yes, with Elastic Common Schema Setting | \* Most current compatible version with Elasticsearch OSS. @@ -77,7 +89,7 @@ Some users report compatibility issues with ingest pipelines on these versions o | Elasticsearch OSS 7.10.2 | *Yes* | *Yes* | No | | ODFE 1.x to 1.12 | *Yes* | *Yes* | No | | ODFE 1.13 | *Yes* | *Yes* | No | -| OpenSearch 1.0 | Yes via version setting | Yes via version setting | No | +| OpenSearch 1.x | Yes via version setting | Yes via version setting | No | | Logstash OSS 7.x to 7.11.x | *Yes* | *Yes* | *Yes* | | Logstash OSS 7.12.x\* | *Yes* | *Yes* | *Yes* | | Logstash 7.13.x with OpenSearch output plugin | *Yes* | *Yes* | *Yes* | diff --git a/_clients/data-prepper/data-prepper-reference.md b/_clients/data-prepper/data-prepper-reference.md new file mode 100644 index 00000000..8936e489 --- /dev/null +++ b/_clients/data-prepper/data-prepper-reference.md @@ -0,0 +1,371 @@ +--- +layout: default +title: Configuration reference +parent: Data Prepper +nav_order: 3 +--- + +# Data Prepper configuration reference + +This page lists all supported Data Prepper server, sources, buffers, processors, and sinks, along with their associated options. For example configuration files, see [Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines/). + +## Data Prepper server options + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +ssl | No | Boolean | Indicates whether TLS should be used for server APIs. Defaults to true. +keyStoreFilePath | No | String | Path to a .jks or .p12 keystore file. Required if ssl is true. +keyStorePassword | No | String | Password for keystore. Optional, defaults to empty string. +privateKeyPassword | No | String | Password for private key within keystore. Optional, defaults to empty string. +serverPort | No | Integer | Port number to use for server APIs. Defaults to 4900 +metricRegistries | No | List | Metrics registries for publishing the generated metrics. Currently supports Prometheus and CloudWatch. Defaults to Prometheus. + +## General pipeline options + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +workers | No | Integer | Essentially the number of application threads. As a starting point for your use case, try setting this value to the number of CPU cores on the machine. Default is 1. +delay | No | Integer | Amount of time in milliseconds workers wait between buffer read attempts. Default is 3,000. + + +## Sources + +Sources define where your data comes from. + + +### otel_trace_source + +Source for the OpenTelemetry Collector. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +port | No | Integer | The port OTel trace source is running on. Default is `21890`. +request_timeout | No | Integer | The request timeout in milliseconds. Default is `10_000`. +health_check_service | No | Boolean | Enables a gRPC health check service under `grpc.health.v1/Health/Check`. Default is `false`. +proto_reflection_service | No | Boolean | Enables a reflection service for Protobuf services (see [gRPC reflection](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md) and [gRPC Server Reflection Tutorial](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md) docs). Default is `false`. +unframed_requests | No | Boolean | Enable requests not framed using the gRPC wire protocol. +thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default is `200`. +max_connection_count | No | Integer | The maximum allowed number of open connections. Default is `500`. +ssl | No | Boolean | Enables connections to the OTel source port over TLS/SSL. Defaults to `true`. +sslKeyCertChainFile | Conditionally | String | File-system path or AWS S3 path to the security certificate (e.g. `"config/demo-data-prepper.crt"` or `"s3://my-secrets-bucket/demo-data-prepper.crt"`). Required if ssl is set to `true`. +sslKeyFile | Conditionally | String | File-system path or AWS S3 path to the security key (e.g. `"config/demo-data-prepper.key"` or `"s3://my-secrets-bucket/demo-data-prepper.key"`). Required if ssl is set to `true`. +useAcmCertForSSL | No | Boolean | Whether to enable TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `useAcmCertForSSL` is set to `true`. +awsRegion | Conditionally | String | Represents the AWS region to use ACM or S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are AWS S3 paths. +authentication | No | Object| An authentication configuration. By default, this runs an unauthenticated server. This uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication use or create a plugin which implements: [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java). + +### http_source + +This is a source plugin that supports HTTP protocol. Currently ONLY support Json UTF-8 codec for incoming request, e.g. `[{"key1": "value1"}, {"key2": "value2"}]`. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +port | No | Integer | The port the source is running on. Default is `2021`. Valid options are between `0` and `65535`. +request_timeout | No | Integer | The request timeout in millis. Default is `10_000`. +thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default is `200`. +max_connection_count | No | Integer | The maximum allowed number of open connections. Default is `500`. +max_pending_requests | No | Integer | The maximum number of allowed tasks in ScheduledThreadPool work queue. Default is `1024`. +authentication | No | Object | An authentication configuration. By default, this runs an unauthenticated server. This uses pluggable authentication for HTTPS. To use basic authentication define the `http_basic` plugin with a `username` and `password`. To provide customer authentication use or create a plugin which implements: [ArmeriaHttpAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/ArmeriaHttpAuthenticationProvider.java). + +### file + +Source for flat file input. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +path | Yes | String | Path to the input file (e.g. `logs/my-log.log`). +format | No | String | Format of each line in the file. Valid options are `json` or `plain`. Default is `plain`. +record_type | No | String | The record type to store. Valid options are `string` or `event`. Default is `string`. If you would like to use the file source for log analytics use cases like grok, set this option to `event`. + +### pipeline + +Source for reading from another pipeline. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +name | Yes | String | Name of the pipeline to read from. + + +### stdin + +Source for console input. Can be useful for testing. No options. + + +## Buffers + +Buffers store data as it passes through the pipeline. If you implement a custom buffer, it can be memory-based (better performance) or disk-based (larger). + + +### bounded_blocking + +The default buffer. Memory-based. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +buffer_size | No | Integer | The maximum number of records the buffer accepts. Default is 512. +batch_size | No | Integer | The maximum number of records the buffer drains after each read. Default is 8. + + +## Processors + +Processors perform some action on your data: filter, transform, enrich, etc. + +Prior to Data Prepper 1.3, Processors were named Preppers. Starting in Data Prepper 1.3, the term Prepper is deprecated in favor of Processor. Data Prepper will continue to support the term "Prepper" until 2.0, where it will be removed. +{: .note } + + +### otel_trace_raw_prepper + +Converts OpenTelemetry data to OpenSearch-compatible JSON documents. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +root_span_flush_delay | No | Integer | Represents the time interval in seconds to flush all the root spans in the processor together with their descendants. Default is 30. +trace_flush_interval | No | Integer | Represents the time interval in seconds to flush all the descendant spans without any root span. Default is 180. + + +### service_map_stateful + +Uses OpenTelemetry data to create a distributed service map for visualization in OpenSearch Dashboards. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +window_duration | No | Integer | Represents the fixed time window in seconds to evaluate service-map relationships. Default is 180. + +### peer_forwarder + +Forwards ExportTraceServiceRequests via gRPC to other Data Prepper instances. Required for operating Data Prepper in a clustered deployment. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +time_out | No | Integer | Forwarded request timeout in seconds. Defaults to 3 seconds. +span_agg_count | No | Integer | Batch size for number of spans per request. Defaults to 48. +target_port | No | Integer | The destination port to forward requests to. Defaults to `21890`. +discovery_mode | No | String | Peer discovery mode to be used. Allowable values are `static`, `dns`, and `aws_cloud_map`. Defaults to `static`. +static_endpoints | No | List | List containing string endpoints of all Data Prepper instances. +domain_name | No | String | Single domain name to query DNS against. Typically used by creating multiple DNS A Records for the same domain. +ssl | No | Boolean | Indicates whether TLS should be used. Default is true. +awsCloudMapNamespaceName | Conditionally | String | Name of your CloudMap Namespace. Required if `discovery_mode` is set to `aws_cloud_map`. +awsCloudMapServiceName | Conditionally | String | Service name within your CloudMap Namespace. Required if `discovery_mode` is set to `aws_cloud_map`. +sslKeyCertChainFile | Conditionally | String | Represents the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Required if `ssl` is set to `true`. +useAcmCertForSSL | No | Boolean | Enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +awsRegion | Conditionally | String | Represents the AWS region to use ACM, S3, or CloudMap. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are AWS S3 paths. +acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `useAcmCertForSSL` is set to `true`. + +### string_converter + +Converts string to uppercase or lowercase. Mostly useful as an example if you want to develop your own processor. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +upper_case | No | Boolean | Whether to convert to uppercase (`true`) or lowercase (`false`). + +### aggregate + +Groups events together based on the keys provided and performs a action on each group. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +identification_keys | Yes | List | A unordered list by which to group Events. Events with the same values for these keys are put into the same group. If an Event does not contain one of the `identification_keys`, then the value of that key is considered to be equal to `null`. At least one identification_key is required. (e.g. `["sourceIp", "destinationIp", "port"]`). +action | Yes | AggregateAction | The action to be performed for each group. One of the available Aggregate Actions must be provided or you can create custom aggregate actions. `remove_duplicates` and `put_all` are available actions. For more information, see [creating custom aggregate actions](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#creating-new-aggregate-actions). +group_duration | No | String | The amount of time that a group should exist before it is concluded automatically. Supports ISO_8601 notation strings ("PT20.345S", "PT15M", etc.) as well as simple notation for seconds (`"60s"`) and milliseconds (`"1500ms"`). Default value is `180s`. + +### date + +Adds a default timestamp to the event or parses timestamp fields, and converts it to ISO 8601 format which can be used as event timestamp. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +match | Conditionally | List | List of `key` and `patterns` where patterns is a list. The list of match can have exactly one `key` and `patterns`. There is no default value. This option cannot be defined at the same time as `from_time_received`. Include multiple date processors in your pipeline if both options should be used. +from_time_received | Conditionally | Boolean | A boolean that is used for adding default timestamp to event data from event metadata which is the time when source receives the event. Default value is `false`. This option cannot be defined at the same time as `match`. Include multiple date processors in your pipeline if both options should be used. +destination | No | String | Field to store the timestamp parsed by date processor. It can be used with both `match` and `from_time_received`. Default value is `@timestamp`. +source_timezone | No | String | Time zone used to parse dates. It is used in case zone or offset cannot be extracted from the value. If zone or offset are part of the value, then timezone is ignored. Find all the available timezones [the list of database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) in the "TZ database name" column. +destination_timezone | No | String | Timezone used for storing timestamp in `destination` field. The available timezone values are the same as `source_timestamp`. +locale | No | String | Locale is used for parsing dates. It's commonly used for parsing month names(`MMM`). It can have language, country and variant fields using IETF BCP 47 or String representation of [Locale](https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html) object. For example `en-US` for IETF BCP 47 and `en_US` for string representation of Locale. Full list of locale fields which includes language, country and variant can be found [the language subtag registry](https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry). Default value is `Locale.ROOT`. + +### drop_events + +Drops all the events that are passed into this processor. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +drop_when | Yes | String | Accepts a Data Prepper Expression string following the [Data Prepper Expression Syntax](https://github.com/opensearch-project/data-prepper/blob/main/docs/expression_syntax.md). Configuring `drop_events` with `drop_when: true` drops all the events received. +handle_failed_events | No | Enum | Specifies how exceptions are handled when an exception occurs while evaluating an event. Default value is `drop`, which drops the event so it doesn't get sent to OpenSearch. Available options are `drop`, `drop_silently`, `skip`, `skip_silently`. For more information, see [handle_failed_events](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/drop-events-processor#handle_failed_events). + +### grok_prepper + +Takes unstructured data and utilizes pattern matching to structure and extract important keys and make data more structured and queryable. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +match | No | Map | Specifies which keys to match specific patterns against. Default is an empty body. +keep_empty_captures | No | Boolean | Enables preserving `null` captures. Default value is `false`. +named_captures_only | No | Boolean | enables whether to keep only named captures. Default value is `true`. +break_on_match | No | Boolean | Specifies whether to match all patterns or stop once the first successful match is found. Default is `true`. +keys_to_overwrite | No | List | Specifies which existing keys are to be overwritten if there is a capture with the same key value. Default is `[]`. +pattern_definitions | No | Map | Allows for custom pattern use inline. Default value is an empty body. +patterns_directories | No | List | Specifies the path of directories that contain customer pattern files. Default value is an empty list. +pattern_files_glob | No | String | Specifies which pattern files to use from the directories specified for `pattern_directories`. Default is `*`. +target_key | No | String | Specifies a parent level key to store all captures. Default value is `null`. +timeout_millis | No | Integer | Maximum amount of time that should take place for the matching. Setting to `0` disables the timeout. Default value is `30,000`. + +### key_value + +Takes in a field and parses it into key/value pairs. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +source | No | String | The key in the event that is parsed. Default value is `message`. +destination | No | String | The key where to output the parsed source to. Doing so overwrites the value of the key if it exists. Default value is `parsed_message` +field_delimiter_regex | Conditionally | String | A regex specifying the delimiter between key/value pairs. Special regex characters such as `[` and `]` must be escaped using `\\`. This option cannot be defined at the same time as `field_split_characters`. +field_split_characters | Conditionally | String | A string of characters to split between key/value pairs. Special regex characters such as `[` and `]` must be escaped using `\\`. Default value is `&`. This option cannot be defined at the same time as `field_delimiter_regex`. +key_value_delimiter_regex| Conditionally | String | A regex specifying the delimiter between a key and a value. Special regex characters such as `[` and `]` must be escaped using `\\`. There is no default value. This option cannot be defined at the same time as `value_split_characters`. +value_split_characters | Conditionally | String | A string of characters to split between keys and values. Special regex characters such as `[` and `]` must be escaped using `\\`. Default value is `=`. This option cannot be defined at the same time as `key_value_delimiter_regex`. +non_match_value | No | String | When a key/value cannot be successfully split, the key/value is be placed in the key field and the specified value in the value field. Default value is `null`. +prefix | No | String | A prefix given to all keys. Default value is empty string. +delete_key_regex | No | String | A regex used to delete characters from the key. Special regex characters such as `[` and `]` must be escaped using `\\`. There is no default value. +delete_value_regex | No | String | A regex used to delete characters from the value. Special regex characters such as `[` and `]` must be escaped using `\\`. There is no default value. + +### add_entries + +Adds an entry to event. `add_entries` is part of [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +entries | Yes | List | List of events to be added. Valid entries are `key`, `value`, and `overwrite_if_key_exists`. +key | N/A | N/A | Key of the new event to be added. +value | N/A | N/A | Value of the new entry to be added. Valid data types are strings, booleans, numbers, null, nested objects, and arrays containing the aforementioned data types. +overwrite_if_key_exists | No | Boolean | If true, the existing value gets overwritten if the key already exists within the event. Default is `false`. + +### copy_values + +Copy values within an event. `copy_values` is part of [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +entries | Yes | List | List of entries to be copied. Valid values are `from_key`, `to_key`, and `overwrite_if_key_exists`. +from_key | N/A | N/A | The key of the entry to be copied. +to_key | N/A | N/A | The key of the new entry to be added. +overwrite_if_to_key_exists | No | Boolean | If true, the existing value gets overwritten if the key already exists within the event. Default is `false`. + + +### delete_entries + +Delete entries in an event. `delete_entries` is part of [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +with_keys | Yes | List | An array of keys of the entries to be deleted. + +### rename_keys + +Rename keys in an event. `rename_keys` is part of [mutate event](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-event-processors#mutate-event-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +entries | Yes | List | List of entries. Valid values are `from_key`, `to_key`, and `overwrite_if_key_exists`. Renaming occurs in the order defined. +from_key | N/A | N/A | The key of the entry to be renamed. +to_key | N/A | N/A | The new key of the entry. +overwrite_if_to_key_exists | No | Boolean | If true, the existing value gets overwritten if `to_key` already exists in the event. + +### substitute_string + +Matches a key's value against a regular expression and replaces all matches with a replacement string. `substitute_string` is part of [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +entries | Yes | List | List of entries. Valid values are `source`, `from`, and `to`. +source | N/A | N/A | The key to modify. +from | N/A | N/A | The Regex String to be replaced. Special regex characters such as `[` and `]` must be escaped using `\\` when using double quotes and `\ ` when using single quotes. See [Java Patterns](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Pattern.html) for more information. +to | N/A | N/A | The String to be substituted for each match of `from`. + +### split_string + +Splits a field into an array using a delimiter character. `split_string` is part of [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +entries | Yes | List | List of entries. Valid values are `source`, `delimiter`, and `delimiter_regex`. +source | N/A | N/A | The key to split. +delimiter | No | N/A | The separator character responsible for the split. Cannot be defined at the same time as `delimiter_regex`. At least `delimiter` or `delimiter_regex` must be defined. +delimiter_regex | No | N/A | The regex string responsible for the split. Cannot be defined at the same time as `delimiter`. At least `delimiter` or `delimiter_regex` must be defined. + +### uppercase_string + +Converts a string to its uppercase counterpart. `uppercase_string` is part of [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +with_keys | Yes | List | A list of keys to convert to uppercase. + +### lowercase_string + +Converts a string to its lowercase counterpart. `lowercase_string` is part of [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +with_keys | Yes | List | A list of keys to convert to lowercase. + +### trim_string + +Strips whitespace from the beginning and end of a key. `trim_string` is part of [mutate string](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/mutate-string-processors#mutate-string-processors) processors. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +with_keys | Yes | List | A list of keys to trim the whitespace from. + +## Sinks + +Sinks define where Data Prepper writes your data to. + + +### opensearch + +Sink for an OpenSearch cluster. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +hosts | Yes | List | List of OpenSearch hosts to write to (e.g. `["https://localhost:9200", "https://remote-cluster:9200"]`). +cert | No | String | Path to the security certificate (e.g. `"config/root-ca.pem"`) if the cluster uses the OpenSearch security plugin. +username | No | String | Username for HTTP basic authentication. +password | No | String | Password for HTTP basic authentication. +aws_sigv4 | No | Boolean | default false. Whether to use IAM signing to connect to an Amazon OpenSearch Service domain. For your access key, secret key, and optional session token, Data Prepper uses the default credential chain (environment variables, Java system properties, `~/.aws/credential`, etc.). +aws_region | No | String | AWS region (e.g. `"us-east-1"`) for the domain if you are connecting to Amazon OpenSearch Service. +aws_sts_role_arn | No | String | IAM role which the sink plugin assumes to sign request to Amazon OpenSearch Service. If not provided the plugin uses the default credentials. +socket_timeout | No | Integer | the timeout in milliseconds for waiting for data (or, put differently, a maximum period inactivity between two consecutive data packets). A timeout value of zero is interpreted as an infinite timeout. If this timeout value is either negative or not set, the underlying Apache HttpClient would rely on operating system settings for managing socket timeouts. +connect_timeout | No | Integer | The timeout in milliseconds used when requesting a connection from the connection manager. A timeout value of zero is interpreted as an infinite timeout. If this timeout value is either negative or not set, the underlying Apache HttpClient would rely on operating system settings for managing connection timeouts. +insecure | No | Boolean | Whether to verify SSL certificates. If set to true, CA certificate verification is disabled and insecure HTTP requests are sent instead. Default is `false`. +proxy | No | String | The address of a [forward HTTP proxy server](https://en.wikipedia.org/wiki/Proxy_server). The format is "<host name or IP>:<port>". Examples: "example.com:8100", "http://example.com:8100", "112.112.112.112:8100". Port number cannot be omitted. +trace_analytics_raw | No | Boolean | Deprecated in favor of `index_type`. Whether to export as trace data to the `otel-v1-apm-span-*` index pattern (alias `otel-v1-apm-span`) for use with the Trace Analytics OpenSearch Dashboards plugin. Default is `false`. +trace_analytics_service_map | No | Boolean | Deprecated in favor of `index_type`. Whether to export as trace data to the `otel-v1-apm-service-map` index for use with the service map component of the Trace Analytics OpenSearch Dashboards plugin. Default is `false`. +index | No | String | Name of the index to export to. Only required if you don't use the `trace-analytics-raw` or `trace-analytics-service-map` presets. In other words, this parameter is applicable and required only if index_type is explicitly `custom` or defaults to `custom`. +index_type | No | String | This index type instructs the Sink plugin what type of data it is handling. Valid values: `custom`, `trace-analytics-raw`, `trace-analytics-service-map`. Default is `custom`. +template_file | No | String | Path to a JSON [index template]({{site.url}}{{site.baseurl}}/opensearch/index-templates/) file (e.g. `/your/local/template-file.json` if you do not use the `trace_analytics_raw` or `trace_analytics_service_map`.) See [otel-v1-apm-span-index-template.json](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-span-index-template.json) for an example. +document_id_field | No | String | The field from the source data to use for the OpenSearch document ID (e.g. `"my-field"`) if you don't use the `trace_analytics_raw` or `trace_analytics_service_map` presets. +dlq_file | No | String | The path to your preferred dead letter queue file (e.g. `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the OpenSearch cluster. +bulk_size | No | Integer (long) | The maximum size (in MiB) of bulk requests to the OpenSearch cluster. Values below 0 indicate an unlimited size. If a single document exceeds the maximum bulk request size, Data Prepper sends it individually. Default is 5. +ism_policy_file | No | String | The absolute file path for an ISM (Index State Management) policy JSON file. This policy file is effective only when there is no built-in policy file for the index type. For example, `custom` index type is currently the only one without a built-in policy file, thus it would use the policy file here if it's provided through this parameter. For more information, see [ISM policies]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/). +number_of_shards | No | Integer | The number of primary shards that an index should have on the destination OpenSearch server. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [create index]({{site.url}}{{site.baseurl}}/opensearch/rest-api/index-apis/create-index/). +number_of_replicas | No | Integer | The number of replica shards each primary shard should have on the destination OpenSearch server. For example, if you have 4 primary shards and set number_of_replicas to 3, the index has 12 replica shards. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [create index]({{site.url}}{{site.baseurl}}/opensearch/rest-api/index-apis/create-index/). + +### file + +Sink for flat file output. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +path | Yes | String | Path for the output file (e.g. `logs/my-transformed-log.log`). + + +### pipeline + +Sink for writing to another pipeline. + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +name | Yes | String | Name of the pipeline to write to. + + +### stdout + +Sink for console output. Can be useful for testing. No options. diff --git a/_clients/data-prepper/get-started.md b/_clients/data-prepper/get-started.md new file mode 100644 index 00000000..11ef4ea9 --- /dev/null +++ b/_clients/data-prepper/get-started.md @@ -0,0 +1,63 @@ +--- +layout: default +title: Get Started +parent: Data Prepper +nav_order: 1 +--- + +# Get started with Data Prepper + +Data Prepper is an independent component, not an OpenSearch plugin, that converts data for use with OpenSearch. It's not bundled with the all-in-one OpenSearch installation packages. + +## 1. Install Data Prepper + +To use the Docker image, pull it like any other image: + +```bash +docker pull opensearchproject/data-prepper:latest +``` + +## 2. Define a pipeline + +Create a Data Prepper pipeline file, `pipelines.yaml`, with the following configuration: + +```yml +simple-sample-pipeline: + workers: 2 + delay: "5000" + source: + random: + sink: + - stdout: +``` + +## 3. Start Data Prepper + +Run the following command with your pipeline configuration YAML. + +```bash +docker run --name data-prepper \ + -v /full/path/to/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml \ + opensearchproject/opensearch-data-prepper:latest +``` + +This sample pipeline configuration above demonstrates a simple pipeline with a source (`random`) sending data to a sink (`stdout`). For more examples and details on more advanced pipeline configurations, see [Pipelines]({{site.url}}{{site.baseurl}}/clients/data-prepper/pipelines). + +After starting Data Prepper, you should see log output and some UUIDs after a few seconds: + +```yml +2021-09-30T20:19:44,147 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900 +2021-09-30T20:19:44,681 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:45,183 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:45,687 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:46,191 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:46,694 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:47,200 [random-source-pool-0] INFO com.amazon.dataprepper.plugins.source.RandomStringSource - Writing to buffer +2021-09-30T20:19:49,181 [simple-test-pipeline-processor-worker-1-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - simple-test-pipeline Worker: Processing 6 records from buffer +07dc0d37-da2c-447e-a8df-64792095fb72 +5ac9b10a-1d21-4306-851a-6fb12f797010 +99040c79-e97b-4f1d-a70b-409286f2a671 +5319a842-c028-4c17-a613-3ef101bd2bdd +e51e700e-5cab-4f6d-879a-1c3235a77d18 +b4ed2d7e-cf9c-4e9d-967c-b18e8af35c90 +``` diff --git a/_clients/data-prepper/index.md b/_clients/data-prepper/index.md new file mode 100644 index 00000000..7fb833f4 --- /dev/null +++ b/_clients/data-prepper/index.md @@ -0,0 +1,15 @@ +--- +layout: default +title: Data Prepper +nav_order: 120 +has_children: true +has_toc: false +--- + +# Data Prepper + +Data Prepper is a server side data collector capable of filtering, enriching, transforming, normalizing and aggregating data for downstream analytics and visualization. + +Data Prepper lets users build custom pipelines to improve the operational view of applications. Two common uses for Data Prepper are trace and log analytics. [Trace analytics]({{site.url}}{{site.baseurl}}/observability-plugin/trace/index/) can help you visualize the flow of events and identify performance problems, and [log analytics]({{site.url}}{{site.baseurl}}/observability-plugin/log-analytics/) can improve searching, analyzing and provide insights into your application. + +To get started building your own custom pipelines with Data Prepper, see the [Get Started]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/) guide. diff --git a/_clients/data-prepper/pipelines.md b/_clients/data-prepper/pipelines.md new file mode 100644 index 00000000..b664d98a --- /dev/null +++ b/_clients/data-prepper/pipelines.md @@ -0,0 +1,153 @@ +--- +layout: default +title: Pipelines +parent: Data Prepper +nav_order: 2 +--- + +# Pipelines + +![Data Prepper Pipeline]({{site.url}}{{site.baseurl}}/images/data-prepper-pipeline.png) + +To use Data Prepper, you define pipelines in a configuration YAML file. Each pipeline is a combination of a source, a buffer, zero or more preppers, and one or more sinks. For example: + +```yml +simple-sample-pipeline: + workers: 2 # the number of workers + delay: 5000 # in milliseconds, how long workers wait between read attempts + source: + random: + buffer: + bounded_blocking: + buffer_size: 1024 # max number of records the buffer accepts + batch_size: 256 # max number of records the buffer drains after each read + processor: + - string_converter: + upper_case: true + sink: + - stdout: +``` + +- Sources define where your data comes from. In this case, the source is a random UUID generator (`random`). + +- Buffers store data as it passes through the pipeline. + + By default, Data Prepper uses its one and only buffer, the `bounded_blocking` buffer, so you can omit this section unless you developed a custom buffer or need to tune the buffer settings. + +- Preppers perform some action on your data: filter, transform, enrich, etc. + + You can have multiple preppers, which run sequentially from top to bottom, not in parallel. The `string_converter` prepper transform the strings by making them uppercase. + +- Sinks define where your data goes. In this case, the sink is stdout. + +## Examples + +This section provides some pipeline examples that you can use to start creating your own pipelines. For more information, see [Data Prepper configuration reference]({{site.url}}{{site.baseurl}}/clients/data-prepper/data-prepper-reference/) guide. + +The Data Prepper repository has several [sample applications](https://github.com/opensearch-project/data-prepper/tree/main/examples) to help you get started. + +### Log ingestion pipeline + +The following example demonstrates how to use HTTP source and Grok prepper plugins to process unstructured log data. + +```yml +log-pipeline: + source: + http: + ssl: false + processor: + - grok: + match: + log: [ "%{COMMONAPACHELOG}" ] + sink: + - opensearch: + hosts: [ "https://opensearch:9200" ] + insecure: true + username: admin + password: admin + index: apache_logs +``` + +This example uses weak security. We strongly recommend securing all plugins which open external ports in production environments. +{: .note} + +### Trace Analytics pipeline + +The following example demonstrates how to build a pipeline that supports the [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). This pipeline takes data from the OpenTelemetry Collector and uses two other pipelines as sinks. These two separate pipelines index trace and the service map documents for the dashboard plugin. + +```yml +entry-pipeline: + delay: "100" + source: + otel_trace_source: + ssl: false + sink: + - pipeline: + name: "raw-pipeline" + - pipeline: + name: "service-map-pipeline" +raw-pipeline: + source: + pipeline: + name: "entry-pipeline" + prepper: + - otel_trace_raw_prepper: + sink: + - opensearch: + hosts: ["https://localhost:9200"] + insecure: true + username: admin + password: admin + trace_analytics_raw: true +service-map-pipeline: + delay: "100" + source: + pipeline: + name: "entry-pipeline" + prepper: + - service_map_stateful: + sink: + - opensearch: + hosts: ["https://localhost:9200"] + insecure: true + username: admin + password: admin + trace_analytics_service_map: true +``` + +## Migrating from Logstash + +Data Prepper supports Logstash configuration files for a limited set of plugins. Simply use the logstash config to run Data Prepper. + +```bash +docker run --name data-prepper \ + -v /full/path/to/logstash.conf:/usr/share/data-prepper/pipelines.conf \ + opensearchproject/opensearch-data-prepper:latest +``` + +This feature is limited by feature parity of Data Prepper. As of Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported: + +- HTTP Input plugin +- Grok Filter plugin +- Elasticsearch Output plugin +- Amazon Elasticsearch Output plugin + +## Configure the Data Prepper server + +Data Prepper itself provides administrative HTTP endpoints such as `/list` to list pipelines and `/metrics/prometheus` to provide Prometheus-compatible metrics data. The port that has these endpoints has a TLS configuration and is specified by a separate YAML file. By default, these endpoints are secured by Data Prepper docker images. We strongly recommend providing your own configuration file for securing production environments. Here is an example `data-prepper-config.yaml`: + +```yml +ssl: true +keyStoreFilePath: "/usr/share/data-prepper/keystore.jks" +keyStorePassword: "password" +privateKeyPassword: "other_password" +serverPort: 1234 +``` + +To configure the Data Prepper server, run Data Prepper with the additional yaml file. + +```bash +docker run --name data-prepper -v /full/path/to/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml \ + /full/path/to/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml \ + opensearchproject/opensearch-data-prepper:latest +```` diff --git a/_clients/go.md b/_clients/go.md index 75ee300e..f1914134 100644 --- a/_clients/go.md +++ b/_clients/go.md @@ -68,8 +68,8 @@ func main() { }`) // Create an index with non-default settings. - res := opensearchapi.CreateRequest{ - Index: IndexName, + res := opensearchapi.IndicesCreateRequest{ + Index: IndexName, Body: mapping, } fmt.Println("creating index", res) diff --git a/_clients/index.md b/_clients/index.md index 2f3513dd..89792d8e 100644 --- a/_clients/index.md +++ b/_clients/index.md @@ -19,18 +19,24 @@ For example, a 1.0.0 client works with an OpenSearch 1.1.0 cluster, but might no * [OpenSearch Python client]({{site.url}}{{site.baseurl}}/clients/python/) * [OpenSearch JavaScript (Node.js) client]({{site.url}}{{site.baseurl}}/clients/javascript/) * [OpenSearch Go client]({{site.url}}{{site.baseurl}}/clients/go/) +* [OpenSearch PHP client]({{site.url}}{{site.baseurl}}/clients/php/) ## Legacy clients Most clients that work with Elasticsearch OSS 7.10.2 *should* work with OpenSearch, but the latest versions of those clients might include license or version checks that artificially break compatibility. This page includes recommendations around which versions of those clients to use for best compatibility with OpenSearch. +{: .note} +OpenSearch 2.0.0 no longer supports compatibility with legacy clients. Due to breaking changes with REST APIs, some features are not supported when using OpenSearch 1.x clients to connect to OpenSearch 2.0. + + Client | Recommended version :--- | :--- [Java low-level REST client](https://search.maven.org/artifact/org.elasticsearch.client/elasticsearch-rest-client/7.13.4/jar) | 7.13.4 [Java high-level REST client](https://search.maven.org/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client/7.13.4/jar) | 7.13.4 [Python Elasticsearch client](https://pypi.org/project/elasticsearch/7.13.4/) | 7.13.4 [Elasticsearch Node.js client](https://www.npmjs.com/package/@elastic/elasticsearch/v/7.13.0) | 7.13.0 +[Ruby Elasticsearch client](https://rubygems.org/gems/elasticsearch/versions/7.13.0) | 7.13.0 If you test a legacy client and verify that it works, please [submit a PR](https://github.com/opensearch-project/documentation-website/pulls) and add it to this table. diff --git a/_clients/java-rest-high-level.md b/_clients/java-rest-high-level.md index d441a36d..35778af7 100644 --- a/_clients/java-rest-high-level.md +++ b/_clients/java-rest-high-level.md @@ -6,9 +6,6 @@ nav_order: 60 # Java high-level REST client -Although the OpenSearch Java high-level REST client is still usable, we recommend that you use the [OpenSearch Java client]({{site.url}}{{site.baseurl}}/clients/java/), which replaces the existing Java high-level REST client. -{: .note} - The OpenSearch Java high-level REST client lets you interact with your OpenSearch clusters and indices through Java methods and data structures rather than HTTP methods and JSON. ## Setup @@ -25,8 +22,24 @@ To start using the OpenSearch Java high-level REST client, ensure that you have You can now start your OpenSearch cluster. The OpenSearch 1.x high-level REST client works with the 1.x versions of OpenSearch. +## Security + +Before using the REST client in your Java application, you must configure the application's truststore to connect to the security plugin. If you are using self-signed certificates or demo configurations, you can use the following command to create a custom truststore and add in root authority certificates. + +If you're using certificates from a trusted Certificate Authority (CA), you don't need to configure the truststore. + +```bash +keytool -import -alias -keystore +``` + +You can now point your Java client to the truststore and set basic authentication credentials that can access a secure cluster (refer to the sample code below on how to do so). + +If you run into issues when configuring security, see [common issues]({{site.url}}{{site.baseurl}}/troubleshoot/index) and [troubleshoot TLS]({{site.url}}{{site.baseurl}}/troubleshoot/tls). + ## Sample code +This code example uses basic credentials that come with the default OpenSearch configuration. If you’re using the OpenSearch Java high-level REST client with your own OpenSearch cluster, be sure to change the code to use your own credentials. + ```java import org.apache.http.HttpHost; import org.apache.http.auth.AuthScope; @@ -62,7 +75,7 @@ public class RESTClientSample { System.setProperty("javax.net.ssl.trustStorePassword", "password-to-keystore"); //Establish credentials to use basic authentication. - //Only for demo purposes. Do not specify your credentials in code. + //Only for demo purposes. Don't specify your credentials in code. final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, diff --git a/_clients/java.md b/_clients/java.md index c51a664f..067af720 100644 --- a/_clients/java.md +++ b/_clients/java.md @@ -6,15 +6,23 @@ nav_order: 65 # Java client +The OpenSearch Java client is currently in its beta phase, so we recommend that you use the [OpenSearch Java high-level REST client]({{site.url}}{{site.baseurl}}/clients/java-rest-high-level). +{: .note} + The OpenSearch Java client allows you to interact with your OpenSearch clusters through Java methods and data structures rather than HTTP methods and raw JSON. For example, you can submit requests to your cluster using objects to create indices, add data to documents, or complete some other operation using the client's built-in methods. -## Setup +## Install the client -To start using the OpenSearch Java client, ensure that you have the following dependency in your project's `pom.xml` file: +To start using the OpenSearch Java client, ensure that you have the following dependencies in your project's `pom.xml` file: ``` + + org.opensearch.client + opensearch-rest-client + {{site.opensearch_version}} + org.opensearch.client opensearch-java @@ -33,9 +41,21 @@ dependencies { You can now start your OpenSearch cluster. -The following example uses credentials that come with the default OpenSearch configuration. If you're using the OpenSearch Java client with your own OpenSearch cluster, be sure to change the code to use your own credentials. +## Security -## Sample code +Before using the REST client in your Java application, you must configure the application's truststore to connect to the security plugin. If you are using self-signed certificates or demo configurations, you can use the following command to create a custom truststore and add in root authority certificates. + +If you're using certificates from a trusted Certificate Authority (CA), you don't need to configure the truststore. + +```bash +keytool -import -alias -keystore +``` + +You can now point your Java client to the truststore and set basic authentication credentials that can access a secure cluster (refer to the sample code below on how to do so). + +If you run into issues when configuring security, see [common issues]({{site.url}}{{site.baseurl}}/troubleshoot/index) and [troubleshoot TLS]({{site.url}}{{site.baseurl}}/troubleshoot/tls). + +## Sample data This section uses a class called `IndexData`, which is a simple Java class that stores basic data and methods. For your own OpenSearch cluster, you might find that you need a more robust class to store your data. @@ -74,7 +94,11 @@ static class IndexData { } ``` -### OpenSearch client example +## Initialize the client with SSL and TLS enabled + +This code example uses basic credentials that come with the default OpenSearch configuration. If you’re using the Java client with your own OpenSearch cluster, be sure to change the code to use your own credentials. + +The following sample code initializes a client with SSL and TLS enabled: ```java import org.apache.http.HttpHost; @@ -85,20 +109,134 @@ import org.apache.http.impl.client.BasicCredentialsProvider; import org.apache.http.impl.nio.client.HttpAsyncClientBuilder; import org.opensearch.client.RestClient; import org.opensearch.client.RestClientBuilder; -import org.opensearch.clients.base.RestClientTransport; -import org.opensearch.clients.base.Transport; -import org.opensearch.clients.json.jackson.JacksonJsonpMapper; -import org.opensearch.clients.opensearch.OpenSearchClient; -import org.opensearch.clients.opensearch._global.IndexRequest; -import org.opensearch.clients.opensearch._global.IndexResponse; -import org.opensearch.clients.opensearch._global.SearchResponse; -import org.opensearch.clients.opensearch.indices.*; -import org.opensearch.clients.opensearch.indices.put_settings.IndexSettingsBody; +import org.opensearch.client.base.RestClientTransport; +import org.opensearch.client.base.Transport; +import org.opensearch.client.json.jackson.JacksonJsonpMapper; +import org.opensearch.client.opensearch.OpenSearchClient; +import org.opensearch.client.opensearch._global.IndexRequest; +import org.opensearch.client.opensearch._global.IndexResponse; +import org.opensearch.client.opensearch._global.SearchResponse; +import org.opensearch.client.opensearch.indices.*; +import org.opensearch.client.opensearch.indices.put_settings.IndexSettingsBody; import java.io.IOException; public class OpenSearchClientExample { public static void main(String[] args) { + RestClient restClient = null; + try{ + System.setProperty("javax.net.ssl.trustStore", "/full/path/to/keystore"); + System.setProperty("javax.net.ssl.trustStorePassword", "password-to-keystore"); + + //Only for demo purposes. Don't specify your credentials in code. + final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); + credentialsProvider.setCredentials(AuthScope.ANY, + new UsernamePasswordCredentials("admin", "admin")); + + //Initialize the client with SSL and TLS enabled + restClient = RestClient.builder(new HttpHost("localhost", 9200, "https")). + setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() { + @Override + public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) { + return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider); + } + }).build(); + Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper()); + OpenSearchClient client = new OpenSearchClient(transport); + } + } +} +``` + +## OpenSearch client examples + +This section has sample code that shows you how to create an index with non-default settings, add a document to the index, search for the document, delete the document, and finally delete the index. + +### Create an index with non-default settings + +```java +String index = "sample-index"; +CreateRequest createIndexRequest = new CreateRequest.Builder().index(index).build(); +client.indices().create(createIndexRequest); + +IndexSettings indexSettings = new IndexSettings.Builder().autoExpandReplicas("0-all").build(); +IndexSettingsBody settingsBody = new IndexSettingsBody.Builder().settings(indexSettings).build(); +PutSettingsRequest putSettingsRequest = new PutSettingsRequest.Builder().index(index).value(settingsBody).build(); +client.indices().putSettings(putSettingsRequest); +``` + +### Index data + +```java +IndexData indexData = new IndexData("first_name", "Bruce"); +IndexRequest indexRequest = new IndexRequest.Builder().index(index).id("1").value(indexData).build(); +client.index(indexRequest); +``` + +### Search for the document + +```java +SearchResponse searchResponse = client.search(s -> s.index(index), IndexData.class); +for (int i = 0; i< searchResponse.hits().hits().size(); i++) { + System.out.println(searchResponse.hits().hits().get(i).source()); +} +``` + +### Delete the document + +The following sample code deletes a document whose ID is 1. + +```java +client.delete(b -> b.index(index).id("1")); +``` + +### Delete the index + +```java +DeleteRequest deleteRequest = new DeleteRequest.Builder().index(index).build(); +DeleteResponse deleteResponse = client.indices().delete(deleteRequest); + +} catch (IOException e){ + System.out.println(e.toString()); +} finally { + try { + if (restClient != null) { + restClient.close(); + } + } catch (IOException e) { + System.out.println(e.toString()); + } + } + } +} +``` + +## Complete code sample + +```java +import org.apache.http.HttpHost; +import org.apache.http.auth.AuthScope; +import org.apache.http.auth.UsernamePasswordCredentials; +import org.apache.http.client.CredentialsProvider; +import org.apache.http.impl.client.BasicCredentialsProvider; +import org.apache.http.impl.nio.client.HttpAsyncClientBuilder; +import org.opensearch.client.RestClient; +import org.opensearch.client.RestClientBuilder; +import org.opensearch.client.base.RestClientTransport; +import org.opensearch.client.base.Transport; +import org.opensearch.client.json.jackson.JacksonJsonpMapper; +import org.opensearch.client.opensearch.OpenSearchClient; +import org.opensearch.client.opensearch._global.IndexRequest; +import org.opensearch.client.opensearch._global.IndexResponse; +import org.opensearch.client.opensearch._global.SearchResponse; +import org.opensearch.client.opensearch.indices.*; +import org.opensearch.client.opensearch.indices.put_settings.IndexSettingsBody; + +import java.io.IOException; + +public class OpenSearchClientExample { + public static void main(String[] args) { + RestClient restClient = null; try{ System.setProperty("javax.net.ssl.trustStore", "/full/path/to/keystore"); System.setProperty("javax.net.ssl.trustStorePassword", "password-to-keystore"); @@ -109,7 +247,7 @@ public class OpenSearchClientExample { new UsernamePasswordCredentials("admin", "admin")); //Initialize the client with SSL and TLS enabled - RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200, "https")). + restClient = RestClient.builder(new HttpHost("localhost", 9200, "https")). setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() { @Override public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) { @@ -148,13 +286,12 @@ public class OpenSearchClientExample { DeleteRequest deleteRequest = new DeleteRequest.Builder().index(index).build(); DeleteResponse deleteResponse = client.indices().delete(deleteRequest); - restClient.close(); } catch (IOException e){ System.out.println(e.toString()); } finally { try { - if (client != null) { - client.close(); + if (restClient != null) { + restClient.close(); } } catch (IOException e) { System.out.println(e.toString()); diff --git a/_clients/javascript.md b/_clients/javascript.md index c670e4b8..45923e05 100644 --- a/_clients/javascript.md +++ b/_clients/javascript.md @@ -1,7 +1,7 @@ --- layout: default title: JavaScript client -nav_order: 90 +nav_order: 100 --- # JavaScript client diff --git a/_clients/logstash/advanced-config.md b/_clients/logstash/advanced-config.md index 3eb7ef18..9410e9ad 100644 --- a/_clients/logstash/advanced-config.md +++ b/_clients/logstash/advanced-config.md @@ -212,7 +212,7 @@ useragent { } ``` -Start Logstah and send an HTTP request. +Start Logstash and send an HTTP request. You can see a field named `ua` with a number of keys including the browser name and version, the OS, and the device. diff --git a/_clients/logstash/index.md b/_clients/logstash/index.md index d8f3ec2d..2947ff73 100644 --- a/_clients/logstash/index.md +++ b/_clients/logstash/index.md @@ -67,10 +67,10 @@ If you're migrating from an existing Logstash installation, you can install the 2. Navigate to the downloaded folder in the terminal and extract the files: ```bash - tar -zxvf logstash-oss-with-opensearch-output-plugin-7.13.2-linux-x64.tar.gz + tar -zxvf logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz ``` -3. Navigate to the `logstash-7.13.2` directory. +3. Navigate to the `logstash-7.16.2` directory. - You can add your pipeline configurations to the `config` directory. Logstash saves any data from the plugins in the `data` directory. The `bin` directory contains the binaries for starting Logstash and managing plugins. ### Docker @@ -78,7 +78,7 @@ If you're migrating from an existing Logstash installation, you can install the 1. Pull the Logstash oss package with the OpenSearch output plugin image: ``` - docker pull opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2 + docker pull opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 ``` 1. Create a Docker network: @@ -90,13 +90,13 @@ If you're migrating from an existing Logstash installation, you can install the 1. Start OpenSearch with this network: ``` - docker run -p 9200:9200 -p 9600:9600 --name opensearch --net test -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0 + docker run -p 9200:9200 -p 9600:9600 --name opensearch --net test -e "discovery.type=single-node" opensearchproject/opensearch:1.2.0 ``` 1. Start Logstash: ``` - docker run -it --rm --name logstash --net test opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2 -e 'input { stdin { } } output { + docker run -it --rm --name logstash --net test opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 -e 'input { stdin { } } output { opensearch { hosts => ["https://opensearch:9200"] index => "opensearch-logstash-docker-%{+YYYY.MM.dd}" diff --git a/_clients/logstash/read-from-opensearch.md b/_clients/logstash/read-from-opensearch.md new file mode 100644 index 00000000..6582048c --- /dev/null +++ b/_clients/logstash/read-from-opensearch.md @@ -0,0 +1,51 @@ +--- +layout: default +title: Read from OpenSearch +parent: Logstash +nav_order: 220 +--- + +# Read from OpenSearch + +As we ship Logstash events to an OpenSearch cluster using the [OpenSearch output plugin](https://github.com/opensearch-project/logstash-output-opensearch), we can also perform read operations on an OpenSearch cluster and load data into Logstash using the [OpenSearch input plugin](https://github.com/opensearch-project/logstash-input-opensearch). + +The OpenSearch input plugin reads the search query results performed on an OpenSearch cluster and loads them into Logstash. This lets you replay test logs, reindex, and perform other operations based on the loaded data. You can schedule ingestions to run periodically by using +[cron expressions](https://opensearch.org/docs/latest/monitoring-plugins/alerting/cron/), or manually load data into Logstash by running the query once. + + + +## OpenSearch input plugin + +To run the OpenSearch input plugin, add the configuration to the `pipeline.conf` file within your Logstash's `config` folder. The example below runs the `match_all` query filter and loads in data once. + +```yml +input { + opensearch { + hosts => "https://hostname:port" + user => "admin" + password => "admin" + index => "logstash-logs-%{+YYYY.MM.dd}" + query => "{ "query": { "match_all": {}} }" + } +} + +filter { +} + +output { +} +``` + +To ingest data according to a schedule, use a cron expression that specifies the schedule you want. For example, to load in data every minute, add `schedule => "* * * * *"` to the input section of your `pipeline.conf` file. + +Like the output plugin, after adding your configuration to the `pipeline.conf` file, start Logstash by providing the path to this file: + + ```bash + $ bin/logstash -f config/pipeline.conf --config.reload.automatic + ``` + +`config/pipeline.conf` is a relative path to the `pipeline.conf` file. You can use an absolute path as well. + +Adding `stdout{}` to the `output{}` section of your `pipeline.conf` file prints the query results to the console. + +To reindex the data into an OpenSearch domain, add the destination domain configuration in the `output{}` section like shown [here](https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin). diff --git a/_clients/logstash/ship-to-opensearch.md b/_clients/logstash/ship-to-opensearch.md index e9ef7260..528c0e6e 100644 --- a/_clients/logstash/ship-to-opensearch.md +++ b/_clients/logstash/ship-to-opensearch.md @@ -75,3 +75,96 @@ output { health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size green | open | logstash-logs-2021.07.01 | iuh648LYSnmQrkGf70pplA | 1 | 1 | 1 | 0 | 10.3kb | 5.1kb ``` + +## Adding different Authentication mechanisms in the Output plugin + +## auth_type to support different authentication mechanisms + +In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type + +Example Configuration for basic authentication: + +```yml +output { + opensearch { + hosts => ["https://hostname:port"] + auth_type => { + type => 'basic' + user => 'admin' + password => 'admin' + } + index => "logstash-logs-%{+YYYY.MM.dd}" + } +} +``` +### Parameters inside auth_type + +- type (string) - We should specify the type of authentication +- We should add credentials required for that authentication like 'user' and 'password' for 'basic' authentication +- We should also add other parameters required for that authentication mechanism like we added 'region' for 'aws_iam' authentication + +## Configuration for AWS IAM Authentication + +To run the Logstash Output Opensearch plugin using aws_iam authentication, simply add a configuration following the below documentation. + +Example Configuration: + +```yml +output { + opensearch { + hosts => ["https://hostname:port"] + auth_type => { + type => 'aws_iam' + aws_access_key_id => 'ACCESS_KEY' + aws_secret_access_key => 'SECRET_KEY' + region => 'us-west-2' + } + index => "logstash-logs-%{+YYYY.MM.dd}" + } +} +``` + +### Required Parameters + +- hosts (array of string) - AmazonOpensearchService domain endpoint : port number +- auth_type (Json object) - Which holds other parameters required for authentication + - type (string) - "aws_iam" + - aws_access_key_id (string) - AWS access key + - aws_secret_access_key (string) - AWS secret access key + - region (string, :default => "us-east-1") - region in which the domain is located + - if we want to pass other optional parameters like profile, session_token,etc. They needs to be added in auth_type +- port (string) - AmazonOpensearchService listens on port 443 for HTTPS +- protocol (string) - The protocol used to connect to AmazonOpensearchService is 'https' + +### Optional Parameters +- The credential resolution logic can be described as follows: + - User passed aws_access_key_id and aws_secret_access_key in configuration + - Environment variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK) + - Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI + - Instance profile credentials delivered through the Amazon EC2 metadata service +- template (path) - You can set the path to your own template here, if you so desire. If not set, the included template will be used. +- template_name (string, default => "logstash") - defines how the template is named inside Opensearch + +## Data streams + +The OpenSearch output plugin can store both time series datasets (such as logs, events, and metrics) and non-time series data in OpenSearch. +The data stream is recommended to index time series datasets (such as logs, metrics, and events) into OpenSearch. + +To know more about data streams, refer to this [documentation](https://opensearch.org/docs/latest/opensearch/data-streams/). + +We can ingest data into a data stream through logstash. We need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below: + +```yml +output { + opensearch { + hosts => ["https://hostname:port"] + auth_type => { + type => 'basic' + user => 'admin' + password => 'admin' + } + index => "my-data-stream" + action => "create" + } +} +``` diff --git a/_clients/php.md b/_clients/php.md new file mode 100644 index 00000000..156180cf --- /dev/null +++ b/_clients/php.md @@ -0,0 +1,100 @@ +--- +layout: default +title: PHP client +nav_order: 90 +--- + +# PHP client + +The OpenSearch PHP client provides a safer and easier way to interact with your OpenSearch cluster. Rather than using OpenSearch from the browser and potentially exposing your data to the public, you can build an OpenSearch client that takes care of sending requests to your cluster. + +The client contains a library of APIs that let you perform different operations on your cluster and return a standard response body. The example here demonstrates some basic operations like creating an index, adding documents, and searching your data. + +## Setup + +To add the client to your project, install it using [composer](https://getcomposer.org/): + +```bash +composer require opensearch-project/opensearch-php +``` + +To install a specific major version of the client, run the following command: + +```bash +composer require opensearch-project/opensearch-php: +``` + +Then require the autload file from composer in your code: + +```php +require __DIR__ . '/vendor/autoload.php'; +``` + +## Sample code + +```php +setHosts(['https://localhost:9200']) + ->setBasicAuthentication('admin', 'admin') // For testing only. Don't store credentials in code. + ->setSSLVerification(false) // For testing only. Use certificate for validation + ->build(); + +$indexName = 'test-index-name'; + +// Print OpenSearch version information on console. +var_dump($client->info()); + +// Create an index with non-default settings. +$client->indices()->create([ + 'index' => $indexName, + 'body' => [ + 'settings' => [ + 'index' => [ + 'number_of_shards' => 4 + ] + ] + ] +]); + +$client->create([ + 'index' => $indexName, + 'id' => 1, + 'body' => [ + 'title' => 'Moneyball', + 'director' => 'Bennett Miller', + 'year' => 2011 + ] +]); + +// Search for it +var_dump( + $client->search([ + 'index' => $indexName, + 'body' => [ + 'size' => 5, + 'query' => [ + 'multi_match' => [ + 'query' => 'miller', + 'fields' => ['title^2', 'director'] + ] + ] + ] + ]) +); + +// Delete a single document +$client->delete([ + 'index' => $indexName, + 'id' => 1, +]); + + +// Delete index +$client->indices()->delete([ + 'index' => $indexName +]); +``` diff --git a/_config.yml b/_config.yml index 4f161714..265e85f6 100644 --- a/_config.yml +++ b/_config.yml @@ -5,9 +5,10 @@ baseurl: "/docs/latest" # the subpath of your site, e.g. /blog url: "https://opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com permalink: /:path/ -opensearch_version: 1.1.0 -opensearch_major_minor_version: 1.1 -lucene_version: 8_9_0 +opensearch_version: 2.0.0-rc1 +opensearch_dashboards_version: 2.0.0-rc1 +opensearch_major_minor_version: 2.0-rc1 +lucene_version: 9_1_0 # Build settings markdown: kramdown @@ -48,6 +49,12 @@ collections: replication-plugin: permalink: /:collection/:path/ output: true + observability-plugin: + permalink: /:collection/:path/ + output: true + ml-commons-plugin: + permalink: /:collection/:path/ + output: true monitoring-plugins: permalink: /:collection/:path/ output: true @@ -90,6 +97,12 @@ just_the_docs: replication-plugin: name: Replication plugin nav_fold: true + observability-plugin: + name: Observability plugin + nav_fold: true + ml-commons-plugin: + name: ML Commons plugin + nav_fold: true monitoring-plugins: name: Monitoring plugins nav_fold: true @@ -107,8 +120,15 @@ just_the_docs: # Enable or disable the site search -# Supports true (default) or false -search_enabled: true +# By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality. +# To disable any search from appearing, both `search_enabled` and `use_custom_search` need to be false. +# To use the OpenSearch-driven search, `search_enabled` has to be false and `use_custom_search` needs to be true. +# If `search_enabled` is true, irrespective of the value of `use_custom_search`, the JSON file-based search appears. +# +# `search_enabled` defaults to true +# `use_custom_search` defaults to false +search_enabled: false +use_custom_search: true search: # Split pages into sections that can be searched individually diff --git a/_dashboards/branding.md b/_dashboards/branding.md new file mode 100644 index 00000000..7460abcc --- /dev/null +++ b/_dashboards/branding.md @@ -0,0 +1,126 @@ +--- +layout: default +title: Custom branding +nav_order: 200 +--- + +# OpenSearch Dashboards custom branding +Introduced 1.2 +{: .label .label-purple } + +By default, OpenSearch Dashboards uses the OpenSearch logo, but if you want to use custom branding elements such as the favicon or main Dashboards logo, you can do so by editing `opensearch_dashboards.yml` or by including a custom `opensearch_dashboards.yml` file when you start your OpenSearch cluster. + +For example, if you're using Docker to start your OpenSearch cluster, include the following lines in the `opensearch-dashboards` section of your `docker-compose.yml` file: + +``` +volumes: + - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml +``` + +Doing so replaces the Docker image's default `opensearch_dashboards.yml` with your custom `opensearch_dashboards.yml` file, so be sure to include your desired settings as well. For example, if you want to configure TLS for OpenSearch Dashboards, see [Configure TLS for OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/install/tls). + +Re-launch OpenSearch Dashboards, and OpenSearch Dashboards now uses your custom elements. + +## Branding elements + +The following elements in OpenSearch Dashboards are customizable: + +![OpenSearch customizable branding elements]({{site.url}}{{site.baseurl}}/images/dashboards-branding-labels.png) + +Setting | Corresponding branding element +:--- | :--- +logo | Header logo. See #1 in the image. +mark | OpenSearch Dashboards mark. See #2 in the image. +loadingLogo | Loading logo used when OpenSearch Dashboards is starting. See #3 in the image. +faviconUrl | Website icon. Loads next to the application title. See #4 in the image. +applicationTitle | The application's title. See #5 in the image. + +To start using your own branding elements in OpenSearch Dashboards, first uncomment this section of `opensearch_dashboards.yml`: + +```yml +# opensearchDashboards.branding: + # logo: + # defaultUrl: "" + # darkModeUrl: "" + # mark: + # defaultUrl: "" + # darkModeUrl: "" + # loadingLogo: + # defaultUrl: "" + # darkModeUrl: "" + # faviconUrl: "" + # applicationTitle: "" +``` + +Add the URLs you want to use as branding elements to the appropriate setting. Valid image types are `SVG`, `PNG`, and `GIF`. + +Customization of dark mode Dashboards is also available, but you first must supply a valid link to `defaultUrl`, and then link to your preferred image with `darkModeUrl`. If you don't provide a `darkModeUrl` link, then Dashboards uses the provided `defaultUrl` element for dark mode. You are not required to customize all branding elements, so if you wanted to, it's perfectly valid to change just the logo or any other element. Leave unchanged elements as commented. + +The following example demonstrates how to use `SVG` files as logos but leaves the other elements as defaults. + +```yml +logo: + defaultUrl: "https://example.com/validUrl.svg" + darkModeUrl: "https://example.com/validDarkModeUrl.svg" +# mark: +# defaultUrl: "" +# darkModeUrl: "" +# loadingLogo: +# defaultUrl: "" +# darkModeUrl: "" +# faviconUrl: "" +applicationTitle: "My custom application" +``` + +We recommend linking to images that are hosted on a web server, but if you really want to use locally hosted images, save your images inside `assets`, and then configure `opensearch_dashboards.yml` to use the correct paths. You can access locally stored images through the `ui/assets` folder. + +The following example assumes the default port of 5601 that Dashboards uses and demonstrates how to link to locally stored images. + +```yml +logo: + defaultUrl: "https://localhost:5601/ui/assets/my-own-image.svg" + darkModeUrl: "https://localhost:5601/ui/assets/dark-mode-my-own-image.svg" +mark: + defaultUrl: "https://localhost:5601/ui/assets/my-own-image2.svg" + darkModeUrl: "https://localhost:5601/ui/assets/dark-mode-my-own-image2.svg" +# loadingLogo: +# defaultUrl: "" +# darkModeUrl: "" +# faviconUrl: "" +applicationTitle: "My custom application" +``` + +## Sample configuration + +The following configuration enables the security plugin and SSL within OpenSearch Dashboards and uses custom branding elements to replace the OpenSearch logo and application title. + +```yml +server.host: "0" +opensearch.hosts: ["https://localhost:9200"] +opensearch.ssl.verificationMode: none +opensearch.username: "kibanaserver" +opensearch.password: "kibanaserver" +opensearch.requestHeadersAllowlist: [ authorization,securitytenant ] +#server.ssl.enabled: true +#server.ssl.certificate: /path/to/your/server/certificate +#server.ssl.key: /path/to/your/server/key + +opensearch_security.multitenancy.enabled: true +opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"] +opensearch_security.readonly_mode.roles: ["kibana_read_only"] +# Use this setting if you are running opensearch-dashboards without https +opensearch_security.cookie.secure: false + +opensearchDashboards.branding: + logo: + defaultUrl: "https://example.com/sample.svg" + darkModeUrl: "https://example.com/dark-mode-sample.svg" + # mark: + # defaultUrl: "" + # darkModeUrl: "" + # loadingLogo: + # defaultUrl: "" + # darkModeUrl: "" + # faviconUrl: "" + applicationTitle: "Just some testing" +``` diff --git a/_dashboards/browser-compatibility.md b/_dashboards/browser-compatibility.md index 9848db46..0033e280 100644 --- a/_dashboards/browser-compatibility.md +++ b/_dashboards/browser-compatibility.md @@ -1,7 +1,6 @@ --- layout: default title: Browser compatibility -parent: OpenSearch Dashboards nav_order: 3 --- diff --git a/_dashboards/dql.md b/_dashboards/dql.md index 3e71145f..71be1a61 100644 --- a/_dashboards/dql.md +++ b/_dashboards/dql.md @@ -6,7 +6,7 @@ nav_order: 99 # Dashboards Query Language -Similar to the [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index) that lets you use the HTTP request body to search for data, you can use the Dashbaords Query Language (DQL) in OpenSearch Dashboards to search for data and visualizations. +Similar to the [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index) that lets you use the HTTP request body to search for data, you can use the Dashboards Query Language (DQL) in OpenSearch Dashboards to search for data and visualizations. For example, if you want to see all visualizations of visits to a host based in the US, enter `geo.dest:US` into the search field, and Dashboards refreshes to display all related data. diff --git a/_dashboards/install/docker.md b/_dashboards/install/docker.md index 29608e63..cca5b1c5 100644 --- a/_dashboards/install/docker.md +++ b/_dashboards/install/docker.md @@ -9,7 +9,7 @@ nav_order: 1 You *can* start OpenSearch Dashboards using `docker run` after [creating a Docker network](https://docs.docker.com/engine/reference/commandline/network_create/) and starting OpenSearch, but the process of connecting OpenSearch Dashboards to OpenSearch is significantly easier with a Docker Compose file. -1. Run `docker pull opensearchproject/opensearch-dashboards:{{site.opensearch_version}}`. +1. Run `docker pull opensearchproject/opensearch-dashboards:{{site.opensearch_dashboards_version}}`. 1. Create a [`docker-compose.yml`](https://docs.docker.com/compose/compose-file/) file appropriate for your environment. A sample file that includes OpenSearch Dashboards is available on the OpenSearch [Docker installation page]({{site.url}}{{site.baseurl}}/opensearch/install/docker#sample-docker-compose-file). diff --git a/_dashboards/install/plugins.md b/_dashboards/install/plugins.md index e0fc9d29..7afca2fb 100644 --- a/_dashboards/install/plugins.md +++ b/_dashboards/install/plugins.md @@ -28,6 +28,48 @@ If you don't want to use the all-in-one installation options, you can install th + + 1.3.1 + + 
alertingDashboards          1.3.1.0
+anomalyDetectionDashboards  1.3.1.0
+ganttChartDashboards        1.3.1.0
+indexManagementDashboards   1.3.1.0
+observabilityDashboards     1.3.1.0
+queryWorkbenchDashboards    1.3.1.0
+reportsDashboards           1.3.1.0
+securityDashboards          1.3.1.0
+
+ + + + 1.3.0 + + 
alertingDashboards          1.3.0.0
+anomalyDetectionDashboards  1.3.0.0
+ganttChartDashboards        1.3.0.0
+indexManagementDashboards   1.3.0.0
+observabilityDashboards     1.3.0.0
+queryWorkbenchDashboards    1.3.0.0
+reportsDashboards           1.3.0.0
+securityDashboards          1.3.0.0
+
+ + + + 1.2.0 + + 
alertingDashboards          1.2.0.0
+anomalyDetectionDashboards  1.2.0.0
+ganttChartDashboards        1.2.0.0
+indexManagementDashboards   1.2.0.0
+observabilityDashboards     1.2.0.0
+queryWorkbenchDashboards    1.2.0.0
+reportsDashboards           1.2.0.0
+securityDashboards          1.2.0.0
+
+ + 1.1.0 diff --git a/_dashboards/install/tls.md b/_dashboards/install/tls.md index 546dd36b..22f123bc 100644 --- a/_dashboards/install/tls.md +++ b/_dashboards/install/tls.md @@ -25,7 +25,7 @@ opensearch.hosts: ["https://localhost:9200"] opensearch.ssl.verificationMode: full opensearch.username: "kibanaserver" opensearch.password: "kibanaserver" -opensearch.requestHeadersWhitelist: [ authorization,securitytenant ] +opensearch.requestHeadersAllowlist: [ authorization,securitytenant ] server.ssl.enabled: true server.ssl.certificate: /usr/share/opensearch-dashboards/config/client-cert.pem server.ssl.key: /usr/share/opensearch-dashboards/config/client-cert-key.pem diff --git a/_dashboards/reporting.md b/_dashboards/reporting.md index 4a2247fa..6f1637c1 100644 --- a/_dashboards/reporting.md +++ b/_dashboards/reporting.md @@ -13,12 +13,12 @@ CSV reports have a non-configurable 10,000 row limit. They have no explicit size {: .tip } -## Create reports from Discovery, Visualize, or Dashboard +## Create reports from Discovery, Visualize, Dashboard, or Notebooks Quickly generate an on-demand report from the current view. 1. From the top menu bar, choose **Reporting**. -1. For dashboards or visualizations, choose **Download PDF** or **Download PNG**. From the Discover page, choose **Download CSV**. +1. For dashboards, visualizations, or notebooks, choose **Download PDF** or **Download PNG**. If you're creating a report from the Discover page, choose **Generate CSV**. Reports generate asynchronously in the background and might take a few minutes, depending on the size of the report. A notification appears when your report is ready to download. @@ -32,11 +32,11 @@ Definitions let you generate reports on a periodic schedule. 1. From the navigation panel, choose **Reporting**. 1. Choose **Create**. 1. Under **Report settings**, enter a name and optional description for your report. -1. Choose the **Report Source** (i.e. the page from which the report is generated). You can generate reports from the **Dashboard**, **Visualize**, or **Discover** pages. -1. Select your dashboard, visualization, or saved search. Then choose a time range for the report. +1. Choose the **Report source** (i.e. the page from which the report is generated). You can generate reports from the **Dashboard**, **Visualize**, **Discover** (saved search), or **Notebooks** pages. +1. Select your dashboard, visualization, saved search, or notebook. Then choose a time range for the report. 1. Choose an appropriate file format for the report. -1. (Optional) Add a header or footer to the report. Headers and footers are only available for dashboard or visualization reports. -1. Under **Report trigger**, choose either **On-demand** or **Schedule**. +1. (Optional) Add a header or footer to the report. Headers and footers are only available for dashboard, visualization, and notebook reports. +1. Under **Report trigger**, choose either **On demand** or **Schedule**. For scheduled reports, select either **Recurring** or **Cron based**. You can receive reports daily or at some other time interval. Cron expressions give you even more flexiblity. See [Cron expression reference]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron/) for more information. @@ -55,3 +55,7 @@ This problem can occur for two reasons: - You don't have the correct version of `headless-chrome` to match the operating system on which OpenSearch Dashboards is running. Download the correct version [here](https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0). - You're missing additional dependencies. Install the required dependencies for your operating system from the [additional libraries](https://github.com/opensearch-project/dashboards-reports/blob/main/dashboards-reports/rendering-engine/headless-chrome/README.md#additional-libaries) section. + +### Characters not loading in reports + +You might encounter an issue where UTF-8 encoded characters look fine in your browser, but they don't load in your generated reports because you're missing the required font dependencies. Install the [font dependencies](https://github.com/opensearch-project/dashboards-reports#missing-font-dependencies), and then generate your reports again. diff --git a/_dashboards/search-telemetry.md b/_dashboards/search-telemetry.md new file mode 100644 index 00000000..37fed631 --- /dev/null +++ b/_dashboards/search-telemetry.md @@ -0,0 +1,44 @@ +--- +layout: default +title: Search telemetry +nav_order: 30 +--- + + +# About search telemetry + +You can use search telemetry to analyze search request performance by success or failure in OpenSearch Dashboards. OpenSearch stores telemetry data in the `.kibana_1` index. + +Because there are thousands of concurrent search requests from OpenSearch Dashboards, the heavy traffic can cause significant load in an OpenSearch cluster. + +OpenSearch clusters perform better with search telemetry turned off. +{: .tip } + +## Turn on search telemetry + +Search usage telemetry is turned off by default. To turn it on, you need to set `data.search.usageTelemetry.enabled` to `true` in the `opensearch_dashboards.yml` file. + +You can find the [OpenSearch Dashboards YAML file](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/config/opensearch_dashboards.yml) in the opensearch-project repository on GitHub. + +Turning on telemetry in the `opensearch_dashboards.yml` file overrides the default search telemetry setting of `false` in the [Data plugin configuration file](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/data/config.ts). +{: .note } + +### Turn search telemetry on or off + +The following table shows the `data.search.usageTelemetry.enabled` values you can set in `opensearch_dashboards.yml` to turn search telemetry on or off. + +OpenSearch Dashboards YAML value | Search telemetry status: on or off +:--- | :--- + `true` | On + `false` | Off + `none` | Off + +#### Sample opensearch_dashboards.yml with telemetry enabled + + This OpenSearch Dashboards YAML file excerpt shows the telemetry setting set to `true` to turn on search telemetry: + + ```json +# Set the value of this setting to false to suppress +# search usage telemetry to reduce the load of the OpenSearch cluster. + data.search.usageTelemetry.enabled: true +``` \ No newline at end of file diff --git a/_data/alert.yml b/_data/alert.yml index ecfc87f2..5aceb23b 100644 --- a/_data/alert.yml +++ b/_data/alert.yml @@ -1 +1 @@ -message: "🌡️ [OpenSearch 1.1.0 arrived October 5 with cross-cluster replication, bucket-level alerting, and much, much more. Grab it here!](/downloads.html)" +message: "This is a pre-release version of OpenSearch 2.0.0. Feel free to try it out and provide feedback. If you are looking for the most recent production-ready release, see the [1.x line](https://opensearch.org/lines/1x.html)" diff --git a/_data/footer.yml b/_data/footer.yml index 3024dcb7..5a9c1524 100644 --- a/_data/footer.yml +++ b/_data/footer.yml @@ -20,19 +20,22 @@ columns: - title: 'Resources' links: - #- + #- # title: 'Documentation' # url: 'https://github.com/opensearch/documentation' - - + - title: FAQ url: '/faq/' + - + title: Testimonials + url: '/testimonials.html' - title: 'Brand Guidelines' url: '/brand.html' - title: 'Trademark Usage Policy' url: '/trademark-usage.html' - - + - title: OpenSearch Disambiguation url: '/disambiguation.html' - @@ -45,5 +48,5 @@ columns: # title: 'Facebook' # url: 'http://www.facebook.com/opensearch' - - title: 'E-mail' - url: 'mailto:opensearch@amazon.com' \ No newline at end of file + title: 'Connect' + url: 'https://opensearch.org/connect.html' diff --git a/_data/versions.json b/_data/versions.json index 5fe13f29..fa2f46fe 100644 --- a/_data/versions.json +++ b/_data/versions.json @@ -1,6 +1,11 @@ { - "current": "1.1", - "past": [ + "current": "2.0", + "all": [ + "2.0", + "1.3", + "1.2", + "1.1", "1.0" - ] + ], + "latest": "1.3" } \ No newline at end of file diff --git a/_im-plugin/index-rollups/settings.md b/_im-plugin/index-rollups/settings.md new file mode 100644 index 00000000..fae08dad --- /dev/null +++ b/_im-plugin/index-rollups/settings.md @@ -0,0 +1,22 @@ +--- +layout: default +title: Settings +parent: Index rollups +nav_order: 30 +--- + +# Index rollup settings + +We don't recommend changing these settings; the defaults should work well for most use cases. + +All settings are available using the OpenSearch `_cluster/settings` operation. None require a restart, and all can be marked `persistent` or `transient`. + +Setting | Default | Description +:--- | :--- | :--- +`plugins.rollup.search.backoff_millis` | 1000 milliseconds | The backoff time between retries for failed rollup jobs. +`plugins.rollup.search.backoff_count` | 5 | How many retries the plugin should attempt for failed rollup jobs. +`plugins.rollup.search.search_all_jobs` | false | Whether OpenSearch should return all jobs that match all specified search terms. If disabled, OpenSearch returns just one, as opposed to all, of the jobs that matches the search terms. +`plugins.rollup.dashboards.enabled` | true | Whether rollups are enabled in OpenSearch Dashboards. +`plugins.rollup.enabled` | true | Whether the rollup plugin is enabled. +`plugins.ingest.backoff_millis` | 1000 milliseconds | The backoff time between data ingestions for rollup jobs. +`plugins.ingest.backoff_count` | 5 | How many retries the plugin should attempt for failed ingestions. diff --git a/_im-plugin/index-transforms/index.md b/_im-plugin/index-transforms/index.md index 38d886ef..7fd19571 100644 --- a/_im-plugin/index-transforms/index.md +++ b/_im-plugin/index-transforms/index.md @@ -9,16 +9,16 @@ has_toc: false # Index transforms -Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indices, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways. +Whereas index rollup jobs let you reduce data granularity by rolling up old data into condensed indexes, transform jobs let you create a different, summarized view of your data centered around certain fields, so you can visualize or analyze the data in different ways. For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories. You can use transform jobs in two ways: 1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow. -2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow. +1. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow. -OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indices and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards. +OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indexes and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards. Your use cases will help you decide which method to use to create transform jobs. @@ -26,7 +26,7 @@ Your use cases will help you decide which method to use to create transform jobs If you don't have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose **Index Management**. Select **Transform Jobs**, and choose **Create Transform Job**. -### Step 1: Choose indices +### Step 1: Choose indexes 1. In the **Job name and description** section, specify a name and an optional description for your job. 2. In the **Indices** section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose **Edit data filter**, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/). @@ -34,7 +34,7 @@ If you don't have any data in your cluster, you can use the sample flight data w ### Step 2: Select fields to transform -After specifying the indices, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations. +After specifying the indexes, you can select the fields you want to use in your transform job, as well as whether to use groupings or aggregations. You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the `DestAirportID` field into a target field of `DestAirportID_terms` field, and you can find the grouped airport IDs in your transformed index after the transform job finishes. @@ -44,16 +44,17 @@ On the other hand, aggregations let you perform simple calculations. For example Currently, transform jobs support histogram, date_histogram, and terms groupings. For more information about groupings, see [Bucket Aggregations]({{site.url}}{{site.baseurl}}/opensearch/bucket-agg/). In terms of aggregations, you can select from `sum`, `avg`, `max`, `min`, `value_count`, `percentiles`, and `scripted_metric`. For more information about aggregations, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg/). -2. Repeat step 1 for any other fields that you want to transform. -3. After selecting the fields that you want to transform and verifying the transformation, choose **Next**. +1. Repeat step 1 for any other fields that you want to transform. +1. After selecting the fields that you want to transform and verifying the transformation, choose **Next**. ### Step 3: Specify a schedule You can configure transform jobs to run once or multiple times on a schedule. Transform jobs are enabled by default. -1. For **transformation execution frequency**, select **Define by fixed interval** and specify a **transform interval**. -2. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur. -3. Choose **Next**. +1. Choose whether the job should be **continuous**. Continuous jobs execute at each **transform execution interval** and incrementally transform newly modified buckets, which can include new data added to the source indexes. Non-continuous jobs execute only once. +1. For **transformation execution interval**, specify a transform interval in minutes, hours, or days. This interval dicatates how often continuous jobs should execute, and non-continuous jobs execute once after the interval elapses. +1. Under **Advanced**, specify an optional amount for **Pages per execution**. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur. +1. Choose **Next**. ### Step 4: Review and confirm details @@ -103,7 +104,6 @@ GET finished_flight_job/_search "hits" : [ { "_index" : "finished_flight_job", - "_type" : "_doc", "_id" : "dSNKGb8U3OJOmC4RqVCi1Q", "_score" : 3.845883, "_source" : { @@ -115,7 +115,6 @@ GET finished_flight_job/_search }, { "_index" : "finished_flight_job", - "_type" : "_doc", "_id" : "_D7oqOy7drx9E-MG96U5RA", "_score" : 3.845883, "_source" : { @@ -127,7 +126,6 @@ GET finished_flight_job/_search }, { "_index" : "finished_flight_job", - "_type" : "_doc", "_id" : "YuZ8tOt1OsBA54e84WuAEw", "_score" : 3.6988301, "_source" : { @@ -139,7 +137,6 @@ GET finished_flight_job/_search }, { "_index" : "finished_flight_job", - "_type" : "_doc", "_id" : "W_-e7bVmH6eu8veJeK8ZxQ", "_score" : 3.6988301, "_source" : { diff --git a/_im-plugin/index-transforms/transforms-apis.md b/_im-plugin/index-transforms/transforms-apis.md index 67acd298..43cff3ba 100644 --- a/_im-plugin/index-transforms/transforms-apis.md +++ b/_im-plugin/index-transforms/transforms-apis.md @@ -28,6 +28,7 @@ PUT _plugins/_transform/ { "transform": { "enabled": true, + "continuous": true, "schedule": { "interval": { "period": 1, @@ -78,6 +79,7 @@ PUT _plugins/_transform/ "transform": { "transform_id": "sample", "schema_version": 7, + "continuous": true, "schedule": { "interval": { "start_time": 1621467964243, @@ -128,17 +130,18 @@ You can specify the following options in the HTTP request body: Option | Data Type | Description | Required :--- | :--- | :--- | :--- enabled | Boolean | If true, the transform job is enabled at creation. | No -schedule | JSON | The schedule the transform job runs on. | Yes +continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` field and run based off of newly transformed buckets, as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is false. | No +schedule | Object | The schedule the transform job runs on. | Yes start_time | Integer | The Unix epoch time of the transform job's start time. | Yes description | String | Describes the transform job. | No metadata_id | String | Any metadata to be associated with the transform job. | No source_index | String | The source index whose data to transform. | Yes target_index | String | The target index the newly transformed data is added into. You can create a new index or update an existing one. | Yes -data_selection_query | JSON | The query DSL to use to filter a subset of the source index for the transform job. See [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl) for more information. | Yes -page_size | Integer | The number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1) | Yes +data_selection_query | Object | The query DSL to use to filter a subset of the source index for the transform job. See [query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl) for more information. | Yes +page_size | Integer | The number of buckets IM processes and indexes concurrently. Higher number means better performance but requires more memory. If your machine runs out of memory, IM automatically adjusts this field and retries until the operation succeeds. | Yes groups | Array | Specifies the grouping(s) to use in the transform job. Supported groups are `terms`, `histogram`, and `date_histogram`. For more information, see [Bucket Aggregations]({{site.url}}{{site.baseurl}}/opensearch/bucket-agg). | Yes if not using aggregations source_field | String | The field(s) to transform | Yes -aggregations | JSON | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg). | Yes if not using groups +aggregations | Object | The aggregations to use in the transform job. Supported aggregations are: `sum`, `max`, `min`, `value_count`, `avg`, `scripted_metric`, and `percentiles`. For more information, see [Metric Aggregations]({{site.url}}{{site.baseurl}}/opensearch/metric-agg). | Yes if not using groups ## Update a transform job Introduced 1.0 @@ -250,12 +253,23 @@ PUT _plugins/_transform/ } ``` -The `Update` operation supports the following URL parameters: +The Update operation supports the following URL parameters: Parameter | Description | Required :---| :--- | :--- -`if_seq_no` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence number. | No -`if_primary_term` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence term. | No +`if_seq_no` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence number. | Yes +`if_primary_term` | Only perform the transform operation if the last operation that changed the transform job has the specified sequence term. | Yes + +You can update the following fields: + +Option | Data Type | Description +:--- | :--- | :--- +schedule | Object | The schedule the transform job runs on. Contains the fields `interval.start_time`, `interval.period`, and `interval.unit`. +start_time | Integer | The Unix epoch start time of the transform job. +period | Integer | How often to execute the transform job. +unit | String | The unit of time associated with the execution period. Available options are `Minutes`, `Hours`, and `Days`. +description | Integer | Describes the transform job. +page_size | Integer | The number of buckets IM processes and indexes concurrently. Higher number means better performance but requires more memory. If your machine runs out of memory, IM automatically adjusts this field and retries until the operation succeeds. ## Get a transform job's details Introduced 1.0 @@ -280,6 +294,7 @@ GET _plugins/_transform/ "transform": { "transform_id": "sample", "schema_version": 7, + "continuous": true, "schedule": { "interval": { "start_time": 1621467964243, @@ -347,6 +362,7 @@ GET _plugins/_transform/ "transform": { "transform_id": "sample", "schema_version": 7, + "continuous": true, "schedule": { "interval": { "start_time": 1621467964243, @@ -399,11 +415,11 @@ You can specify these options as the `GET` API operation’s URL parameters to f Parameter | Description | Required :--- | :--- | :--- -from | The starting index to search from. (Default: 0) | No -size | Specifies the amount of results to return (Default: 10) | No +from | The starting index to search from. Default is 0. | No +size | Specifies the amount of results to return. Default is 10. | No search |The search term to use to filter results. | No sortField | The field to sort results with. | No -sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. (Default: ASC) | No +sortDirection | Specifies the direction to sort results in. Can be `ASC` or `DESC`. Default is ASC. | No For example, this request returns two results starting from the eighth index. @@ -584,6 +600,12 @@ GET _plugins/_transform//_explain "sample": { "metadata_id": "PzmjweME5xbgkenl9UpsYw", "transform_metadata": { + "continuous_stats": { + "last_timestamp": 1621883525672, + "documents_behind": { + "sample_index": 72 + } + }, "transform_id": "sample", "last_updated_at": 1621883525873, "status": "finished", @@ -691,7 +713,7 @@ POST _plugins/_transform/_preview Introduced 1.0 {: .label .label-purple } -Deletes a transform job. This operation does not delete the source or target indices. +Deletes a transform job. This operation does not delete the source or target indexes. **Sample Request** @@ -709,7 +731,6 @@ DELETE _plugins/_transform/ { "delete": { "_index": ".opensearch-ism-config", - "_type": "_doc", "_id": "sample", "_version": 4, "result": "deleted", diff --git a/_im-plugin/ism/api.md b/_im-plugin/ism/api.md index 3f7e0d1c..bdc6a565 100644 --- a/_im-plugin/ism/api.md +++ b/_im-plugin/ism/api.md @@ -7,7 +7,7 @@ nav_order: 20 # ISM API -Use the index state management operations to programmatically work with policies and managed indices. +Use the index state management operations to programmatically work with policies and managed indexes. --- @@ -163,8 +163,8 @@ POST _plugins/_ism/add/index_1 } ``` -If you use a wildcard `*` while adding a policy to an index, the ISM plugin interprets `*` as all indices, including system indices like `.opendistro-security`, which stores users, roles, and tenants. A delete action in your policy might accidentally delete all user roles and tenants in your cluster. -Don't use the broad `*` wildcard, and instead add a prefix, such as `my-logs*`, when specifying indices with the `_ism/add` API. +If you use a wildcard `*` while adding a policy to an index, the ISM plugin interprets `*` as all indexes, including system indexes like `.opendistro-security`, which stores users, roles, and tenants. A delete action in your policy might accidentally delete all user roles and tenants in your cluster. +Don't use the broad `*` wildcard, and instead add a prefix, such as `my-logs*`, when specifying indexes with the `_ism/add` API. {: .warning } --- @@ -176,6 +176,8 @@ Introduced 1.0 Updates a policy. Use the `seq_no` and `primary_term` parameters to update an existing policy. If these numbers don't match the existing policy or the policy doesn't exist, ISM throws an error. +It's possible that the policy currently applied to your index isn't the most up-to-date policy available. To see what policy is currently applied to your index, see [Explain index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). To get the most up-to-date version of a policy, see [Get policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy). + #### Request ```json @@ -390,9 +392,9 @@ POST _plugins/_ism/remove/index_1 Introduced 1.0 {: .label .label-purple } -Updates the managed index policy to a new policy (or to a new version of the policy). You can use an index pattern to update multiple indices at once. When updating multiple indices, you might want to include a state filter to only affect certain managed indices. The change policy filters out all the existing managed indices and only applies the change to the ones in the state that you specify. You can also explicitly specify the state that the managed index transitions to after the change policy takes effect. +Updates the managed index policy to a new policy (or to a new version of the policy). You can use an index pattern to update multiple indexes at once. When updating multiple indexes, you might want to include a state filter to only affect certain managed indexes. The change policy filters out all the existing managed indexes and only applies the change to the ones in the state that you specify. You can also explicitly specify the state that the managed index transitions to after the change policy takes effect. -A policy change is an asynchronous background process. The changes are queued and are not executed immediately by the background process. This delay in execution protects the currently running managed indices from being put into a broken state. If the policy you are changing to has only some small configuration changes, then the change takes place immediately. For example, if the policy changes the `min_index_age` parameter in a rollover condition from `1000d` to `100d`, this change takes place immediately in its next execution. If the change modifies the state, actions, or the order of actions of the current state the index is in, then the change happens at the end of its current state before transitioning to a new state. +A policy change is an asynchronous background process. The changes are queued and are not executed immediately by the background process. This delay in execution protects the currently running managed indexes from being put into a broken state. If the policy you are changing to has only some small configuration changes, then the change takes place immediately. For example, if the policy changes the `min_index_age` parameter in a rollover condition from `1000d` to `100d`, this change takes place immediately in its next execution. If the change modifies the state, actions, or the order of actions of the current state the index is in, then the change happens at the end of its current state before transitioning to a new state. In this example, the policy applied on the `index_1` index is changed to `policy_1`, which could either be a completely new policy or an updated version of its existing policy. The process only applies the change if the index is currently in the `searches` state. After this change in policy takes place, `index_1` transitions to the `delete` state. @@ -428,7 +430,7 @@ POST _plugins/_ism/change_policy/index_1 Introduced 1.0 {: .label .label-purple } -Retries the failed action for an index. For the retry call to succeed, ISM must manage the index, and the index must be in a failed state. You can use index patterns (`*`) to retry multiple failed indices. +Retries the failed action for an index. For the retry call to succeed, ISM must manage the index, and the index must be in a failed state. You can use index patterns (`*`) to retry multiple failed indexes. #### Request @@ -456,7 +458,7 @@ POST _plugins/_ism/retry/index_1 Introduced 1.0 {: .label .label-purple } -Gets the current state of the index. You can use index patterns to get the status of multiple indices. +Gets the current state of the index. You can use index patterns to get the status of multiple indexes. #### Request @@ -475,6 +477,40 @@ GET _plugins/_ism/explain/index_1 } ``` +Optionally, you can add the `show_policy` parameter to your request's path to get the policy that is currently applied to your index, which is useful for seeing whether the policy applied to your index is the latest one. To get the most up-to-date policy, see [Get Policy API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#get-policy). + +#### Request + +```json +GET _plugins/_ism/explain/index_1?show_policy=true +``` + +#### Sample response + +```json +{ + "index_1": { + "index.plugins.index_state_management.policy_id": "sample-policy", + "index.opendistro.index_state_management.policy_id": "sample-policy", + "index": "index_1", + "index_uuid": "gCFlS_zcTdih8xyxf3jQ-A", + "policy_id": "sample-policy", + "enabled": true, + "policy": { + "policy_id": "sample-policy", + "description": "ingesting logs", + "last_updated_time": 1647284980148, + "schema_version": 13, + "error_notification": null, + "default_state": "ingest", + "states": [...], + "ism_template": null + } + }, + "total_managed_indices": 1 +} +``` + The `plugins.index_state_management.policy_id` setting is deprecated starting from ODFE version 1.13.0. We retain this field in the response API for consistency. --- @@ -497,7 +533,6 @@ DELETE _plugins/_ism/policies/policy_1 ```json { "_index": ".opendistro-ism-config", - "_type": "_doc", "_id": "policy_1", "_version": 3, "result": "deleted", diff --git a/_im-plugin/ism/index.md b/_im-plugin/ism/index.md index 8004146c..61d761b2 100644 --- a/_im-plugin/ism/index.md +++ b/_im-plugin/ism/index.md @@ -12,7 +12,7 @@ has_toc: false OpenSearch Dashboards {: .label .label-yellow :} -If you analyze time-series data, you likely prioritize new data over old data. You might periodically perform certain operations on older indices, such as reducing replica count or deleting them. +If you analyze time-series data, you likely prioritize new data over old data. You might periodically perform certain operations on older indexes, such as reducing replica count or deleting them. Index State Management (ISM) is a plugin that lets you automate these periodic, administrative operations by triggering them based on changes in the index age, index size, or number of documents. Using the ISM plugin, you can define *policies* that automatically handle index rollovers or deletions to fit your use case. @@ -39,8 +39,8 @@ You can use the visual editor or JSON editor to create policies. Compared to the 2. Choose **Create policy**. 3. Choose **Visual editor**. 4. In the **Policy info** section, enter a policy ID and an optional description. -5. In the **Error notification** section, set up an optional error notification that gets sent whenever a policy execution fails. For more information, see [Error notifications]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#error-notifications). -6. In **ISM templates**, enter any ISM template patterns to automatically apply this policy to existing and future indices. For example, if you specify a template of `sample-index*`, the ISM plugin automatically applies this policy to any indices whose names start with `sample-index`. +5. In the **Error notification** section, set up an optional error notification that gets sent whenever a policy execution fails. For more information, see [Error notifications]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#error-notifications). If you're using auto rollovers in your policy, we recommend setting up error notifications, which notify you of unexpectedly large indexes if rollovers fail. +6. In **ISM templates**, enter any ISM template patterns to automatically apply this policy to future indexes. For example, if you specify a template of `sample-index*`, the ISM plugin automatically applies this policy to any indexes whose names start with `sample-index`. Your pattern cannot contain any of the following characters: `:`, `"`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, and `<`. 7. In **States**, add any states you want to include in the policy. Each state has [actions]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/#actions) the plugin executes when the index enters a certain state, and [transitions]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies/#transitions), which have conditions that, when met, transition the index into a destination state. The first state you create in a policy is automatically set as the initial state. Each policy must have at least one state, but actions and transitions are optional. 8. Choose **Create**. @@ -54,10 +54,10 @@ You can use the visual editor or JSON editor to create policies. Compared to the 5. In the **Define policy** section, enter your policy. 6. Choose **Create**. -After you create a policy, your next step is to attach it to an index or indices. +After you create a policy, your next step is to attach it to an index or indexes. You can set up an `ism_template` in the policy so when an index that matches the ISM template pattern is created, the plugin automatically attaches the policy to the index. -The following example demonstrates how to create a policy that automatically gets attached to all indices whose names start with `index_name-`. +The following example demonstrates how to create a policy that automatically gets attached to all indexes whose names start with `index_name-`. ```json PUT _plugins/_ism/policies/policy_id @@ -76,7 +76,7 @@ PUT _plugins/_ism/policies/policy_id If you have more than one template that matches an index pattern, ISM uses the priority value to determine which template to apply. -For an example ISM template policy, see [Sample policy with ISM template]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#sample-policy-with-ism-template). +For an example ISM template policy, see [Sample policy with ISM template for auto rollover]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#sample-policy-with-ism-template-for-auto-rollover). Older versions of the plugin include the `policy_id` in an index template, so when an index is created that matches the index template pattern, the index will have the policy attached to it: @@ -94,13 +94,13 @@ PUT _index_template/ } ``` -The `opendistro.index_state_management.policy_id` setting is deprecated. You can continue to automatically manage newly created indices with the ISM template field. +The `opendistro.index_state_management.policy_id` setting is deprecated. You can continue to automatically manage newly created indexes with the ISM template field. {: .note } -### Step 2: Attach policies to indices +### Step 2: Attach policies to indexes -1. Choose **Indices**. -2. Choose the index or indices that you want to attach your policy to. +1. Choose **indexes**. +2. Choose the index or indexes that you want to attach your policy to. 3. Choose **Apply policy**. 4. From the **Policy ID** menu, choose the policy that you created. You can see a preview of your policy. @@ -112,13 +112,13 @@ After you attach a policy to an index, ISM creates a job that runs every 5 minut ISM does not run jobs if the cluster state is red. -### Step 3: Manage indices +### Step 3: Manage indexes -1. Choose **Managed Indices**. -2. To change your policy, see [Change Policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/managedindices#change-policy). +1. Choose **Managed indexes**. +2. To change your policy, see [Change Policy]({{site.url}}{{site.baseurl}}/im-plugin/ism/managedindexes#change-policy). 3. To attach a rollover alias to your index, select your policy and choose **Add rollover alias**. Make sure that the alias that you enter already exists. For more information about the rollover operation, see [rollover]({{site.url}}{{site.baseurl}}/im-plugin/ism/policies#rollover). 4. To remove a policy, choose your policy, and then choose **Remove policy**. 5. To retry a policy, choose your policy, and then choose **Retry policy**. -For information about managing your policies, see [Managed Indices]({{site.url}}{{site.baseurl}}/im-plugin/ism/managedindices/). +For information about managing your policies, see [Managed indexes]({{site.url}}{{site.baseurl}}/im-plugin/ism/managedindexes/). diff --git a/_im-plugin/ism/managedindices.md b/_im-plugin/ism/managedindexes.md similarity index 100% rename from _im-plugin/ism/managedindices.md rename to _im-plugin/ism/managedindexes.md diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index fa055862..c2a2ccda 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -10,7 +10,7 @@ has_children: false Policies are JSON documents that define the following: -- The *states* that an index can be in, including the default state for new indices. For example, you might name your states "hot," "warm," "delete," and so on. For more information, see [States](#states). +- The *states* that an index can be in, including the default state for new indexes. For example, you might name your states "hot," "warm," "delete," and so on. For more information, see [States](#states). - Any *actions* that you want the plugin to take when an index enters a state, such as performing a rollover. For more information, see [Actions](#actions). - The conditions that must be met for an index to move into a new state, known as *transitions*. For example, if an index is more than eight weeks old, you might want to move it to the "delete" state. For more information, see [Transitions](#transitions). @@ -57,7 +57,9 @@ Field | Description | Type | Required Actions are the steps that the policy sequentially executes on entering a specific state. -They are executed in the order in which they are defined. +ISM executes actions in the order in which they are defined. For example, if you define actions [A,B,C,D], ISM executes action A, and then goes into a sleep period based on the cluster setting `plugins.index_state_management.job_interval`. Once the sleep period ends, ISM continues to execute the remaining actions. However, if ISM cannot successfully execute action A, the operation ends, and actions B, C, and D do not get executed. + +Optionally, you can define an action's timeout period, which, if exceeded, forcibly fails the action. For example, if timeout is set to `1d`, and ISM has not completed the action within one day, even after retries, the action fails. This table lists the parameters that you can define for an action. @@ -97,6 +99,7 @@ ISM supports the following operations: - [read_only](#read_only) - [read_write](#read_write) - [replica_count](#replica_count) +- [shrink](#shrink) - [close](#close) - [open](#open) - [delete](#delete) @@ -160,6 +163,59 @@ Parameter | Description | Type | Required For information about setting replicas, see [Primary and replica shards]({{site.url}}{{site.baseurl}}/opensearch#primary-and-replica-shards). +### shrink + +Allows you to reduce the number of primary shards in your indexes. With this action, you can specify: + +- The number of primary shards that the target index should contain. +- A max shard size for the primary shards in the target index. +- Specify a percentage to shrink the number of primary shards in the target index. + +```json +"shrink": { + "num_new_shards": 1, + "target_index_name_template": { + "source": "{{ctx.index}}_shrunken" + }, + "aliases": [ + "my-alias": {} + ], + "force_unsafe": false +} +``` + +Parameter | Description | Type | Example | Required +:--- | :--- |:--- |:--- | +`num_new_shards` | The maximum number of primary shards in the shrunken index. | integer | `5` | Yes, however it cannot be used with `max_shard_size` or `percentage_of_source_shards` +`max_shard_size` | The maximum size in bytes of a shard for the target index. | keyword | `5gb` | Yes, however it cannot be used with `num_new_shards` or `percentage_of_source_shards` +`percentage_of_source_shards` | Percentage of the number of original primary shards to shrink. This parameter indicates the minimum percentage to use when shrinking the number of primary shards. Must be between 0.0 and 1.0, exclusive. | Percentage | `0.5` | Yes, however it cannot be used with `max_shard_size` or `num_new_shards` +`target_index_name_template` | The name of the shrunken index. Accepts strings and the Mustache variables `{{ctx.index}}` and `{{ctx.indexUuid}}`. | `string` or Mustache template | `{"source": "{{ctx.index}}_shrunken"}` | No +`aliases` | Aliases to add to the new index. | object | `myalias` | No, but must be an array of alias objects +`force_unsafe` | If true, executes the shrink action even if there are no replicas. | boolean | `false` | No + +If you want to add `aliases` to the action, the parameter must include an array of [alias objects]({{site.url}}{{site.baseurl}}/opensearch/rest-api/alias/). For example, + +```json +"aliases": [ + { + "my-alias": {} + }, + { + "my-second-alias": { + "is_write_index": false, + "filter": { + "multi_match": { + "query": "QUEEN", + "fields": ["speaker", "text_entry"] + } + }, + "index_routing" : "1", + "search_routing" : "1" + } + }, +] +``` + ### close Closes the managed index. @@ -170,7 +226,7 @@ Closes the managed index. } ``` -Closed indices remain on disk, but consume no CPU or memory. You can't read from, write to, or search closed indices. +Closed indexes remain on disk, but consume no CPU or memory. You can't read from, write to, or search closed indexes. Closing an index is a good option if you need to retain data for longer than you need to actively search it and have sufficient disk space on your data nodes. If you need to search the data again, reopening a closed index is simpler than restoring an index from a snapshot. @@ -198,14 +254,17 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will be performed if the value **has reached** or _exceeded_ the configured limit **when the check is performed**. For example with `min_size` configured to a value of 100GiB, ISM might check the index at 99 GiB and not perform the rollover. However, if the index has grown past the limit (e.g. 105GiB) by the next check, the operation is performed. + The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. Parameter | Description | Type | Example | Required :--- | :--- |:--- |:--- | -`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of the primaries is 100 GiB, so the rollover occurs. ISM doesn't check indices continually, so it doesn't roll over indices at exactly 100 GiB. Instead, if an index is continuously growing, ISM might check it at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. | `string` | `20gb` or `5mb` | No -`min_doc_count` | The minimum number of documents required to roll over the index. | `number` | `2000000` | No -`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. | `string` | `5d` or `7h` | No +`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. See **Important** note above. | `string` | `20gb` or `5mb` | No +`min_primary_shard_size` | The minimum storage size of a **single primary shard** required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and **one of** the primary shards in the index has a size greater than the condition, the rollover occurs. See **Important** note above. | `string` | `20gb` or `5mb` | No +`min_doc_count` | The minimum number of documents required to roll over the index. See **Important** note above. | `number` | `2000000` | No +`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. See **Important** note above. | `string` | `5d` or `7h` | No ```json { @@ -215,6 +274,14 @@ Parameter | Description | Type | Example | Required } ``` +```json +{ + "rollover": { + "min_primary_shard_size": "30gb" + } +} +``` + ```json { "rollover": { @@ -307,27 +374,27 @@ Parameter | Description | Type ### snapshot -Backup your cluster’s indices and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/). +Backup your cluster’s indexes and state. For more information about snapshots, see [Take and restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshot-restore/). The `snapshot` operation has the following parameters: Parameter | Description | Type | Required | Default :--- | :--- |:--- |:--- | `repository` | The repository name that you register through the native snapshot API operations. | `string` | Yes | - -`snapshot` | The name of the snapshot. | `string` | Yes | - +`snapshot` | The name of the snapshot. Accepts strings and the Mustache variables `{{ctx.index}}` and `{{ctx.indexUuid}}`. If the Mustache variables are invalid, then the snapshot name defaults to the index's name. | `string` or Mustache template | Yes | - ```json { "snapshot": { "repository": "my_backup", - "snapshot": "my_snapshot" + "snapshot": "{{ctx.indexUuid}}" } } ``` ### index_priority -Set the priority for the index in a specific state. Unallocated shards of indices are recovered in the order of their priority, whenever possible. The indices with higher priority values are recovered first followed by the indices with lower priority values. +Set the priority for the index in a specific state. Unallocated shards of indexes are recovered in the order of their priority, whenever possible. The indexes with higher priority values are recovered first followed by the indexes with lower priority values. The `index_priority` operation has the following parameter: @@ -375,7 +442,7 @@ Parameter | Description | Type | Required Transitions define the conditions that need to be met for a state to change. After all actions in the current state are completed, the policy starts checking the conditions for transitions. -Transitions are evaluated in the order in which they are defined. For example, if the conditions for the first transition are met, then this transition takes place and the rest of the transitions are dismissed. +ISM evaluates transitions in the order in which they are defined. For example, if you define transitions: [A,B,C,D], ISM iterates through this list of transitions until it finds a transition that evaluates to `true`, it then stops and sets the next state to the one defined in that transition. On its next execution, ISM dismisses the rest of the transitions and starts in that new state. If you don't specify any conditions in a transition and leave it empty, then it's assumed to be the equivalent of always true. This means that the policy transitions the index to this state the moment it checks. @@ -392,7 +459,7 @@ Parameter | Description | Type | Required :--- | :--- |:--- |:--- | `min_index_age` | The minimum age of the index required to transition. | `string` | No `min_doc_count` | The minimum document count of the index required to transition. | `number` | No -`min_size` | The minimum size of the index required to transition. | `string` | No +`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to transition. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so your index is transitioned to the next state. | `string` | No `cron` | The `cron` job that triggers the transition if no other transition happens first. | `object` | No `cron.cron.expression` | The `cron` expression that triggers the transition. | `string` | Yes `cron.cron.timezone` | The timezone that triggers the transition. | `string` | Yes @@ -412,7 +479,7 @@ The following example transitions the index to a `cold` state after a period of ISM checks the conditions on every execution of the policy based on the set interval. -This example uses the `cron` condition to transition indices every Saturday at 5:00 PT: +This example uses the `cron` condition to transition indexes every Saturday at 5:00 PT: ```json "transitions": [ @@ -533,10 +600,12 @@ The destination system **must** return a response otherwise the `error_notificat You can use the same options for `ctx` variables as the [notification](#notification) operation. -## Sample policy with ISM template +## Sample policy with ISM template for auto rollover The following sample template policy is for a rollover use case. +If you want to skip rollovers for an index, set `index.plugins.index_state_management.rollover_skip` to `true` in the settings of that index. + 1. Create a policy with an `ism_template` field: ```json @@ -612,7 +681,7 @@ The following sample template policy is for a rollover use case. ## Example policy -The following example policy implements a `hot`, `warm`, and `delete` workflow. You can use this policy as a template to prioritize resources to your indices based on their levels of activity. +The following example policy implements a `hot`, `warm`, and `delete` workflow. You can use this policy as a template to prioritize resources to your indexes based on their levels of activity. In this case, an index is initially in a `hot` state. After a day, it changes to a `warm` state, where the number of replicas increases to 5 to improve the read performance. @@ -630,7 +699,8 @@ After 30 days, the policy moves this index into a `delete` state. The service se "actions": [ { "rollover": { - "min_index_age": "1d" + "min_index_age": "1d", + "min_primary_shard_size": "30gb" } } ], @@ -678,7 +748,11 @@ After 30 days, the policy moves this index into a `delete` state. The service se } ] } - ] + ], + "ism_template": { + "index_patterns": ["log*"], + "priority": 100 + } } } ``` diff --git a/_im-plugin/ism/settings.md b/_im-plugin/ism/settings.md index 77ce3a92..31fbe5a2 100644 --- a/_im-plugin/ism/settings.md +++ b/_im-plugin/ism/settings.md @@ -17,6 +17,7 @@ Setting | Default | Description :--- | :--- | :--- `plugins.index_state_management.enabled` | True | Specifies whether ISM is enabled or not. `plugins.index_state_management.job_interval` | 5 minutes | The interval at which the managed index jobs are run. +`plugins.index_state_management.jitter` | 0.6 | A randomized delay that is added to a job's base run time to prevent a surge of activity from all indices at the same time. A value of 0.6 means a delay of 0-60% of a job interval is added to the base interval. For example, if you have a base interval time of 30 minutes, a value of 0.6 means an amount anywhere between 0 to 18 minutes gets added to your job interval. Maximum is 1, which means an additional interval time of 100%. This maximum cannot exceed `plugins.jobscheduler.jitter_limit`, which also has a default of 0.6. For example, if `plugins.index_state_management.jitter` is set to 0.8, ISM uses `plugins.jobscheduler.jitter_limit` of 0.6 instead. `plugins.index_state_management.coordinator.sweep_period` | 10 minutes | How often the routine background sweep is run. `plugins.index_state_management.coordinator.backoff_millis` | 50 milliseconds | The backoff time between retries for failures in the `ManagedIndexCoordinator` (such as when we update managed indices). `plugins.index_state_management.coordinator.backoff_count` | 2 | The count of retries for failures in the `ManagedIndexCoordinator`. diff --git a/_includes/footer.html b/_includes/footer.html index 1966ff1b..16995615 100644 --- a/_includes/footer.html +++ b/_includes/footer.html @@ -3,10 +3,10 @@

OpenSearch Links

- + {% for column in site.data.footer.columns %}
- +

{{ column.title }}

diff --git a/_layouts/default.html b/_layouts/default.html index c5408662..13819568 100755 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -57,7 +57,6 @@ layout: table_wrappers