Add SQL and PPL features to notebooks

Add composable index templates

_dashboards/notebooks.md

@@ -1,28 +1,30 @@
 ---
 layout: default
-title: Notebooks (experimental)
+title: Notebooks
 nav_order: 50
 redirect_from: /docs/notebooks/
 has_children: false
 ---
 
-# OpenSearch Dashboards notebooks (experimental)
+# Notebooks
 
-Notebooks have a known issue with [tenants]({{site.url}}{{site.baseurl}}/security-plugin/access-control/multi-tenancy/). If you open a notebook and can't see its visualizations, you might be under the wrong tenant, or you might not have access to the tenant at all.
+An OpenSearch Dashboards notebook is an interface that lets you easily combine code snippets, live visualizations, and narrative text in a single notebook interface.
-{: .warning }
-
-An OpenSearch Dashboards notebook is an interface that lets you easily combine live visualizations and narrative text in a single notebook interface.
 
 Notebooks let you interactively explore data by running different visualizations that you can share with team members to collaborate on a project.
 
-A notebook is a document composed of two elements: OpenSearch Dashboards visualizations and paragraphs (Markdown). Choose multiple timelines to compare and contrast visualizations.
+A notebook is a document composed of two elements: code blocks (Markdown/SQL/PPL) and visualizations. Choose multiple timelines to compare and contrast visualizations.
+
+You can also generate [reports]({{site.url}}{{site.baseurl}}/dashboards/reporting/) directly from your notebooks.
 
 Common use cases include creating postmortem reports, designing runbooks, building live infrastructure reports, and writing documentation.
 
+Tenants in OpenSearch Dashboards are spaces for saving notebooks and other OpenSearch Dashboards objects. For more information, see [OpenSearch Dashboards multi-tenancy]({{site.url}}{{site.baseurl}}/security-plugin/access-control/multi-tenancy/).
+{: .note }
 
-## Get Started with notebooks
 
-To get started, choose **OpenSearch Dashboards Notebooks** within OpenSearch Dashboards.
+## Get started with notebooks
+
+To get started, choose **Notebooks** within OpenSearch Dashboards.
 
 
 ### Step 1: Create a notebook
@@ -32,28 +34,58 @@ A notebook is an interface for creating reports.
 1. Choose **Create notebook** and enter a descriptive name.
 1. Choose **Create**.
 
-Choose **Notebook actions** to rename, duplicate, or delete a notebook.
+Choose **Actions** to rename, duplicate, or delete a notebook.
 
 
 ### Step 2: Add a paragraph
 
-Paragraphs combine text and visualizations for describing data.
+Paragraphs combine code blocks and visualizations for describing data.
 
+#### Add a code block
 
-#### Add a markdown paragraph
+Code blocks support markdown, SQL, and PPL languages.
 
-1. To add text, choose **Add markdown paragraph**.
+Specify the input language on the first line using `%[language type]` syntax.
-1. Add rich text with markdown syntax.
+For example, type `%md` for markdown, `%sql` for SQL, and `%ppl` for PPL.
+
+##### Sample markdown block
+
+```
+%md
+Add in text formatted in markdown.
+```
 
 ![Markdown paragraph]({{site.url}}{{site.baseurl}}/images/markdown-notebook.png)
 
+##### Sample SQL block
 
-#### Add a visualization paragraph
+```sql
+%sql
+Select * from opensearch_dashboards_sample_data_flights limit 20;
+```
 
-1. To add a visualization, choose **Add OpenSearch Dashboards visualization paragraph**.
+![SQL paragraph]({{site.url}}{{site.baseurl}}/images/sql-notebook.png)
+
+##### Sample PPL block
+
+```
+%ppl
+source=opensearch_dashboards_sample_data_logs | head 20
+```
+
+![PPL paragraph]({{site.url}}{{site.baseurl}}/images/ppl-notebook.png)
+
+
+#### Add a visualization
+
+1. To add a visualization, choose **Add paragraph** and select **Visualization**.
 1. In **Title**, select your visualization and choose a date range. You can choose multiple timelines to compare and contrast visualizations.
 1. To run and save a paragraph, choose **Run**.
 
+![Visualization paragraph]({{site.url}}{{site.baseurl}}/images/visualization-notebook.png)
+
+## Paragraph actions
+
 You can perform the following actions on paragraphs:
 
 - Add a new paragraph to the top of a report.
@@ -62,3 +94,25 @@ You can perform the following actions on paragraphs:
 - Clear the outputs of all paragraphs.
 - Delete all the paragraphs.
 - Move paragraphs up and down.
+
+## Sample notebooks
+
+We prepared the following sample notebooks that showcase a variety of use cases:
+
+- Using SQL to query the OpenSearch Dashboards sample flight data.
+- Using PPL to query the OpenSearch Dashboards sample web logs data.
+- Using PPL and visualizations to perform sample root cause event analysis on the OpenSearch Dashboards sample web logs data.
+
+To add a sample notebook, choose **Actions** and select **Add sample notebooks**.
+
+## Create a report
+
+You can use notebooks to create PNG and PDF reports:
+
+1. From the top menu bar, choose **Reporting actions**.
+1. You can choose to **Download PDF** or **Download PNG**.
+
+   Reports generate asynchronously in the background and might take a few minutes, depending on the size of the report. A notification appears when your report is ready to download.
+
+1. To create a schedule-based report, choose **Create report definition**. For steps to create a report definition, see [Create reports using a definition]({{site.url}}{{site.baseurl}}/dashboards/reporting#create-reports-using-a-definition).
+1. To see all your reports, choose **View all reports**.

_dashboards/reporting.md

@@ -9,6 +9,9 @@ nav_order: 20
 
 You can use OpenSearch Dashboards to create PNG, PDF, and CSV reports. To create reports, you must have the correct permissions. For a summary of the predefined roles and the permissions they grant, see the [security plugin]({{site.url}}{{site.baseurl}}/security-plugin/access-control/users-roles#predefined-roles).
 
+CSV reports have a non-configurable 10,000 row limit. They have no explicit size limit (e.g. in MB), but extremely large documents could cause report generation to fail with an out of memory error from the V8 JavaScript engine.
+{: .tip }
+
 
 ## Create reports from Discovery, Visualize, or Dashboard
 

_opensearch/index-templates.md

@@ -8,16 +8,7 @@ nav_order: 14
 
 Index templates let you initialize new indices with predefined mappings and settings. For example, if you continuously index log data, you can define an index template so that all of these indices have the same number of shards and replicas.
 
----
+### Create a template
-
-#### Table of contents
-1. TOC
-{:toc}
-
-
----
-
-## Create a template
 
 To create an index template, use a POST request:
 
@@ -106,7 +97,7 @@ GET logs-2020-01-01
 
 Any additional indices that match this pattern---`logs-2020-01-02`, `logs-2020-01-03`, and so on---will inherit the same mappings and settings.
 
-## Retrieve a template
+### Retrieve a template
 
 To list all index templates:
 
@@ -138,7 +129,7 @@ To check if a specific template exists:
 HEAD _index_template/<name>
 ```
 
-## Configure multiple templates
+### Configure multiple templates
 
 You can create multiple index templates for your indices. If the index name matches more than one template, OpenSearch merges all mappings and settings from all matching templates and applies them to the index.
 
@@ -184,7 +175,7 @@ PUT _index_template/template-02
 
 Because `template-02` has a higher `priority` value, it takes precedence over `template-01` . The `logs-2020-01-02` index would have the `number_of_shards` value as 3.
 
-## Delete a template
+### Delete a template
 
 You can delete an index template using its name:
 
@@ -192,11 +183,225 @@ You can delete an index template using its name:
 DELETE _index_template/daily_logs
 ```
 
+## Composable index templates
+
+Managing multiple index templates has the following challenges:
+
+- If you have duplication between index templates, storing these index templates results in a bigger cluster state.
+- If you want to make a change across all your index templates, you have to manually make the change for each template.
+- If an index matches multiple templates, OpenSearch might merge the templates in an unexpected way that you discover only after an index is created.
+
+You can use composable index templates to overcome these challenges. Composable index templates let you abstract common settings, mappings, and aliases into a reusable building block called a component template.
+
+You can combine component templates to compose an index template.
+
+Settings and mappings that you specify directly in the [create index]({{site.url}}{{site.baseurl}}/opensearch/rest-api/create-index/) request override any settings or mappings specified in an index template and its component templates.
+{: .note }
+
+### Create a component template
+
+Let's define two component templates⁠---`component_template_1` and `component_template_2`:
+
+#### Component template 1
+
+```json
+PUT _component_template/component_template_1
+{
+  "template": {
+    "mappings": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        }
+      }
+    }
+  }
+}
+```
+
+#### Component template 2
+
+```json
+PUT _component_template/component_template_2
+{
+  "template": {
+    "mappings": {
+      "properties": {
+        "ip_address": {
+          "type": "ip"
+        }
+      }
+    }
+  }
+}
+```
+
+### Use component templates to create an index template
+
+When creating index templates, you need to include the component templates in a `composed_of` list.
+
+OpenSearch applies the component templates in the order in which you specify them within the index template. The settings, mappings, and aliases that you specify inside the index template are applied last.
+
+```json
+PUT _index_template/daily_logs
+{
+  "index_patterns": [
+    "logs-2020-01-*"
+  ],
+  "template": {
+    "aliases": {
+      "my_logs": {}
+    },
+    "settings": {
+      "number_of_shards": 2,
+      "number_of_replicas": 1
+    },
+    "mappings": {
+      "properties": {
+        "timestamp": {
+          "type": "date",
+          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
+        },
+        "value": {
+          "type": "double"
+        }
+      }
+    }
+  },
+  "priority": 200,
+  "composed_of": [
+    "component_template_1",
+    "component_template_2"
+  ],
+  "version": 3,
+  "_meta": {
+    "description": "using component templates"
+  }
+}
+```
+
+If you create an index named `logs-2020-01-01`, you can see that it derives its mappings and settings from both the component templates:
+
+```json
+PUT logs-2020-01-01
+GET logs-2020-01-01
+```
+
+#### Sample response
+
+```json
+{
+  "logs-2020-01-01": {
+    "aliases": {
+      "my_logs": {}
+    },
+    "mappings": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        },
+        "ip_address": {
+          "type": "ip"
+        },
+        "timestamp": {
+          "type": "date",
+          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
+        },
+        "value": {
+          "type": "double"
+        }
+      }
+    },
+    "settings": {
+      "index": {
+        "creation_date": "1625382479459",
+        "number_of_shards": "2",
+        "number_of_replicas": "1",
+        "uuid": "rYUlpOXDSUSuZifQLPfa5A",
+        "version": {
+          "created": "7100299"
+        },
+        "provided_name": "logs-2020-01-01"
+      }
+    }
+  }
+}
+```
+
+### Simulate multi-component templates
+
+For index templates composed of multiple component templates, you can simulate applying a new template to verify whether the settings are applied as you expect.
+
+To simulate the settings that would be applied to a specific index name:
+
+```json
+POST _index_template/_simulate_index/<index_name>
+```
+
+To simulate the settings that would be applied from an existing template:
+
+```json
+POST _index_template/_simulate/<index_template>
+```
+
+You can also specify a template definition in the simulate request:
+
+```json
+POST _index_template/_simulate
+{
+  "index_patterns": [
+    "logs-2020-01-*"
+  ],
+  "template": {
+    "settings" : {
+        "index.number_of_shards" : 3
+    }
+  },
+  "composed_of": ["component_template_1", "component_template_2"]
+}
+```
+
+The `_simulate` API returns the final settings, mappings, and aliases that will be applied to indices that match the index pattern. You can also see any overlapping templates whose configuration is superseded by the simulated template body or higher priority templates:
+
+```json
+{
+  "template" : {
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "3"
+      }
+    },
+    "mappings" : {
+      "properties" : {
+        "@timestamp" : {
+          "type" : "date"
+        },
+        "ip_address" : {
+          "type" : "ip"
+        }
+      }
+    },
+    "aliases" : { }
+  },
+  "overlapping" : [
+    {
+      "name" : "daily_logs",
+      "index_patterns" : [
+        "logs-2020-01-*"
+      ]
+    }
+  ]
+}
+```
+
 ## Index template options
 
 You can specify the following template options:
 
 Option | Type | Description | Required
 :--- | :--- | :--- | :---
-`priority` | `Number` | The priority of the index template.  | No
+`template` | `Object` |  Specify index settings, mappings, and aliases. | No
-`create` | `Boolean` | Whether this index template should replace an existing one. | No
+`priority` | `Integer` | The priority of the index template.  | No
+`composed_of` | `String array` |  The names of component templates applied on a new index together with the current template.  | No
+`version` | `Integer` | Specify a version number to simplify template management. Default is `null`. | No
+`_meta ` | `Object` | Specify meta information about the template. | No

_opensearch/rest-api/create-alias.md

@@ -0,0 +1,80 @@
+---
+layout: default
+title: Alias
+parent: REST API reference
+grand_parent: OpenSearch
+nav_order: 8
+---
+
+# Alias
+
+An alias is a virtual pointer that you can use to reference one or more indices. Creating and updating aliases are atomic operations, so you can reindex your data and point an alias at it without any downtime.
+
+
+## Example
+
+```json
+POST _aliases
+{
+  "actions": [
+    {
+      "add": {
+        "index": "movies",
+        "alias": "movies-alias1"
+      }
+    },
+    {
+      "remove": {
+        "index": "old-index",
+        "alias": "old-index-alias"
+      }
+    }
+
+  ]
+}
+```
+
+## Path and HTTP methods
+
+```
+POST _aliases
+```
+
+## URL parameters
+
+All alias parameters are optional.
+
+Parameter | Data Type | Description
+:--- | :--- | :---
+master_timeout | Time | The amount of time to wait for a response from the master node. Default is `30s`.
+timeout | Time | The amount of time to wait for a response from the cluster. Default is `30s`.
+
+## Request body
+
+In your request body, you need to specify what action to take, the alias name, and the index you want to associate with the alias. Other fields are optional.
+
+Field | Data Type | Description | Required
+:--- | :--- | :--- | :---
+actions | Array | Set of actions you want to perform on the index. Valid options are: `add`, `remove`, and `remove_index`. You must have at least one action in the array. | Yes
+add | N/A | Adds an alias to the specified index. | No
+remove | N/A | Removes an alias from the specified index. | No
+remove_index | N/A | Deletes an index. | No
+index | String | Name of the index you want to associate with the alias. Supports wildcard expressions. | Yes if you don't supply an `indices` field in the body.
+indices | Array | Array of index names you want to associate with the alias. | Yes if you don't supply an `index` field in the body.
+alias | String | The name of the alias. | Yes if you don't supply an `aliases` field in the body.
+aliases | Array | Array of alias names. | Yes if you don't supply an `alias` field in the body.
+filter | Object | A filter to use with the alias, so the alias points to a filtered part of the index. | No
+is_hidden | Boolean | Specifies whether the alias should be hidden from results that include wildcard expressions | No
+must_exist | Boolean | Specifies whether the alias to remove must exist. | No
+is_write_index | Boolean | Specifies whether the index should be a write index. An alias can only have one write index at a time. | No
+routing | String | Used to assign a custom value to a shard for specific operations. | No
+index_routing | String | Assigns a custom value to a shard only for index operations. | No
+search_routing | String | Assigns a custom value to a shard only for search operations. | No
+
+## Response
+
+```json
+{
+    "acknowledged": true
+}
+```

_opensearch/rest-api/create-index.md

@@ -37,6 +37,12 @@ PUT /sample-index1
 }
 ```
 
+## Path and HTTP methods
+
+```
+PUT <index-name>
+```
+
 ## Index naming restrictions
 
 OpenSearch indices have the following naming restrictions:

_search-plugins/sql/settings.md

@@ -21,7 +21,7 @@ PUT _cluster/settings
 }
 ```
 
-Similarly, you can also update the settings by sending the request to the plugin setting endpoing `_plugins/_query/setting`:
+Similarly, you can also update the settings by sending the request to the plugin setting endpoint `_plugins/_query/setting`:
 ```json
 PUT _plugins/_query/settings
 {
@@ -37,4 +37,4 @@ Setting | Default | Description
 `plugins.sql.slowlog` | 2 seconds | Configure the time limit (in seconds) for slow queries. The plugin logs slow queries as `Slow query: elapsed=xxx (ms)` in `opensearch.log`.
 `plugins.sql.cursor.keep_alive` | 1 minute | This value configures how long the cursor context is kept open. Cursor contexts are resource heavy, so we recommend a low value.
 `plugins.query.memory_limit` | 85% | This setting configures the heap memory usage limit for the circuit breaker of the query engine.
-`plugins.query.size_limit` | 200 | The setting sets the default size of index that the query engine fetches from OpenSearch.
+`plugins.query.size_limit` | 200 | The setting sets the default size of index that the query engine fetches from OpenSearch.

_security-plugin/configuration/disable.md

@@ -38,7 +38,7 @@ If you disable the security plugin in `opensearch.yml` (or delete the plugin ent
 
    ```
    FROM opensearchproject/opensearch-dashboards:{{site.opensearch_version}}
-   RUN /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove opensearchSecurityOpenSearch Dashboards
+   RUN /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove security-dashboards
    COPY --chown=opensearch-dashboards:opensearch-dashboards opensearch_dashboards.yml /usr/share/opensearch-dashboards/config/
    ```