Fix Sinks and Sources in Data Prepper config (#2315)
* Fix Sinks and Sources in Data Prepper config Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/configuration/sources/http-source.md Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Caroline <[email protected]>
This commit is contained in:
Naarcha-AWS
Caroline
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: file sink
|
||||
parent: Sinks
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 45
|
||||
---
|
||||
|
||||
@@ -15,10 +16,10 @@ Option | Required | Type | Description
|
||||
:--- | :--- | :--- | :---
|
||||
path | Yes | String | Path for the output file (e.g. `logs/my-transformed-log.log`).
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: OpenSearch sink
|
||||
parent: Sinks
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 45
|
||||
---
|
||||
|
||||
@@ -34,10 +35,10 @@ ism_policy_file | No | String | The absolute file path for an ISM (Index State M
|
||||
number_of_shards | No | Integer | The number of primary shards that an index should have on the destination OpenSearch server. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/).
|
||||
number_of_replicas | No | Integer | The number of replica shards each primary shard should have on the destination OpenSearch server. For example, if you have 4 primary shards and set number_of_replicas to 3, the index has 12 replica shards. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/).
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: Pipeline sink
|
||||
parent: Sinks
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 45
|
||||
---
|
||||
|
||||
@@ -15,10 +16,10 @@ Option | Required | Type | Description
|
||||
:--- | :--- | :--- | :---
|
||||
name | Yes | String | Name of the pipeline to write to.
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: stdout sink
|
||||
parent: Sinks
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 45
|
||||
---
|
||||
|
||||
@@ -11,10 +12,10 @@ nav_order: 45
|
||||
|
||||
Sink for console output. Can be useful for testing. No options.
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,13 +2,11 @@
|
||||
layout: default
|
||||
title: http_source
|
||||
parent: Sources
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
|
||||
# Overview
|
||||
|
||||
## http_source
|
||||
# http_source
|
||||
|
||||
This is a source plugin that supports HTTP protocol. Currently ONLY support Json UTF-8 codec for incoming request, e.g. `[{"key1": "value1"}, {"key2": "value2"}]`.
|
||||
|
||||
@@ -31,10 +29,10 @@ acm_private_key_password | No | String | ACM private key password that decrypts
|
||||
acm_certificate_timeout_millis | No | Integer | Timeout in milliseconds for ACM to get certificates. Default is 120000.
|
||||
aws_region | Conditionally | String | AWS region to use ACM or S3. Required if `use_acm_certificate_for_ssl` is set to true or `ssl_certificate_file` and `ssl_key_file` is AWS S3 path.
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,13 +2,11 @@
|
||||
layout: default
|
||||
title: otel_metrics_source
|
||||
parent: Sources
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 10
|
||||
---
|
||||
|
||||
|
||||
# Overview
|
||||
|
||||
## otel_metrics_source
|
||||
# otel_metrics_source
|
||||
|
||||
Source for the OpenTelemetry Collector for collecting metric data.
|
||||
|
||||
@@ -29,10 +27,10 @@ acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN.
|
||||
awsRegion | Conditionally | String | Represents the AWS Region to use ACM or S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are AWS S3 paths.
|
||||
authentication | No | Object | An authentication configuration. By default, an unauthenticated server is created for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java).
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: otel_trace_source source
|
||||
parent: Sources
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 15
|
||||
---
|
||||
|
||||
@@ -30,10 +31,10 @@ acmCertificateArn | Conditionally | String | Represents the ACM certificate ARN.
|
||||
awsRegion | Conditionally | String | Represents the AWS region to use ACM or S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` are AWS S3 paths.
|
||||
authentication | No | Object | An authentication configuration. By default, an unauthenticated server is created for the pipeline. This parameter uses pluggable authentication for HTTPS. To use basic authentication, define the `http_basic` plugin with a `username` and `password`. To provide customer authentication, use or create a plugin that implements [GrpcAuthenticationProvider](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/armeria-common/src/main/java/com/amazon/dataprepper/armeria/authentication/GrpcAuthenticationProvider.java).
|
||||
|
||||
## Configuration
|
||||
<!--- ## Configuration
|
||||
|
||||
Content will be added to this section.
|
||||
|
||||
## Metrics
|
||||
|
||||
Content will be added to this section.
|
||||
Content will be added to this section. --->
|
||||
@@ -2,6 +2,7 @@
|
||||
layout: default
|
||||
title: s3
|
||||
parent: Sources
|
||||
grand_parent: Configuring Data Prepper
|
||||
nav_order: 20
|
||||
---
|
||||
|
||||
|
||||
@@ -8,4 +8,4 @@ nav_order: 42
|
||||
|
||||
# Sources
|
||||
|
||||
Sources define where your data comes from.
|
||||
Sources define where your data comes from within a Data Prepper pipeline.
|
||||
|
||||
Reference in New Issue
Block a user