Adding Honeycomb specific example for Agentcore observability (#1330)

* Adding Honeycomb specific example for Agentcore observability --------- Co-authored-by: andy-dufour <brigad3decuisine@gmail.com>
2026-05-22 14:43:35 +00:00 · 2026-04-16 15:59:54 -04:00
parent 9730eff02b
commit 8adf5c5ccc
4 changed files with 491 additions and 0 deletions
@@ -0,0 +1,8 @@
+boto3
+bedrock-agentcore
+bedrock-agentcore-starter-toolkit
+strands-agents[otel]
+strands-agents-tools
+opentelemetry-sdk
+opentelemetry-exporter-otlp-proto-http
+opentelemetry-instrumentation-bedrock
@@ -0,0 +1,481 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "5c0122e65c053f38",
+      "metadata": {},
+      "source": [
+        "# Strands Agent with Honeycomb Observability on Amazon Bedrock AgentCore Runtime\n",
+        "\n",
+        "## Overview\n",
+        "\n",
+        "This notebook demonstrates deploying a Strands agent to Amazon Bedrock AgentCore Runtime with Honeycomb integration. The implementation uses Amazon Bedrock Claude models and sends telemetry data to Honeycomb through OpenTelemetry (OTEL).\n",
+        "\n",
+        "## Key Components\n",
+        "\n",
+        "- **Strands Agents**: Python framework for building LLM-powered agents with built-in telemetry support\n",
+        "- **Amazon Bedrock AgentCore Runtime**: Managed runtime service for hosting and scaling agents on AWS\n",
+        "- **Honeycomb**: Observability platform for modern applications with trace views\n",
+        "- **OpenTelemetry**: Industry-standard protocol for collecting and exporting telemetry data\n",
+        "\n",
+        "## Architecture\n",
+        "\n",
+        "The agent is containerized and deployed to AgentCore Runtime, which provides HTTP endpoints for invocation. Telemetry data flows from the Strands agent through an OTLP exporter directly to Honeycomb's trace endpoint for monitoring and debugging. The implementation disables AgentCore's default ADOT observability to use Honeycomb instead.\n",
+        "\n",
+        "## Prerequisites\n",
+        "\n",
+        "- Python 3.10+\n",
+        "- AWS credentials configured with Bedrock and AgentCore permissions\n",
+        "- [Honeycomb](https://honeycomb.io/) account with API key\n",
+        "- Docker installed locally\n",
+        "- Access to Amazon Bedrock Claude models in your configured region"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "initial_id",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install --force-reinstall -U -r requirements.txt"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "def21rvknub",
+      "metadata": {},
+      "source": [
+        "## Configure Credentials\n",
+        "\n",
+        "Create a `.env` file in this directory with your API keys:\n",
+        "\n",
+        "```\n",
+        "AWS_DEFAULT_REGION=us-east-1\n",
+        "HONEYCOMB_API_KEY=your-honeycomb-api-key\n",
+        "HONEYCOMB_DATASET=llmobs\n",
+        "```\n",
+        "\n",
+        "You can obtain your Honeycomb API key from **Environment Settings → API Keys** in the [Honeycomb UI](https://ui.honeycomb.io/)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f20297fa",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%load_ext dotenv\n",
+        "%dotenv"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "161406db",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Safe verification (no secrets printed)\n",
+        "import boto3\n",
+        "try:\n",
+        "    resp = boto3.client('sts').get_caller_identity()\n",
+        "    print('AWS identity:', {k: resp[k] for k in ('Account','Arn','UserId') if k in resp})\n",
+        "except Exception as e:\n",
+        "    print('Credential check failed:', type(e).__name__, str(e))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "932110e6",
+      "metadata": {},
+      "source": [
+        "## Agent Implementation\n",
+        "\n",
+        "The agent file (`strands_claude.py`) implements a travel assistant with calculator and weather tools. Key configuration includes:\n",
+        "- **`DISABLE_ADOT_OBSERVABILITY=true`**: Disables AgentCore's built-in ADOT pipeline so we can set our own TracerProvider ([AgentCore observability docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html))\n",
+        "- **Direct OTLP export**: Configures an OpenTelemetry TracerProvider that exports traces to Honeycomb's OTLP HTTP endpoint (`https://api.honeycomb.io/v1/traces`)\n",
+        "- **Exporter headers / dataset**: Set the OTLP exporter headers to include your Honeycomb API key and dataset, e.g. `{'x-honeycomb-team': '<HONEYCOMB_API_KEY>', 'x-honeycomb-dataset': '<DATASET_NAME>'}`. Use the dataset name you want to view traces under in Honeycomb ([Honeycomb LLM docs](https://docs.honeycomb.io/send-data/llm/)).\n",
+        "- **`OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`**: Enables OpenTelemetry v1.37+ GenAI semantic conventions required by Strands Agents ([Honeycomb LLM docs](https://docs.honeycomb.io/send-data/llm/))\n",
+        "- **Automatic trace export**: All agent invocations, tool calls, and LLM interactions are automatically traced and sent to Honeycomb once the exporter is configured"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "3b845b32",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile strands_claude.py\n",
+        "import os\n",
+        "import logging\n",
+        "\n",
+        "logging.basicConfig(level=logging.ERROR, format=\"[%(levelname)s] %(message)s\")\n",
+        "logger = logging.getLogger(__name__)\n",
+        "logger.setLevel(os.getenv(\"AGENT_RUNTIME_LOG_LEVEL\", \"INFO\").upper())\n",
+        "\n",
+        "# =============================================================================\n",
+        "# Honeycomb - OpenTelemetry Configuration\n",
+        "# Must be configured BEFORE any other OpenTelemetry imports\n",
+        "# =============================================================================\n",
+        "\n",
+        "# Disable AgentCore's built-in ADOT so we can set our own TracerProvider\n",
+        "# See: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html\n",
+        "os.environ[\"DISABLE_ADOT_OBSERVABILITY\"] = \"true\"\n",
+        "\n",
+        "# Required for strands-agents GenAI semantic conventions (v1.37+)\n",
+        "os.environ[\"OTEL_SEMCONV_STABILITY_OPT_IN\"] = \"gen_ai_latest_experimental\"\n",
+        "\n",
+        "honeycomb_api_key = os.environ.get(\"HONEYCOMB_API_KEY\")\n",
+        "honeycomb_dataset = os.environ.get(\"HONEYCOMB_DATASET\", \"llmobs\")\n",
+        "honeycomb_endpoint = os.environ.get(\"HONEYCOMB_OTLP_ENDPOINT\", \"https://api.honeycomb.io/v1/traces\")\n",
+        "service_name = os.environ.get(\"OTEL_SERVICE_NAME\", \"agentcore-honeycomb-demo\")\n",
+        "\n",
+        "if honeycomb_api_key:\n",
+        "    from opentelemetry import trace\n",
+        "    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n",
+        "    from opentelemetry.sdk.trace import TracerProvider\n",
+        "    from opentelemetry.sdk.trace.export import SimpleSpanProcessor\n",
+        "    from opentelemetry.sdk.resources import Resource\n",
+        "    from opentelemetry.instrumentation.bedrock import BedrockInstrumentor\n",
+        "\n",
+        "    resource = Resource.create({\"service.name\": service_name})\n",
+        "    exporter = OTLPSpanExporter(\n",
+        "        endpoint=honeycomb_endpoint,\n",
+        "        headers={\"x-honeycomb-team\": honeycomb_api_key, \"x-honeycomb-dataset\": honeycomb_dataset},\n",
+        "    )\n",
+        "    provider = TracerProvider(resource=resource)\n",
+        "    provider.add_span_processor(SimpleSpanProcessor(exporter))\n",
+        "    trace.set_tracer_provider(provider)\n",
+        "    BedrockInstrumentor().instrument(capture_content=True)\n",
+        "    logger.info(\"Honeycomb & Bedrock instrumentation configured (service: %s)\", service_name)\n",
+        "else:\n",
+        "    logger.warning(\"HONEYCOMB_API_KEY not set. Traces will not be sent to Honeycomb.\")\n",
+        "\n",
+        "# =============================================================================\n",
+        "# Agent\n",
+        "# =============================================================================\n",
+        "\n",
+        "from bedrock_agentcore.runtime import BedrockAgentCoreApp\n",
+        "from strands import Agent, tool\n",
+        "from strands.models import BedrockModel\n",
+        "from strands_tools import calculator\n",
+        "\n",
+        "\n",
+        "def get_bedrock_model():\n",
+        "    region = os.getenv(\"AWS_DEFAULT_REGION\", \"us-east-1\")\n",
+        "    model_id = os.getenv(\"BEDROCK_MODEL_ID\", \"us.anthropic.claude-sonnet-4-5-20250929-v1:0\")\n",
+        "    return BedrockModel(\n",
+        "        model_id=model_id,\n",
+        "        region_name=region,\n",
+        "        max_tokens=1024,\n",
+        "    )\n",
+        "\n",
+        "\n",
+        "bedrock_model = get_bedrock_model()\n",
+        "\n",
+        "system_prompt = \"\"\"You are a helpful travel assistant. You can perform mathematical calculations \n",
+        "and check weather information. Always provide helpful, accurate responses and use tools when appropriate.\"\"\"\n",
+        "\n",
+        "\n",
+        "@tool\n",
+        "def weather():\n",
+        "    \"\"\"Get current weather.\"\"\"\n",
+        "    return \"sunny and 72F\"\n",
+        "\n",
+        "\n",
+        "app = BedrockAgentCoreApp()\n",
+        "\n",
+        "\n",
+        "def initialize_agent():\n",
+        "    \"\"\"Initialize the agent (telemetry is already configured at module level).\"\"\"\n",
+        "    return Agent(\n",
+        "        model=bedrock_model,\n",
+        "        system_prompt=system_prompt,\n",
+        "        tools=[calculator, weather],\n",
+        "    )\n",
+        "\n",
+        "\n",
+        "@app.entrypoint\n",
+        "def strands_agent_bedrock(payload, context=None):\n",
+        "    \"\"\"Invoke the agent with a payload.\"\"\"\n",
+        "    user_input = payload.get(\"prompt\", payload.get(\"text\", payload.get(\"message\", \"Hello\")))\n",
+        "    logger.info(\"[%s] User input: %s\", getattr(context, 'session_id', 'local'), user_input)\n",
+        "\n",
+        "    agent = initialize_agent()\n",
+        "    response = agent(user_input)\n",
+        "    return response.message['content'][0]['text']\n",
+        "\n",
+        "\n",
+        "if __name__ == \"__main__\":\n",
+        "    app.run()\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "680f3c05",
+      "metadata": {},
+      "source": [
+        "### Configure AgentCore Runtime deployment\n",
+        "\n",
+        "Next we will use our starter toolkit to configure the AgentCore Runtime deployment with an entrypoint, the execution role we just created and a requirements file. We will also configure the starter kit to auto create the Amazon ECR repository on launch.\n",
+        "\n",
+        "During the configure step, your docker file will be generated based on your application code. Please note that when using the `bedrock_agentcore_starter_toolkit` to configure your agent, it configures AgentCore Observability by default so, to use Honeycomb, you need to remove configuration for AgentCore Observability as explained below:\n",
+        "\n",
+        "<div style=\"text-align:left\">\n",
+        "    <img src=\"../images/configure.png\" width=\"40%\"/>\n",
+        "</div>"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "2e79eba2",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from bedrock_agentcore_starter_toolkit import Runtime\n",
+        "from boto3.session import Session\n",
+        "boto_session = Session()\n",
+        "region = boto_session.region_name\n",
+        "\n",
+        "agentcore_runtime = Runtime()\n",
+        "agent_name = \"strands_honeycomb_agent\"\n",
+        "response = agentcore_runtime.configure(\n",
+        "    entrypoint=\"strands_claude.py\",\n",
+        "    auto_create_execution_role=True,\n",
+        "    auto_create_ecr=True,\n",
+        "    requirements_file=\"requirements.txt\",\n",
+        "    region=region,\n",
+        "    agent_name=agent_name,\n",
+        "    memory_mode=\"NO_MEMORY\",\n",
+        "    disable_otel=True,\n",
+        ")\n",
+        "response"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "c4vnazx93s",
+      "metadata": {},
+      "source": [
+        "## Deploy to AgentCore Runtime\n",
+        "\n",
+        "Now that we've got a docker file, let's launch the agent to the AgentCore Runtime. This will create the Amazon ECR repository and the AgentCore Runtime.\n",
+        "\n",
+        "### Honeycomb Configuration\n",
+        "\n",
+        "To send traces to Honeycomb, you need:\n",
+        "- **Honeycomb API Key**: Get this from your Honeycomb account at Organization Settings → API Keys\n",
+        "\n",
+        "The agent code (`strands_claude.py`) automatically configures all OTLP settings when `HONEYCOMB_API_KEY` is provided:\n",
+        "- **Endpoint**: `https://api.honeycomb.io/v1/traces` (override with `HONEYCOMB_OTLP_ENDPOINT`)\n",
+        "- **Headers**: `x-honeycomb-team={key}, x-honeycomb-dataset={dataset}` — set `HONEYCOMB_DATASET` to the dataset where traces should appear (e.g. `llmobs`).\n",
+        "- **Semantic Conventions**: `gen_ai_latest_experimental` (for LLM Observability)\n",
+        "\n",
+        "You can override the OTLP endpoint or dataset using the `HONEYCOMB_OTLP_ENDPOINT` and `HONEYCOMB_DATASET` environment variables when launching the runtime.\n",
+        "\n",
+        "<div style=\"text-align:left\">\n",
+        "    <img src=\"../images/launch.png\" width=\"75%\"/>\n",
+        "</div>"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "17a32ab8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%load_ext dotenv\n",
+        "%dotenv\n",
+        "import os\n",
+        "# Honeycomb configuration\n",
+        "honeycomb_api_key = os.environ.get(\"HONEYCOMB_API_KEY\")  # Replace with your Honeycomb API key\n",
+        "\n",
+        "launch_result = agentcore_runtime.launch(\n",
+        "    env_vars={\n",
+        "        \"HONEYCOMB_API_KEY\": honeycomb_api_key,\n",
+        "        \"HONEYCOMB_DATASET\": \"llmobs\",  # Dataset to store traces in\n",
+        "        # Optionally override endpoint: \"HONEYCOMB_OTLP_ENDPOINT\": \"https://api.honeycomb.io/v1/traces\",\n",
+        "        \"OTEL_SERVICE_NAME\": \"agentcore-honeycomb-demo\",\n",
+        "        \"DISABLE_ADOT_OBSERVABILITY\": \"true\",  # Disable AgentCore's default observability\n",
+        "    },\n",
+        ")\n",
+        "launch_result"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a0ae9c09",
+      "metadata": {},
+      "source": [
+        "## Check Deployment Status\n",
+        "\n",
+        "Wait for the runtime to be ready before invoking:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "afa6ac09",
+      "metadata": {
+        "jupyter": {
+          "source_hidden": true
+        }
+      },
+      "outputs": [],
+      "source": [
+        "import time\n",
+        "status_response = agentcore_runtime.status()\n",
+        "status = status_response.endpoint['status']\n",
+        "end_status = ['READY', 'CREATE_FAILED', 'DELETE_FAILED', 'UPDATE_FAILED']\n",
+        "while status not in end_status:\n",
+        "    time.sleep(10)\n",
+        "    status_response = agentcore_runtime.status()\n",
+        "    status = status_response.endpoint['status']\n",
+        "    print(status)\n",
+        "status"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b7f89c56",
+      "metadata": {
+        "jupyter": {
+          "source_hidden": true
+        }
+      },
+      "source": [
+        "### Invoking AgentCore Runtime\n",
+        "\n",
+        "Finally, we can invoke our AgentCore Runtime with a payload\n",
+        "\n",
+        "<div style=\"text-align:left\">\n",
+        "    <img src=\"../images/invoke.png\" width=\"75%\"/>\n",
+        "</div>"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 46,
+      "id": "3d909e42",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "invoke_response = agentcore_runtime.invoke({\"prompt\": \"What is 25 + 17 and what's the weather like for a trip to Paris?\"})"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "dc34c4f3",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from IPython.display import Markdown, display\n",
+        "display(Markdown(\"\".join(invoke_response['response'])))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "05efe60e",
+      "metadata": {},
+      "source": [
+        "## View Traces in Honeycomb\n",
+        "\n",
+        "After invoking the agent, traces appear in Honeycomb within a few minutes:\n",
+        "\n",
+        "1. Go to [Honeycomb Observability](https://ui.honeycomb.io/)\n",
+        "2. Search for `ml_app:agentcore-Honeycomb-demo`\n",
+        "\n",
+        "The traces will include:\n",
+        "- Agent invocation details with full request/response context\n",
+        "- Tool calls (calculator, weather) with execution time\n",
+        "- Model interactions with latency and token usage\n",
+        "- Prompt and completion content\n",
+        "\n",
+        "<div style=\"text-align:left\">\n",
+        "    <img src=\"../images/honeycomb-agentcore-trace.png\" width=\"75%\"/>\n",
+        "</div>\n",
+        "\n",
+        "### Honeycomb Features\n",
+        "\n",
+        "Honeycomb provides purpose-built views for GenAI applications:\n",
+        "\n",
+        "- **Trace Explorer**: View end-to-end agent traces with prompt/response content, tool calls, and model interactions in a single timeline\n",
+        "- **Token Usage Tracking**: Monitor input and output token consumption across models to optimize costs and performance\n",
+        "- **Latency Analysis**: Track time-to-first-token and total response time for each model invocation\n",
+        "- **Error Monitoring**: Identify failed model calls, tool errors, and agent exceptions with full context\n",
+        "- **Evaluations**: Attach quality scores and custom evaluations to traces for continuous improvement\n",
+        "\n",
+        "For more information, see the [Honeycomb documentation](https://docs.honeycomb.io/send-data/llm/)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "7d3fdfe4",
+      "metadata": {},
+      "source": [
+        "## Cleanup (Optional)\n",
+        "\n",
+        "Clean up the deployed resources:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "76a6cf14",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!agentcore destroy --delete-ecr-repo --force --dry-run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b118ad38",
+      "metadata": {},
+      "source": [
+        "## Summary\n",
+        "\n",
+        "You have successfully deployed a Strands agent to Amazon Bedrock AgentCore Runtime with Honeycomb. The implementation demonstrates:\n",
+        "- Disabling AgentCore's built-in ADOT to use a custom observability provider\n",
+        "- Configuring an OpenTelemetry TracerProvider to export traces directly to Honeycomb\n",
+        "- Using `HONEYCOMB_DATASET` (or the `x-honeycomb-dataset` header) to route traces to a Honeycomb dataset.\n",
+        "- Enabling GenAI semantic conventions for LLM-specific trace views\n",
+        "- Invocation through the AgentCore starter toolkit SDK\n",
+        "\n",
+        "### Resources\n",
+        "\n",
+        "- [AgentCore Observability docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html)\n",
+        "- [Honeycomb For LLMs](https://docs.honeycomb.io/send-data/llm)\n",
+        "- [Strands Agents Observability](https://strandsagents.com/latest/documentation/docs/user-guide/observability-evaluation/observability/)\n",
+        "- [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "3.13.7",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.13.7"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}
@@ -13,6 +13,7 @@ The publish folder contains:
 - **Arize**: AI and Agent engineering platform
 - **Braintrust**: AI evaluation and monitoring platform
 - **Datadog**: Unified observability platform for monitoring, APM, logs, and traces
+- **Honeycomb**: Observability platform built for high-cardinality exploration
 - **Instana**: Real-Time APM and Observability Platform
 - **Langfuse**: LLM observability and analytics
 - **OpenLIT**: Open-source observability platform for LLM applications
@@ -3,6 +3,7 @@
 - aarora79
 - akshseh
 - anajmi
+- andy-dufour
 - amit-lulla
 - architec
 - arunskum