Add Weather Agent use case with harness, gateway, guardrails, evaluation and observability (#1648)
* Add Weather Agent use case with harness, gateway, guardrails, evaluations, and observability * fix: resolve lint errors and security finding (unused imports, f-strings, bind to localhost) * fix: address PR feedback (README clarifications, region detection, graceful eval error, trace display) * feat: add Skills (xlsx report generation) and Optimization (system prompt recommendations) * feat: add Skills (xlsx report generation) and Optimization (system prompt recommendations) * fix: address PR feedback (README clarifications, region detection, graceful eval error, trace display)
This commit is contained in:
JobRamos
@@ -0,0 +1,11 @@
|
||||
venv/
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
resource_info.json
|
||||
optimization_result.json
|
||||
backend.log
|
||||
frontend.log
|
||||
backend.pid
|
||||
frontend.pid
|
||||
pr-response.txt
|
||||
__pycache__/
|
||||
@@ -0,0 +1,179 @@
|
||||
# Weather Agent — Harness + Evaluations + Gateway + Observability
|
||||
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
A full-stack weather agent web app that integrates **six AgentCore capabilities** in a single demo:
|
||||
|
||||
1. **AgentCore Gateway** — Creates a Gateway resource with an Exa MCP target, routing all tool calls through the managed proxy for centralized observability
|
||||
2. **Guardrails** — Bedrock guardrail that anonymizes PII (email, phone, address) in agent responses
|
||||
3. **Observability** — CloudWatch traces with full agent loop visibility
|
||||
4. **Skills** — Generate weather forecast Excel spreadsheets using the xlsx skill (fetched from Git at invocation time)
|
||||
5. **Evaluations** — Batch evaluation scoring with built-in evaluators (Helpfulness, Correctness, Coherence, etc.)
|
||||
6. **Optimization** — AI-generated system prompt recommendations based on agent traces
|
||||
|
||||
The web app features:
|
||||
- A **chat interface** where users ask weather questions
|
||||
- **Weather data cards** that update in real time (temperature, wind, UV, sunrise/sunset)
|
||||
- A **Traces panel** showing live trace IDs from CloudWatch (searchable in GenAI Observability)
|
||||
- A **Skills panel** to generate weather forecast XLSX reports
|
||||
- An **Evaluations panel** that triggers batch evaluations and displays scores
|
||||
- An **Optimization panel** that generates AI-improved system prompts from your traces
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
./start.sh
|
||||
```
|
||||
|
||||
One command: installs dependencies, provisions AWS resources (Gateway, Harness, Guardrail), starts the backend and frontend. Open **http://localhost:5173**.
|
||||
|
||||
To stop servers: `Ctrl+C`. To delete AWS resources: `./cleanup.sh`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React + Vite) — http://localhost:5173 │
|
||||
│ │
|
||||
│ ┌─────────────────────┐ ┌─────────────────────────────────┐ │
|
||||
│ │ Chat Panel │ │ Weather / Traces / Evaluations │ │
|
||||
│ │ (send queries) │ │ (live cards, trace IDs, scores)│ │
|
||||
│ └──────────┬───────────┘ └────────────────┬────────────────┘ │
|
||||
└─────────────┼─────────────────────────────────┼──────────────────┘
|
||||
│ │
|
||||
│ POST /api/chat (SSE) │ GET /api/traces
|
||||
│ │ POST /api/evaluate
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Backend (FastAPI) — http://localhost:8000 │
|
||||
│ │
|
||||
│ ┌────────────┐ ┌────────┐ ┌──────────────┐ ┌─────────────┐ │
|
||||
│ │ resources │ │ agent │ │observability │ │ evaluation │ │
|
||||
│ │ .py │ │ .py │ │ .py │ │ .py │ │
|
||||
│ └─────┬──────┘ └───┬────┘ └──────┬───────┘ └──────┬──────┘ │
|
||||
└────────┼──────────────┼──────────────┼─────────────────┼─────────┘
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ AWS (AgentCore + Bedrock + CloudWatch) │
|
||||
│ │
|
||||
│ AC Gateway ──► Exa MCP ──► Web Search (live weather data) │
|
||||
│ Harness ─────► Claude Haiku 4.5 (agent orchestration) │
|
||||
│ Guardrail ───► PII anonymization (email, phone, address) │
|
||||
│ Skills ──────► xlsx skill (Git-fetched, weather report gen) │
|
||||
│ CloudWatch ──► Trace observability (GenAI Observability) │
|
||||
│ Batch Eval ──► Built-in evaluators (Helpfulness, Correctness…) │
|
||||
│ Optimization ► System prompt recommendations from traces │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Web App Flow
|
||||
|
||||
1. **Start** — `./start.sh` provisions Gateway + Harness + Guardrail (or reuses existing ones)
|
||||
2. **Chat** — User asks weather questions; agent searches via Gateway's Exa MCP target
|
||||
3. **Weather Cards** — Parsed metrics (temperature, wind, UV, etc.) appear as visual cards
|
||||
4. **Traces** — Each invocation generates traces visible in the Traces tab and in CloudWatch > GenAI Observability > Bedrock AgentCore > Traces
|
||||
5. **Skills** — Click "Generate Report" to create an XLSX weather forecast using the xlsx skill
|
||||
6. **Evaluations** — Click "Run Eval" to trigger a batch evaluation; results show scores for Helpfulness, Correctness, Coherence, and more (also visible in Bedrock AgentCore > Evaluations > Batch evaluation)
|
||||
7. **Optimization** — Click "Optimize" to generate an AI-improved system prompt from your traces (also visible in Bedrock AgentCore > Optimizations > Recommendations)
|
||||
8. **Cleanup** — `./cleanup.sh` deletes all AWS resources including batch evaluations
|
||||
|
||||
|
||||
## Key Features
|
||||
|
||||
### AgentCore Gateway
|
||||
The demo creates an AgentCore Gateway resource (`create_gateway` + `create_gateway_target`) and passes it to the harness as `type: "agentcore_gateway"`. The Gateway acts as a managed proxy between the agent and external tool servers:
|
||||
- Centralized routing for MCP tool traffic
|
||||
- Automatic observability (every tool call through the Gateway is traced)
|
||||
- Configurable auth (NONE in this demo, supports IAM/OAuth)
|
||||
|
||||
### Bedrock Guardrails
|
||||
A guardrail anonymizes PII in agent responses. If you ask the agent to include personal info (email, phone), the guardrail masks it before the response reaches you.
|
||||
|
||||
### Observability
|
||||
Every `invoke_harness` call automatically generates traces in CloudWatch. The Traces tab shows trace IDs that you can search in:
|
||||
- **CloudWatch > GenAI Observability > Bedrock AgentCore > Traces**
|
||||
|
||||
### Skills (xlsx)
|
||||
The "Generate Report" button creates a 7-day weather forecast Excel spreadsheet using the AgentCore xlsx skill. The skill is fetched from Git (`https://github.com/anthropics/skills`) at invocation time — no container setup or pre-installation required. The report uses the last city you asked about.
|
||||
|
||||
### Batch Evaluations
|
||||
The "Run Eval" button triggers a batch evaluation that scores your session using built-in evaluators:
|
||||
- InstructionFollowing, Helpfulness, Correctness, Faithfulness, ResponseRelevance, Coherence, Conciseness, Refusal
|
||||
|
||||
Results appear in the web app and are also visible in:
|
||||
- **Bedrock AgentCore > Evaluations > Batch evaluation**
|
||||
|
||||
### Optimization
|
||||
The "Optimize" button analyzes your agent's traces and generates an AI-improved system prompt optimized for goal success. It uses the `start_recommendation` API with your harness traces as input. The recommended prompt and explanation are displayed in the web app.
|
||||
|
||||
Results are also visible in:
|
||||
- **Bedrock AgentCore > Optimizations > Recommendations**
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- Node.js 18+
|
||||
- AWS CLI configured with credentials (`aws sts get-caller-identity` should work). Recommended region: **us-east-1** (`export AWS_DEFAULT_REGION=us-east-1`)
|
||||
- Model access enabled for Claude Haiku 4.5 in Amazon Bedrock
|
||||
- [CloudWatch Transaction Search](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html) enabled — **required** for Traces, Evaluations, and Optimization to work. After enabling, wait 10-15 minutes before using these feature so the Bedrock AgentCore dashboard in CloudWatch can become available. Only traces from invocations *after* enabling will be indexed.
|
||||
|
||||
## AWS Permissions Required
|
||||
|
||||
> **Note:** The policies below use broad access for simplicity in this demo. In production environments, follow the principle of least privilege and create custom IAM policies scoped to only the specific resources and actions your agent needs.
|
||||
|
||||
| Policy | Purpose |
|
||||
|--------|---------|
|
||||
| `BedrockAgentCoreFullAccess` | Harness, Gateway, Batch Evaluations |
|
||||
| `AmazonBedrockFullAccess` | Model invocation, Guardrails |
|
||||
| `IAMFullAccess` | Create the harness execution role (first run only) |
|
||||
| `CloudWatchFullAccessV2` | Query traces + batch evaluation output logs |
|
||||
|
||||
## Running
|
||||
|
||||
### Web App (recommended)
|
||||
|
||||
```bash
|
||||
./start.sh
|
||||
```
|
||||
|
||||
One command: creates a virtual environment, installs Python and Node.js packages, provisions AWS resources, starts the FastAPI backend and React frontend.
|
||||
|
||||
Open **http://localhost:5173** once the script prints "App is running!".
|
||||
|
||||
```bash
|
||||
# Stop servers without deleting AWS resources:
|
||||
# Press Ctrl+C (resources persist for next ./start.sh)
|
||||
|
||||
# Stop servers AND delete all AWS resources:
|
||||
./cleanup.sh
|
||||
```
|
||||
|
||||
## Sample Prompts
|
||||
|
||||
- "What's the weather in Tokyo?"
|
||||
- "What's the wind speed in Vancouver right now?"
|
||||
- "What's the UV index in Miami today?"
|
||||
- "When is sunrise and sunset in London?"
|
||||
|
||||
<!-- ### CLI-only mode
|
||||
|
||||
For a headless demo that runs in the terminal and cleans up after itself. Run this separately — not while the web app is running, since both use the same AWS resources.
|
||||
|
||||
```bash
|
||||
./run.sh
|
||||
``` -->
|
||||
|
||||
## Clean Up
|
||||
|
||||
```bash
|
||||
# Delete all AWS resources (gateway, harness, guardrail, batch evaluations, IAM role):
|
||||
./cleanup.sh
|
||||
|
||||
# To also remove the virtual environment and node_modules:
|
||||
rm -rf venv frontend/node_modules
|
||||
```
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Agent invocation — streaming wrapper around invoke_harness."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Generator
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
|
||||
from utils.client import get_agentcore_client
|
||||
|
||||
MODEL_ID = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
|
||||
SYSTEM_PROMPT = (
|
||||
"You are a weather assistant. You ONLY answer questions about weather, "
|
||||
"climate, and atmospheric conditions (temperature, wind, humidity, UV index, "
|
||||
"sunrise, sunset, moon phase, forecasts, air quality, precipitation). "
|
||||
"If the user asks about anything unrelated to weather, politely redirect them. "
|
||||
"For example: 'I'm a weather assistant — I can help with forecasts, current conditions, "
|
||||
"UV index, wind, sunrise/sunset, and more. What location would you like weather for?'\n\n"
|
||||
"When answering weather questions:\n"
|
||||
"- Always search for real-time data using your tools\n"
|
||||
"- Include specific numbers with units (temperature in °F/°C, wind in km/h or mph)\n"
|
||||
"- Mention the city name in your response\n"
|
||||
"- Keep responses concise and well-structured"
|
||||
)
|
||||
|
||||
|
||||
def invoke_agent(harness_arn: str, gateway_arn: str, session_id: str, message: str) -> Generator[dict, None, None]:
|
||||
"""Stream agent response as SSE-friendly dicts."""
|
||||
client = get_agentcore_client()
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "agentcore_gateway",
|
||||
"name": "gateway",
|
||||
"config": {"agentCoreGateway": {"gatewayArn": gateway_arn}},
|
||||
}
|
||||
]
|
||||
|
||||
prefixed_message = f"[INSTRUCTIONS: {SYSTEM_PROMPT}]\n\nUser question: {message}"
|
||||
|
||||
response = client.invoke_harness(
|
||||
harnessArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
messages=[{"role": "user", "content": [{"text": prefixed_message}]}],
|
||||
model={"bedrockModelConfig": {"modelId": MODEL_ID}},
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
for event in response["stream"]:
|
||||
if "contentBlockStart" in event:
|
||||
start = event["contentBlockStart"].get("start", {})
|
||||
if "toolUse" in start:
|
||||
yield {"type": "tool", "name": start["toolUse"].get("name", "?")}
|
||||
elif "contentBlockDelta" in event:
|
||||
delta = event["contentBlockDelta"].get("delta", {})
|
||||
if "text" in delta:
|
||||
yield {"type": "text", "content": delta["text"]}
|
||||
elif "messageStop" in event:
|
||||
yield {"type": "done"}
|
||||
elif "internalServerException" in event:
|
||||
yield {"type": "error", "content": str(event["internalServerException"])}
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Evaluations — run batch evaluation against harness session traces."""
|
||||
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import boto3
|
||||
|
||||
from resources import REGION
|
||||
|
||||
EVALUATOR_IDS = [
|
||||
"Builtin.InstructionFollowing",
|
||||
"Builtin.Helpfulness",
|
||||
"Builtin.Correctness",
|
||||
"Builtin.Faithfulness",
|
||||
"Builtin.ResponseRelevance",
|
||||
"Builtin.Coherence",
|
||||
"Builtin.Conciseness",
|
||||
"Builtin.Refusal",
|
||||
]
|
||||
|
||||
|
||||
def _discover_log_group(harness_name: str) -> str | None:
|
||||
"""Find the CloudWatch log group for a harness by prefix search."""
|
||||
logs = boto3.client("logs", region_name=REGION)
|
||||
prefix = f"/aws/bedrock-agentcore/runtimes/harness_{harness_name}-"
|
||||
resp = logs.describe_log_groups(logGroupNamePrefix=prefix, limit=5)
|
||||
groups = resp.get("logGroups", [])
|
||||
if groups:
|
||||
# Return the most recently created one
|
||||
groups.sort(key=lambda g: g.get("creationTime", 0), reverse=True)
|
||||
return groups[0]["logGroupName"]
|
||||
return None
|
||||
|
||||
|
||||
def run_batch_evaluation(harness_id: str, harness_name: str = None) -> dict:
|
||||
"""Start a batch evaluation job and poll until complete. Returns results."""
|
||||
client = boto3.client("bedrock-agentcore", region_name=REGION)
|
||||
|
||||
if not hasattr(client, "start_batch_evaluation"):
|
||||
return {
|
||||
"error": (
|
||||
"start_batch_evaluation is not available in your boto3 version. "
|
||||
"Please upgrade: pip install 'boto3>=1.43.27'"
|
||||
),
|
||||
"scores": [],
|
||||
}
|
||||
|
||||
# Discover the log group dynamically (it has a random suffix)
|
||||
log_group = None
|
||||
if harness_name:
|
||||
log_group = _discover_log_group(harness_name)
|
||||
if not log_group:
|
||||
# Fallback: try with harness_id directly
|
||||
log_group = _discover_log_group(harness_id)
|
||||
if not log_group:
|
||||
return {"error": f"Could not find log group for harness {harness_name or harness_id}", "scores": []}
|
||||
|
||||
# Service name format: harness_{name}.DEFAULT (without the random suffix)
|
||||
# Log group: harness_WeatherAgent_537bb0c9-d9RslKDml1-DEFAULT
|
||||
# Service: harness_WeatherAgent_537bb0c9.DEFAULT
|
||||
log_group_basename = log_group.split("/")[-1] # harness_WeatherAgent_537bb0c9-d9RslKDml1-DEFAULT
|
||||
parts = log_group_basename.rsplit("-", 2) # ['harness_WeatherAgent_537bb0c9', 'd9RslKDml1', 'DEFAULT']
|
||||
service_name = f"{parts[0]}.DEFAULT" if len(parts) >= 3 else log_group_basename.replace("-DEFAULT", ".DEFAULT")
|
||||
|
||||
batch_name = f"weather_eval_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
try:
|
||||
resp = client.start_batch_evaluation(
|
||||
batchEvaluationName=batch_name,
|
||||
evaluators=[{"evaluatorId": eid} for eid in EVALUATOR_IDS],
|
||||
dataSourceConfig={
|
||||
"cloudWatchLogs": {
|
||||
"serviceNames": [service_name],
|
||||
"logGroupNames": [log_group],
|
||||
}
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
return {"error": str(e), "scores": []}
|
||||
|
||||
batch_id = resp["batchEvaluationId"]
|
||||
|
||||
# Poll until complete (timeout after 5 minutes)
|
||||
deadline = time.monotonic() + 300
|
||||
status = "PENDING"
|
||||
while time.monotonic() < deadline:
|
||||
time.sleep(10)
|
||||
try:
|
||||
result = client.get_batch_evaluation(batchEvaluationId=batch_id)
|
||||
status = result.get("status", "UNKNOWN")
|
||||
if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED"):
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if status not in ("COMPLETED", "COMPLETED_WITH_ERRORS"):
|
||||
# Try to get failure details
|
||||
failure_reason = ""
|
||||
try:
|
||||
failure_reason = result.get("failureReasons", result.get("statusReason", ""))
|
||||
if not failure_reason:
|
||||
# Check evaluationResults for per-session errors
|
||||
eval_res = result.get("evaluationResults", {})
|
||||
failed_count = eval_res.get("numberOfSessionsFailed", 0)
|
||||
completed_count = eval_res.get("numberOfSessionsCompleted", 0)
|
||||
if failed_count > 0:
|
||||
failure_reason = f"{failed_count} session(s) failed, {completed_count} completed"
|
||||
except Exception:
|
||||
pass
|
||||
error_msg = f"Evaluation did not complete (status: {status})"
|
||||
if failure_reason:
|
||||
error_msg += f". {failure_reason}"
|
||||
print(f"[eval] Failed: {error_msg}")
|
||||
print(f"[eval] Full response: {result}")
|
||||
return {
|
||||
"batch_id": batch_id,
|
||||
"batch_name": batch_name,
|
||||
"status": status,
|
||||
"error": error_msg,
|
||||
"scores": [],
|
||||
}
|
||||
|
||||
# Extract evaluator results
|
||||
scores = []
|
||||
eval_results = result.get("evaluationResults", {})
|
||||
summaries = eval_results.get("evaluatorSummaries", [])
|
||||
|
||||
for summary in summaries:
|
||||
eid = summary.get("evaluatorId", "")
|
||||
stats = summary.get("statistics", {})
|
||||
avg_score = stats.get("averageScore")
|
||||
evaluated = summary.get("totalEvaluated", 0)
|
||||
|
||||
name = eid.replace("Builtin.", "") if eid.startswith("Builtin.") else eid
|
||||
scores.append({
|
||||
"evaluator": name,
|
||||
"score": avg_score,
|
||||
"evaluated_sessions": evaluated,
|
||||
})
|
||||
|
||||
return {
|
||||
"batch_id": batch_id,
|
||||
"batch_name": batch_name,
|
||||
"status": status,
|
||||
"total_sessions": eval_results.get("numberOfSessionsCompleted", 0),
|
||||
"scores": scores,
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
"""FastAPI backend for the Weather Agent web app."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel
|
||||
from sse_starlette.sse import EventSourceResponse
|
||||
|
||||
from resources import ensure_resources
|
||||
from agent import invoke_agent
|
||||
from observability import get_recent_traces, get_transaction_search_status
|
||||
from evaluation import run_batch_evaluation
|
||||
from skills import generate_weather_report
|
||||
from optimization import run_optimization
|
||||
|
||||
# Global state
|
||||
_state: dict = {}
|
||||
_sessions: dict[str, list] = {}
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
global _state
|
||||
print("[backend] Starting — provisioning AWS resources...")
|
||||
_state = await asyncio.to_thread(ensure_resources)
|
||||
print(f"[backend] Ready. Harness: {_state['harness_id']}")
|
||||
yield
|
||||
print("[backend] Shutting down")
|
||||
|
||||
|
||||
app = FastAPI(title="Weather Agent", lifespan=lifespan)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["http://localhost:5173", "http://localhost:3000"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
|
||||
class ChatRequest(BaseModel):
|
||||
message: str
|
||||
session_id: str | None = None
|
||||
|
||||
|
||||
class EvalRequest(BaseModel):
|
||||
session_id: str
|
||||
|
||||
|
||||
class ReportRequest(BaseModel):
|
||||
session_id: str
|
||||
city: str | None = None
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok", "harness_id": _state.get("harness_id")}
|
||||
|
||||
|
||||
@app.get("/api/status")
|
||||
async def status():
|
||||
return {
|
||||
"ready": bool(_state.get("harness_id")),
|
||||
"harness_id": _state.get("harness_id"),
|
||||
"harness_name": _state.get("harness_name"),
|
||||
"gateway_id": _state.get("gateway_id"),
|
||||
"gateway_name": _state.get("gateway_name"),
|
||||
"guardrail_id": _state.get("guardrail_id"),
|
||||
"guardrail_name": _state.get("guardrail_name"),
|
||||
"region": _state.get("region"),
|
||||
}
|
||||
|
||||
|
||||
@app.post("/api/chat")
|
||||
async def chat(req: ChatRequest):
|
||||
if not _state.get("harness_arn"):
|
||||
raise HTTPException(503, "Resources not ready")
|
||||
|
||||
session_id = req.session_id or str(uuid.uuid4()).upper()
|
||||
|
||||
if session_id not in _sessions:
|
||||
_sessions[session_id] = []
|
||||
|
||||
_sessions[session_id].append({"role": "user", "content": req.message})
|
||||
|
||||
async def generate():
|
||||
full_text = ""
|
||||
yield json.dumps({"type": "session_id", "session_id": session_id})
|
||||
|
||||
for event in invoke_agent(
|
||||
_state["harness_arn"], _state["gateway_arn"], session_id, req.message
|
||||
):
|
||||
if event["type"] == "text":
|
||||
full_text += event["content"]
|
||||
yield json.dumps(event)
|
||||
|
||||
_sessions[session_id].append({"role": "assistant", "content": full_text})
|
||||
|
||||
return EventSourceResponse(generate(), media_type="text/event-stream")
|
||||
|
||||
|
||||
@app.get("/api/traces")
|
||||
async def traces(minutes: int = 10):
|
||||
result = await asyncio.to_thread(get_recent_traces, _state.get("harness_name"), minutes)
|
||||
tx_status = await asyncio.to_thread(get_transaction_search_status)
|
||||
return {"traces": result, "transaction_search": tx_status}
|
||||
|
||||
|
||||
@app.post("/api/evaluate")
|
||||
async def evaluate(req: EvalRequest):
|
||||
if not _state.get("harness_id"):
|
||||
raise HTTPException(503, "Resources not ready")
|
||||
|
||||
result = await asyncio.to_thread(
|
||||
run_batch_evaluation,
|
||||
_state["harness_id"],
|
||||
_state.get("harness_name"),
|
||||
)
|
||||
return {"session_id": req.session_id, **result}
|
||||
|
||||
|
||||
@app.post("/api/generate-report")
|
||||
async def generate_report(req: ReportRequest):
|
||||
if not _state.get("harness_arn"):
|
||||
raise HTTPException(503, "Resources not ready")
|
||||
|
||||
result = await asyncio.to_thread(
|
||||
generate_weather_report,
|
||||
_state["harness_arn"],
|
||||
_state["harness_id"],
|
||||
req.session_id,
|
||||
req.city or "the cities discussed",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/api/optimize")
|
||||
async def optimize():
|
||||
if not _state.get("harness_name"):
|
||||
raise HTTPException(503, "Resources not ready")
|
||||
|
||||
result = await asyncio.to_thread(
|
||||
run_optimization, _state["harness_name"]
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@app.get("/api/sessions")
|
||||
async def sessions():
|
||||
return {
|
||||
sid: {"turns": len(msgs), "last_message": msgs[-1]["content"][:80] if msgs else ""}
|
||||
for sid, msgs in _sessions.items()
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="127.0.0.1", port=8000)
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Observability — query traces from the aws/spans CloudWatch log group."""
|
||||
|
||||
import time
|
||||
|
||||
import boto3
|
||||
|
||||
from resources import REGION
|
||||
|
||||
SPANS_LOG_GROUP = "aws/spans"
|
||||
|
||||
|
||||
def get_recent_traces(harness_name: str = None, minutes: int = 10) -> list[dict]:
|
||||
"""Query aws/spans log group for recent traces from this harness."""
|
||||
logs = boto3.client("logs", region_name=REGION)
|
||||
|
||||
end_time = int(time.time())
|
||||
start_time = end_time - (minutes * 60)
|
||||
|
||||
query = """fields traceId, @timestamp
|
||||
| filter ispresent(traceId) and traceId != ''
|
||||
| stats count() as spans by traceId
|
||||
| sort @timestamp desc
|
||||
| limit 20"""
|
||||
|
||||
try:
|
||||
resp = logs.start_query(
|
||||
logGroupName=SPANS_LOG_GROUP,
|
||||
startTime=start_time,
|
||||
endTime=end_time,
|
||||
queryString=query,
|
||||
)
|
||||
query_id = resp["queryId"]
|
||||
|
||||
for _ in range(15):
|
||||
time.sleep(2)
|
||||
result = logs.get_query_results(queryId=query_id)
|
||||
if result["status"] in ("Complete", "Failed", "Cancelled"):
|
||||
break
|
||||
|
||||
if result["status"] != "Complete":
|
||||
return []
|
||||
|
||||
traces = []
|
||||
for row in result.get("results", []):
|
||||
fields = {f["field"]: f["value"] for f in row}
|
||||
trace_id = fields.get("traceId", "")
|
||||
spans = fields.get("spans", "0")
|
||||
|
||||
if trace_id:
|
||||
traces.append({
|
||||
"trace_id": trace_id,
|
||||
"spans": int(spans),
|
||||
"has_error": False,
|
||||
"has_fault": False,
|
||||
})
|
||||
|
||||
return traces
|
||||
|
||||
except Exception as e:
|
||||
return [{"error": str(e)}]
|
||||
|
||||
|
||||
def get_transaction_search_status() -> dict:
|
||||
"""Check if Transaction Search is enabled."""
|
||||
xray = boto3.client("xray", region_name=REGION)
|
||||
try:
|
||||
rules = xray.get_indexing_rules()
|
||||
sampling = rules["IndexingRules"][0]["Rule"]["Probabilistic"]["DesiredSamplingPercentage"]
|
||||
return {"enabled": True, "sampling_percentage": sampling}
|
||||
except Exception as e:
|
||||
return {"enabled": False, "error": str(e)}
|
||||
@@ -0,0 +1,116 @@
|
||||
"""Optimization — generate system prompt recommendations from agent traces."""
|
||||
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import boto3
|
||||
|
||||
from resources import REGION
|
||||
from agent import SYSTEM_PROMPT
|
||||
|
||||
|
||||
def _discover_log_group_arn(harness_name: str) -> tuple[str, str] | None:
|
||||
"""Find the log group ARN and service name for a harness."""
|
||||
logs = boto3.client("logs", region_name=REGION)
|
||||
prefix = f"/aws/bedrock-agentcore/runtimes/harness_{harness_name}-"
|
||||
resp = logs.describe_log_groups(logGroupNamePrefix=prefix, limit=5)
|
||||
groups = resp.get("logGroups", [])
|
||||
if not groups:
|
||||
return None
|
||||
groups.sort(key=lambda g: g.get("creationTime", 0), reverse=True)
|
||||
log_group = groups[0]
|
||||
log_group_arn = log_group["arn"]
|
||||
log_group_name = log_group["logGroupName"]
|
||||
|
||||
basename = log_group_name.split("/")[-1]
|
||||
parts = basename.rsplit("-", 2)
|
||||
service_name = f"{parts[0]}.DEFAULT" if len(parts) >= 3 else basename.replace("-DEFAULT", ".DEFAULT")
|
||||
|
||||
return log_group_arn, service_name
|
||||
|
||||
|
||||
def run_optimization(harness_name: str, evaluator: str = "Builtin.GoalSuccessRate") -> dict:
|
||||
"""Run a system prompt recommendation and return the result."""
|
||||
client = boto3.client("bedrock-agentcore", region_name=REGION)
|
||||
|
||||
# Discover log group
|
||||
result = _discover_log_group_arn(harness_name)
|
||||
if not result:
|
||||
return {"error": "Could not find log group. Send some chat messages first.", "status": "FAILED"}
|
||||
|
||||
log_group_arn, service_name = result
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
start_time = now - timedelta(days=7)
|
||||
rec_name = f"weather_rec_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
# Start recommendation
|
||||
try:
|
||||
resp = client.start_recommendation(
|
||||
name=rec_name,
|
||||
type="SYSTEM_PROMPT_RECOMMENDATION",
|
||||
recommendationConfig={
|
||||
"systemPromptRecommendationConfig": {
|
||||
"systemPrompt": {"text": SYSTEM_PROMPT},
|
||||
"agentTraces": {
|
||||
"cloudwatchLogs": {
|
||||
"logGroupArns": [log_group_arn],
|
||||
"serviceNames": [service_name],
|
||||
"startTime": start_time,
|
||||
"endTime": now,
|
||||
}
|
||||
},
|
||||
"evaluationConfig": {
|
||||
"evaluators": [
|
||||
{"evaluatorArn": f"arn:aws:bedrock-agentcore:::evaluator/{evaluator}"}
|
||||
]
|
||||
},
|
||||
}
|
||||
},
|
||||
clientToken=str(uuid.uuid4()),
|
||||
)
|
||||
except Exception as e:
|
||||
return {"error": str(e), "status": "FAILED"}
|
||||
|
||||
rec_id = resp["recommendationId"]
|
||||
|
||||
# Poll for completion (timeout 5 minutes)
|
||||
status = "PENDING"
|
||||
for _ in range(30):
|
||||
time.sleep(10)
|
||||
try:
|
||||
rec = client.get_recommendation(recommendationId=rec_id)
|
||||
status = rec.get("status", "UNKNOWN")
|
||||
if status in ("COMPLETED", "FAILED"):
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if status != "COMPLETED":
|
||||
error_msg = ""
|
||||
if status == "FAILED":
|
||||
rec_result = rec.get("recommendationResult", {}).get(
|
||||
"systemPromptRecommendationResult", {}
|
||||
)
|
||||
error_msg = rec_result.get("errorMessage", "Unknown error")
|
||||
return {
|
||||
"status": status,
|
||||
"error": error_msg or f"Recommendation did not complete (status: {status})",
|
||||
"recommendation_name": rec_name,
|
||||
}
|
||||
|
||||
# Extract result
|
||||
rec_result = rec.get("recommendationResult", {}).get(
|
||||
"systemPromptRecommendationResult", {}
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "COMPLETED",
|
||||
"recommendation_name": rec_name,
|
||||
"recommendation_id": rec_id,
|
||||
"evaluator": evaluator,
|
||||
"current_prompt": SYSTEM_PROMPT,
|
||||
"recommended_prompt": rec_result.get("recommendedSystemPrompt", ""),
|
||||
"explanation": rec_result.get("explanation", ""),
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
boto3>=1.43.27
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
sse-starlette
|
||||
pydantic
|
||||
bedrock-agentcore
|
||||
requests
|
||||
@@ -0,0 +1,247 @@
|
||||
"""AWS resource lifecycle — create or reuse Gateway, Harness, Guardrail."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
|
||||
from utils.iam import create_harness_role, delete_harness_role
|
||||
from utils.client import get_agentcore_control_client
|
||||
|
||||
STATE_FILE = Path(__file__).parent.parent / "resource_info.json"
|
||||
REGION = os.environ.get("AWS_DEFAULT_REGION") or boto3.session.Session().region_name or "us-east-1"
|
||||
|
||||
|
||||
def _poll(get_fn, extract_fn, target="READY", timeout=300, interval=5):
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
resp = get_fn()
|
||||
status = extract_fn(resp)
|
||||
if status == target:
|
||||
return resp
|
||||
if status in ("FAILED", "CREATE_FAILED", "DELETE_FAILED"):
|
||||
raise RuntimeError(f"Resource failed: {status}")
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError(f"Not {target} after {timeout}s")
|
||||
time.sleep(interval)
|
||||
|
||||
|
||||
def _load_state() -> dict | None:
|
||||
if STATE_FILE.exists():
|
||||
return json.loads(STATE_FILE.read_text())
|
||||
return None
|
||||
|
||||
|
||||
def _save_state(state: dict):
|
||||
STATE_FILE.write_text(json.dumps(state, indent=2))
|
||||
|
||||
|
||||
def _resources_alive(state: dict) -> bool:
|
||||
try:
|
||||
control = get_agentcore_control_client()
|
||||
h = control.get_harness(harnessId=state["harness_id"])
|
||||
if h["harness"]["status"] != "READY":
|
||||
return False
|
||||
gw = boto3.client("bedrock-agentcore-control", region_name=REGION)
|
||||
g = gw.get_gateway(gatewayIdentifier=state["gateway_id"])
|
||||
if g["status"] != "READY":
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def ensure_resources() -> dict:
|
||||
"""Create or reuse all AWS resources. Returns resource dict."""
|
||||
existing = _load_state()
|
||||
if existing and _resources_alive(existing):
|
||||
print("[resources] Reusing existing resources")
|
||||
return existing
|
||||
|
||||
print("[resources] Provisioning new resources...")
|
||||
control = get_agentcore_control_client()
|
||||
gw_control = boto3.client("bedrock-agentcore-control", region_name=REGION)
|
||||
bedrock = boto3.client("bedrock", region_name=REGION)
|
||||
|
||||
# IAM role
|
||||
role_arn = create_harness_role()
|
||||
time.sleep(10)
|
||||
|
||||
# Gateway
|
||||
gateway_name = f"WeatherGW-{uuid.uuid4().hex[:8]}"
|
||||
resp = gw_control.create_gateway(
|
||||
name=gateway_name, roleArn=role_arn, protocolType="MCP", authorizerType="NONE"
|
||||
)
|
||||
gateway_id = resp["gatewayId"]
|
||||
gateway_arn = resp["gatewayArn"]
|
||||
_poll(
|
||||
lambda: gw_control.get_gateway(gatewayIdentifier=gateway_id),
|
||||
lambda r: r["status"],
|
||||
)
|
||||
|
||||
# MCP target (Exa search)
|
||||
resp = gw_control.create_gateway_target(
|
||||
gatewayIdentifier=gateway_id,
|
||||
name="exa-weather",
|
||||
targetConfiguration={"mcp": {"mcpServer": {"endpoint": "https://mcp.exa.ai/mcp"}}},
|
||||
)
|
||||
target_id = resp["targetId"]
|
||||
_poll(
|
||||
lambda: gw_control.get_gateway_target(gatewayIdentifier=gateway_id, targetId=target_id),
|
||||
lambda r: r["status"],
|
||||
)
|
||||
|
||||
# Harness
|
||||
harness_name = f"WeatherAgent_{uuid.uuid4().hex[:8]}"
|
||||
resp = control.create_harness(
|
||||
harnessName=harness_name,
|
||||
executionRoleArn=role_arn,
|
||||
systemPrompt=[
|
||||
{
|
||||
"text": (
|
||||
"You are a weather assistant. You ONLY answer questions about weather, "
|
||||
"climate, and atmospheric conditions (temperature, wind, humidity, UV index, "
|
||||
"sunrise, sunset, moon phase, forecasts, air quality, precipitation). "
|
||||
"If the user asks about anything unrelated to weather, politely redirect them. "
|
||||
"For example: 'I'm a weather assistant — I can help with forecasts, current conditions, "
|
||||
"UV index, wind, sunrise/sunset, and more. What location would you like weather for?' "
|
||||
"When answering weather questions: always search for real-time data using your tools, "
|
||||
"include specific numbers with units (temperature in F/C, wind in km/h or mph), "
|
||||
"mention the city name in your response, and keep responses concise and well-structured."
|
||||
)
|
||||
}
|
||||
],
|
||||
)
|
||||
harness_id = resp["harness"]["harnessId"]
|
||||
harness_arn = resp["harness"]["arn"]
|
||||
_poll(
|
||||
lambda: control.get_harness(harnessId=harness_id),
|
||||
lambda r: r["harness"]["status"],
|
||||
)
|
||||
|
||||
# Guardrail
|
||||
guardrail_id = None
|
||||
guardrail_version = None
|
||||
guardrail_name = None
|
||||
try:
|
||||
guardrail_name = f"weather-pii-{uuid.uuid4().hex[:6]}"
|
||||
gr = bedrock.create_guardrail(
|
||||
name=guardrail_name,
|
||||
description="Anonymize PII in weather agent responses",
|
||||
sensitiveInformationPolicyConfig={
|
||||
"piiEntitiesConfig": [
|
||||
{"type": "EMAIL", "action": "ANONYMIZE"},
|
||||
{"type": "PHONE", "action": "ANONYMIZE"},
|
||||
{"type": "ADDRESS", "action": "ANONYMIZE"},
|
||||
{"type": "US_SOCIAL_SECURITY_NUMBER", "action": "ANONYMIZE"},
|
||||
]
|
||||
},
|
||||
blockedInputMessaging="Content blocked.",
|
||||
blockedOutputsMessaging="Content blocked.",
|
||||
)
|
||||
guardrail_id = gr["guardrailId"]
|
||||
gv = bedrock.create_guardrail_version(guardrailIdentifier=guardrail_id, description="v1")
|
||||
guardrail_version = gv["version"]
|
||||
except Exception as e:
|
||||
print(f"[resources] Guardrail creation failed (non-critical): {e}")
|
||||
|
||||
state = {
|
||||
"gateway_id": gateway_id,
|
||||
"gateway_arn": gateway_arn,
|
||||
"gateway_name": gateway_name,
|
||||
"target_id": target_id,
|
||||
"harness_id": harness_id,
|
||||
"harness_arn": harness_arn,
|
||||
"harness_name": harness_name,
|
||||
"guardrail_id": guardrail_id,
|
||||
"guardrail_name": guardrail_name,
|
||||
"guardrail_version": guardrail_version,
|
||||
"role_arn": role_arn,
|
||||
"region": REGION,
|
||||
}
|
||||
_save_state(state)
|
||||
print("[resources] All resources ready")
|
||||
return state
|
||||
|
||||
|
||||
def destroy_resources():
|
||||
"""Delete all resources and remove state file."""
|
||||
state = _load_state()
|
||||
if not state:
|
||||
print("[resources] No state file found")
|
||||
return
|
||||
|
||||
control = get_agentcore_control_client()
|
||||
gw_control = boto3.client("bedrock-agentcore-control", region_name=REGION)
|
||||
bedrock = boto3.client("bedrock", region_name=REGION)
|
||||
|
||||
if state.get("harness_id"):
|
||||
try:
|
||||
control.delete_harness(harnessId=state["harness_id"])
|
||||
print(f" Deleted harness: {state['harness_id']}")
|
||||
except Exception as e:
|
||||
print(f" Warning: {e}")
|
||||
|
||||
if state.get("gateway_id") and state.get("target_id"):
|
||||
try:
|
||||
gw_control.delete_gateway_target(
|
||||
gatewayIdentifier=state["gateway_id"], targetId=state["target_id"]
|
||||
)
|
||||
print(f" Deleted target: {state['target_id']}")
|
||||
time.sleep(10)
|
||||
except Exception as e:
|
||||
print(f" Warning: {e}")
|
||||
|
||||
if state.get("gateway_id"):
|
||||
try:
|
||||
gw_control.delete_gateway(gatewayIdentifier=state["gateway_id"])
|
||||
print(f" Deleted gateway: {state['gateway_id']}")
|
||||
except Exception as e:
|
||||
print(f" Warning: {e}")
|
||||
|
||||
if state.get("guardrail_id"):
|
||||
try:
|
||||
bedrock.delete_guardrail(guardrailIdentifier=state["guardrail_id"])
|
||||
print(f" Deleted guardrail: {state['guardrail_id']}")
|
||||
except Exception as e:
|
||||
print(f" Warning: {e}")
|
||||
|
||||
# Delete batch evaluations created by this app
|
||||
dp_client = boto3.client("bedrock-agentcore", region_name=REGION)
|
||||
try:
|
||||
evals = dp_client.list_batch_evaluations()
|
||||
for ev in evals.get("batchEvaluations", evals.get("items", [])):
|
||||
ev_name = ev.get("batchEvaluationName", ev.get("name", ""))
|
||||
ev_id = ev.get("batchEvaluationId", "")
|
||||
if ev_name.startswith("weather_eval_"):
|
||||
try:
|
||||
dp_client.delete_batch_evaluation(batchEvaluationId=ev_id)
|
||||
print(f" Deleted batch evaluation: {ev_name}")
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f" Warning (batch evals): {e}")
|
||||
|
||||
# Delete recommendations created by this app
|
||||
try:
|
||||
recs = dp_client.list_recommendations()
|
||||
for rec in recs.get("recommendationSummaries", recs.get("recommendations", recs.get("items", []))):
|
||||
rec_name = rec.get("name", "")
|
||||
rec_id = rec.get("recommendationId", "")
|
||||
if rec_name.startswith("weather_rec_"):
|
||||
try:
|
||||
dp_client.delete_recommendation(recommendationId=rec_id)
|
||||
print(f" Deleted recommendation: {rec_name}")
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f" Warning (recommendations): {e}")
|
||||
|
||||
delete_harness_role()
|
||||
STATE_FILE.unlink(missing_ok=True)
|
||||
print("[resources] Cleanup complete")
|
||||
@@ -0,0 +1,212 @@
|
||||
"""Skills — generate weather forecast reports as XLSX spreadsheets.
|
||||
|
||||
Tries Git-based skill fetching first (no container needed). Falls back to
|
||||
path-based approach with Node.js container if Git parameter is not supported.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ParamValidationError
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
|
||||
from utils.client import get_agentcore_control_client, get_agentcore_client
|
||||
|
||||
from resources import REGION
|
||||
|
||||
NODE_CONTAINER = "public.ecr.aws/docker/library/node:slim"
|
||||
MODEL_ID = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
|
||||
_skill_installed: dict[str, bool] = {}
|
||||
|
||||
|
||||
def _run_command(client, harness_arn: str, session_id: str, cmd: str) -> str:
|
||||
"""Run a shell command on the agent VM."""
|
||||
output = ""
|
||||
resp = client.invoke_agent_runtime_command(
|
||||
agentRuntimeArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
body={"command": cmd},
|
||||
)
|
||||
for event in resp["stream"]:
|
||||
if "chunk" in event and "contentDelta" in event["chunk"]:
|
||||
delta = event["chunk"]["contentDelta"]
|
||||
if "stdout" in delta:
|
||||
output += delta["stdout"]
|
||||
if "stderr" in delta:
|
||||
output += delta["stderr"]
|
||||
return output
|
||||
|
||||
|
||||
def _build_prompt(city: str) -> str:
|
||||
return (
|
||||
f"Create an Excel spreadsheet with a 7-day weather forecast for {city}. "
|
||||
f"The first row must be a title: '{city} - 7-Day Weather Forecast' merged across all columns. "
|
||||
"Then include columns for: Day, Condition, High (°F), Low (°F), Wind (mph), Humidity (%), UV Index. "
|
||||
"Add realistic weather data that varies day to day. "
|
||||
"Include a summary row at the bottom with averages. "
|
||||
"Apply formatting: bold title, bold headers, alternating row colors, conditional formatting "
|
||||
"(red for high temps > 90°F, blue for low temps < 40°F). "
|
||||
"Save it as /tmp/weather_forecast.xlsx"
|
||||
)
|
||||
|
||||
|
||||
def _download_file(client, harness_arn: str, session_id: str) -> str:
|
||||
"""Download the generated xlsx file as base64."""
|
||||
b64_data = ""
|
||||
resp = client.invoke_agent_runtime_command(
|
||||
agentRuntimeArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
body={"command": "base64 /tmp/weather_forecast.xlsx 2>/dev/null"},
|
||||
)
|
||||
for event in resp["stream"]:
|
||||
if "chunk" in event and "contentDelta" in event["chunk"]:
|
||||
delta = event["chunk"]["contentDelta"]
|
||||
if "stdout" in delta:
|
||||
b64_data += delta["stdout"]
|
||||
return b64_data.strip().replace("\n", "")
|
||||
|
||||
|
||||
def _try_git_skill(client, harness_arn: str, session_id: str, city: str) -> str | None:
|
||||
"""Try invoking with Git-based skill fetch (newer boto3 only)."""
|
||||
try:
|
||||
response = client.invoke_harness(
|
||||
harnessArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
skills=[{"git": {"url": "https://github.com/anthropics/skills", "path": "skills/xlsx"}}],
|
||||
messages=[{"role": "user", "content": [{"text": _build_prompt(city)}]}],
|
||||
model={"bedrockModelConfig": {"modelId": MODEL_ID}},
|
||||
timeoutSeconds=300,
|
||||
)
|
||||
agent_text = ""
|
||||
for event in response["stream"]:
|
||||
if "contentBlockDelta" in event:
|
||||
delta = event["contentBlockDelta"].get("delta", {})
|
||||
if "text" in delta:
|
||||
agent_text += delta["text"]
|
||||
elif "internalServerException" in event:
|
||||
print(f"[skills] Stream error: {event['internalServerException']}")
|
||||
return None
|
||||
print(f"[skills] Git skill completed. Agent response length: {len(agent_text)}")
|
||||
return agent_text
|
||||
except ParamValidationError:
|
||||
print("[skills] Git-based skill not supported, falling back to path approach")
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"[skills] Git-based skill exception: {type(e).__name__}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _install_skill_path(client, control, harness_id: str, harness_arn: str, session_id: str) -> bool:
|
||||
"""Install xlsx skill via shell (requires Node.js container)."""
|
||||
if _skill_installed.get(session_id):
|
||||
return True
|
||||
|
||||
# Attach Node.js container if needed
|
||||
print("[skills] Checking container...")
|
||||
harness_info = control.get_harness(harnessId=harness_id)
|
||||
current_artifact = harness_info["harness"].get("environmentArtifact", {})
|
||||
has_container = bool(current_artifact.get("containerConfiguration", {}).get("containerUri"))
|
||||
|
||||
if not has_container:
|
||||
print(f"[skills] Attaching Node.js container: {NODE_CONTAINER}")
|
||||
control.update_harness(
|
||||
harnessId=harness_id,
|
||||
environmentArtifact={
|
||||
"optionalValue": {"containerConfiguration": {"containerUri": NODE_CONTAINER}}
|
||||
},
|
||||
)
|
||||
for _ in range(24):
|
||||
status = control.get_harness(harnessId=harness_id)["harness"]["status"]
|
||||
if status == "READY":
|
||||
break
|
||||
time.sleep(5)
|
||||
|
||||
# Install skill
|
||||
print("[skills] Installing xlsx skill via npx...")
|
||||
_run_command(
|
||||
client, harness_arn, session_id,
|
||||
"apt-get update -qq && apt-get install git -y -qq > /dev/null 2>&1 && "
|
||||
"npx skills add https://github.com/anthropics/skills --skill xlsx --yes 2>&1 | tail -3"
|
||||
)
|
||||
|
||||
# Verify
|
||||
verify = _run_command(client, harness_arn, session_id, "ls .agents/skills/xlsx/ 2>/dev/null && echo OK || echo MISSING")
|
||||
if "OK" in verify:
|
||||
_skill_installed[session_id] = True
|
||||
print("[skills] Skill installed successfully")
|
||||
return True
|
||||
print("[skills] Skill installation failed")
|
||||
return False
|
||||
|
||||
|
||||
def generate_weather_report(harness_arn: str, harness_id: str, session_id: str, city: str = "the cities discussed") -> dict:
|
||||
"""Generate a weather forecast XLSX report. Tries Git skill first, falls back to path."""
|
||||
try:
|
||||
return _generate_report_inner(harness_arn, harness_id, session_id, city)
|
||||
except Exception as e:
|
||||
print(f"[skills] Unhandled exception: {type(e).__name__}: {e}")
|
||||
return {"success": False, "error": f"{type(e).__name__}: {e}"}
|
||||
|
||||
|
||||
def _generate_report_inner(harness_arn: str, harness_id: str, session_id: str, city: str) -> dict:
|
||||
client = get_agentcore_client(config=Config(read_timeout=360))
|
||||
control = get_agentcore_control_client()
|
||||
|
||||
# Try Git-based skill first (simpler, no container needed)
|
||||
print("[skills] Trying Git-based skill fetch...")
|
||||
agent_text = _try_git_skill(client, harness_arn, session_id, city)
|
||||
|
||||
if agent_text is not None:
|
||||
# Git skill ran — check if file was generated
|
||||
print(f"[skills] Git skill completed. Agent text length: {len(agent_text)}")
|
||||
b64_clean = _download_file(client, harness_arn, session_id)
|
||||
if b64_clean:
|
||||
return {
|
||||
"success": True,
|
||||
"file_data": b64_clean,
|
||||
"filename": "weather_forecast.xlsx",
|
||||
"agent_response": agent_text[:500],
|
||||
}
|
||||
print("[skills] Git skill ran but no file generated, falling back to path approach")
|
||||
|
||||
# Fallback: install skill via path
|
||||
if not _install_skill_path(client, control, harness_id, harness_arn, session_id):
|
||||
return {"success": False, "error": "Failed to install xlsx skill"}
|
||||
|
||||
# Invoke with path-based skill
|
||||
print("[skills] Invoking with path-based skill...")
|
||||
response = client.invoke_harness(
|
||||
harnessArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
skills=[{"path": ".agents/skills/xlsx"}],
|
||||
messages=[{"role": "user", "content": [{"text": _build_prompt(city)}]}],
|
||||
model={"bedrockModelConfig": {"modelId": MODEL_ID}},
|
||||
timeoutSeconds=300,
|
||||
)
|
||||
agent_text = ""
|
||||
for event in response["stream"]:
|
||||
if "contentBlockDelta" in event:
|
||||
delta = event["contentBlockDelta"].get("delta", {})
|
||||
if "text" in delta:
|
||||
agent_text += delta["text"]
|
||||
|
||||
# Download the file
|
||||
b64_clean = _download_file(client, harness_arn, session_id)
|
||||
|
||||
if b64_clean:
|
||||
return {
|
||||
"success": True,
|
||||
"file_data": b64_clean,
|
||||
"filename": "weather_forecast.xlsx",
|
||||
"agent_response": agent_text[:500] if agent_text else "",
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"error": "No file generated",
|
||||
"agent_response": agent_text[:500] if agent_text else "",
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Weather Agent — Cleanup
|
||||
# Deletes all AWS resources and stops any running servers.
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${YELLOW}Weather Agent — Cleanup${NC}"
|
||||
echo "============================================================"
|
||||
|
||||
# Stop servers
|
||||
echo ""
|
||||
echo "Stopping servers..."
|
||||
[ -f backend.pid ] && kill "$(cat backend.pid)" 2>/dev/null && rm -f backend.pid && echo " Stopped backend"
|
||||
[ -f frontend.pid ] && kill "$(cat frontend.pid)" 2>/dev/null && rm -f frontend.pid && echo " Stopped frontend"
|
||||
lsof -ti:8000 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
lsof -ti:5173 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
|
||||
# Delete AWS resources
|
||||
if [ -f "resource_info.json" ]; then
|
||||
echo ""
|
||||
echo "Deleting AWS resources..."
|
||||
|
||||
if [ -d "venv" ]; then
|
||||
source venv/bin/activate
|
||||
else
|
||||
echo -e "${RED} No venv found. Create one first: python3 -m venv venv && source venv/bin/activate && pip install boto3${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import sys
|
||||
sys.path.insert(0, 'backend')
|
||||
from resources import destroy_resources
|
||||
destroy_resources()
|
||||
"
|
||||
deactivate 2>/dev/null || true
|
||||
else
|
||||
echo ""
|
||||
echo " No resource_info.json found — nothing to delete in AWS"
|
||||
fi
|
||||
|
||||
# Clean local artifacts
|
||||
echo ""
|
||||
echo "Cleaning local files..."
|
||||
rm -f backend.log frontend.log backend.pid frontend.pid
|
||||
echo " Removed log and pid files"
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}Cleanup complete.${NC}"
|
||||
echo ""
|
||||
echo " To run the app again:"
|
||||
echo " ./start.sh"
|
||||
echo ""
|
||||
echo " To also remove the virtual environment and node_modules:"
|
||||
echo " rm -rf venv frontend/node_modules"
|
||||
@@ -0,0 +1,12 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Weather Agent — AgentCore Harness</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.jsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
+1795
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "weather-agent-ui",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"start": "vite",
|
||||
"build": "vite build"
|
||||
},
|
||||
"dependencies": {
|
||||
"react": "^18.3.1",
|
||||
"react-dom": "^18.3.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@vitejs/plugin-react": "^4.3.3",
|
||||
"vite": "^6.4.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,427 @@
|
||||
:root {
|
||||
--bg-primary: #0f1419;
|
||||
--bg-secondary: #1a2332;
|
||||
--bg-card: #1e2d3d;
|
||||
--bg-input: #253341;
|
||||
--text-primary: #e1e8ed;
|
||||
--text-secondary: #8899a6;
|
||||
--accent: #1da1f2;
|
||||
--accent-green: #17bf63;
|
||||
--accent-orange: #f5a623;
|
||||
--accent-red: #e0245e;
|
||||
--border: #2f3d4d;
|
||||
--radius: 12px;
|
||||
}
|
||||
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: var(--bg-primary);
|
||||
color: var(--text-primary);
|
||||
min-height: 100vh;
|
||||
}
|
||||
|
||||
.app {
|
||||
display: grid;
|
||||
grid-template-rows: auto 1fr auto;
|
||||
height: 100vh;
|
||||
}
|
||||
|
||||
/* Header */
|
||||
.header {
|
||||
padding: 16px 24px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.header h1 {
|
||||
font-size: 1.2rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.header .status {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 0.8rem;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
background: var(--accent-green);
|
||||
}
|
||||
|
||||
.status-dot.offline { background: var(--accent-red); }
|
||||
.status-dot.loading { background: var(--accent-orange); animation: pulse 1s infinite; }
|
||||
|
||||
@keyframes pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.4; }
|
||||
}
|
||||
|
||||
/* Main layout */
|
||||
.main {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 1px;
|
||||
background: var(--border);
|
||||
overflow: hidden;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
/* Chat panel */
|
||||
.chat-panel {
|
||||
background: var(--bg-primary);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-height: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.messages {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 16px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.message {
|
||||
max-width: 85%;
|
||||
padding: 10px 14px;
|
||||
border-radius: var(--radius);
|
||||
font-size: 0.9rem;
|
||||
line-height: 1.5;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.message.user {
|
||||
align-self: flex-end;
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.message.assistant {
|
||||
align-self: flex-start;
|
||||
background: var(--bg-card);
|
||||
}
|
||||
|
||||
.message.tool {
|
||||
align-self: flex-start;
|
||||
background: transparent;
|
||||
border: 1px solid var(--border);
|
||||
color: var(--text-secondary);
|
||||
font-size: 0.8rem;
|
||||
padding: 6px 10px;
|
||||
}
|
||||
|
||||
.message.error {
|
||||
align-self: flex-start;
|
||||
background: rgba(224, 36, 94, 0.15);
|
||||
border: 1px solid var(--accent-red);
|
||||
color: var(--accent-red);
|
||||
}
|
||||
|
||||
.chat-input {
|
||||
padding: 12px 16px;
|
||||
border-top: 1px solid var(--border);
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.chat-input input {
|
||||
flex: 1;
|
||||
padding: 10px 14px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
background: var(--bg-input);
|
||||
color: var(--text-primary);
|
||||
font-size: 0.9rem;
|
||||
outline: none;
|
||||
}
|
||||
|
||||
.chat-input input:focus { border-color: var(--accent); }
|
||||
|
||||
.chat-input button {
|
||||
padding: 10px 20px;
|
||||
border: none;
|
||||
border-radius: var(--radius);
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
font-size: 0.9rem;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.chat-input button:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
/* Right panel */
|
||||
.right-panel {
|
||||
background: var(--bg-secondary);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.panel-tabs {
|
||||
display: flex;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.panel-tabs button {
|
||||
flex: 1;
|
||||
padding: 12px;
|
||||
border: none;
|
||||
background: transparent;
|
||||
color: var(--text-secondary);
|
||||
font-size: 0.85rem;
|
||||
cursor: pointer;
|
||||
border-bottom: 2px solid transparent;
|
||||
}
|
||||
|
||||
.panel-tabs button.active {
|
||||
color: var(--accent);
|
||||
border-bottom-color: var(--accent);
|
||||
}
|
||||
|
||||
.panel-content {
|
||||
flex: 1;
|
||||
padding: 16px;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
/* Weather cards */
|
||||
.weather-cards {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.weather-card {
|
||||
background: var(--bg-card);
|
||||
border-radius: var(--radius);
|
||||
padding: 16px;
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.weather-card .icon {
|
||||
font-size: 1.5rem;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.weather-card .label {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-secondary);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
.weather-card .value {
|
||||
font-size: 1.4rem;
|
||||
font-weight: 600;
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
.weather-card .detail {
|
||||
font-size: 0.8rem;
|
||||
color: var(--text-secondary);
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
/* Traces panel */
|
||||
.traces-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.trace-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 10px 12px;
|
||||
background: var(--bg-card);
|
||||
border-radius: 8px;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.trace-item .trace-dot {
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
border-radius: 50%;
|
||||
background: var(--accent-green);
|
||||
}
|
||||
|
||||
.trace-item .trace-dot.error { background: var(--accent-red); }
|
||||
|
||||
.trace-item .trace-id {
|
||||
color: var(--text-secondary);
|
||||
font-family: monospace;
|
||||
}
|
||||
|
||||
.trace-item .trace-duration {
|
||||
margin-left: auto;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
/* Eval panel */
|
||||
.eval-results {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.eval-item {
|
||||
background: var(--bg-card);
|
||||
border-radius: 8px;
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.eval-item .eval-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.eval-item .eval-name {
|
||||
font-size: 0.85rem;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.eval-item .eval-score {
|
||||
font-size: 0.85rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.eval-bar {
|
||||
height: 4px;
|
||||
background: var(--bg-input);
|
||||
border-radius: 2px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.eval-bar-fill {
|
||||
height: 100%;
|
||||
border-radius: 2px;
|
||||
transition: width 0.5s ease;
|
||||
}
|
||||
|
||||
.eval-label {
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-secondary);
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
/* Footer */
|
||||
.footer {
|
||||
padding: 8px 24px;
|
||||
border-top: 1px solid var(--border);
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-secondary);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
/* Empty states */
|
||||
.empty-state {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
color: var(--text-secondary);
|
||||
text-align: center;
|
||||
padding: 40px;
|
||||
}
|
||||
|
||||
.empty-state .empty-icon {
|
||||
font-size: 2.5rem;
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
.empty-state p {
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
/* Typing indicator */
|
||||
.typing-dots {
|
||||
display: flex;
|
||||
gap: 4px;
|
||||
align-items: center;
|
||||
padding: 4px 0;
|
||||
}
|
||||
|
||||
.typing-dots span {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
background: var(--text-secondary);
|
||||
animation: bounce 1.4s infinite ease-in-out both;
|
||||
}
|
||||
|
||||
.typing-dots span:nth-child(1) { animation-delay: 0s; }
|
||||
.typing-dots span:nth-child(2) { animation-delay: 0.2s; }
|
||||
.typing-dots span:nth-child(3) { animation-delay: 0.4s; }
|
||||
|
||||
@keyframes bounce {
|
||||
0%, 80%, 100% { transform: scale(0.6); opacity: 0.4; }
|
||||
40% { transform: scale(1); opacity: 1; }
|
||||
}
|
||||
|
||||
/* Loading spinner */
|
||||
.spinner {
|
||||
display: inline-block;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
border: 2px solid var(--border);
|
||||
border-top-color: var(--accent);
|
||||
border-radius: 50%;
|
||||
animation: spin 0.8s linear infinite;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to { transform: rotate(360deg); }
|
||||
}
|
||||
|
||||
/* Provisioning overlay */
|
||||
.provisioning {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100vh;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.provisioning h2 {
|
||||
font-size: 1.1rem;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.provisioning p {
|
||||
color: var(--text-secondary);
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.btn-secondary {
|
||||
padding: 8px 16px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
background: transparent;
|
||||
color: var(--text-primary);
|
||||
font-size: 0.8rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.btn-secondary:hover { border-color: var(--accent); }
|
||||
@@ -0,0 +1,548 @@
|
||||
import React, { useState, useEffect, useRef } from 'react';
|
||||
import { getStatus, streamChat, getTraces, runEvaluation, generateReport, runOptimization } from './api';
|
||||
|
||||
function App() {
|
||||
const [ready, setReady] = useState(false);
|
||||
const [status, setStatus] = useState(null);
|
||||
const [messages, setMessages] = useState([]);
|
||||
const [input, setInput] = useState('');
|
||||
const [streaming, setStreaming] = useState(false);
|
||||
const [sessionId, setSessionId] = useState(null);
|
||||
const [activeTab, setActiveTab] = useState('weather');
|
||||
const [traces, setTraces] = useState([]);
|
||||
const [evalResults, setEvalResults] = useState([]);
|
||||
const [evalLoading, setEvalLoading] = useState(false);
|
||||
const [reportLoading, setReportLoading] = useState(false);
|
||||
const [reportResult, setReportResult] = useState(null);
|
||||
const [optimizeLoading, setOptimizeLoading] = useState(false);
|
||||
const [optimizeResult, setOptimizeResult] = useState(null);
|
||||
const [evalBatchId, setEvalBatchId] = useState(null);
|
||||
const [weatherData, setWeatherData] = useState([]);
|
||||
const messagesEndRef = useRef(null);
|
||||
|
||||
useEffect(() => {
|
||||
const poll = setInterval(async () => {
|
||||
try {
|
||||
const s = await getStatus();
|
||||
setStatus(s);
|
||||
if (s.ready) {
|
||||
setReady(true);
|
||||
clearInterval(poll);
|
||||
}
|
||||
} catch (e) { /* backend not up yet */ }
|
||||
}, 2000);
|
||||
return () => clearInterval(poll);
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' });
|
||||
}, [messages]);
|
||||
|
||||
const parseWeatherData = (text) => {
|
||||
const cards = [];
|
||||
|
||||
// Extract city name from the text
|
||||
const cityMatch = text.match(/(?:weather|temperature|wind|conditions|data|information|forecast)\s+(?:in|for|at|of)\s+([A-Z][a-zA-Z\s\-]+?)(?:[,.:;\n]|\s+(?:is|are|shows|right|today|this|currently|here|based|for\s+you))/i)
|
||||
|| text.match(/(?:in|for)\s+([A-Z][a-zA-Z\s\-]+?)(?:[,.:;\n]|\s+(?:is|are|right now|today|this|currently|here's|based|for\s+you))/);
|
||||
let city = cityMatch ? cityMatch[1].trim() : '';
|
||||
city = city.replace(/\s+for$/i, '');
|
||||
|
||||
const tempMatch = text.match(/(-?\d+\.?\d*)\s*°?\s*[CF]|temperature[:\s]+(-?\d+\.?\d*)/i);
|
||||
if (tempMatch) cards.push({ icon: '🌡️', label: 'Temperature', value: tempMatch[0], detail: city || 'Current' });
|
||||
|
||||
const windMatch = text.match(/(\d+\.?\d*)\s*(km\/h|mph|m\/s|kph|knots)/i);
|
||||
if (windMatch) cards.push({ icon: '💨', label: 'Wind', value: windMatch[0], detail: city || 'Speed' });
|
||||
|
||||
const uvMatch = text.match(/UV\s*(?:index)?[:\s]*(\d+\.?\d*)/i);
|
||||
if (uvMatch) cards.push({ icon: '☀️', label: 'UV Index', value: uvMatch[1], detail: city || (uvMatch[1] > 6 ? 'High — use sunscreen' : 'Moderate') });
|
||||
|
||||
const sunriseMatch = text.match(/sunrise[:\s|]*(\d{1,2}:\d{2}\s*(?:AM|PM)?)/i);
|
||||
if (sunriseMatch) cards.push({ icon: '🌅', label: 'Sunrise', value: sunriseMatch[1], detail: city });
|
||||
|
||||
const sunsetMatch = text.match(/sunset[:\s|]*(\d{1,2}:\d{2}\s*(?:AM|PM)?)/i)
|
||||
|| text.match(/(\d{1,2}:\d{2}\s*PM)\s*(?:\||\n|$)/i);
|
||||
if (sunsetMatch) cards.push({ icon: '🌇', label: 'Sunset', value: sunsetMatch[1], detail: city });
|
||||
|
||||
const humidityMatch = text.match(/humidity[:\s]*(\d+\.?\d*)\s*%?/i);
|
||||
if (humidityMatch) cards.push({ icon: '💧', label: 'Humidity', value: `${humidityMatch[1]}%`, detail: city });
|
||||
|
||||
const moonMatch = text.match(/(waxing|waning|full|new|crescent|gibbous|quarter)\s*(moon|gibbous|crescent|quarter)?/i);
|
||||
if (moonMatch) cards.push({ icon: '🌙', label: 'Moon', value: moonMatch[0], detail: city });
|
||||
|
||||
return cards;
|
||||
};
|
||||
|
||||
const handleSend = async () => {
|
||||
if (!input.trim() || streaming) return;
|
||||
|
||||
const userMsg = input.trim();
|
||||
setInput('');
|
||||
setMessages(prev => [...prev, { role: 'user', content: userMsg }]);
|
||||
setStreaming(true);
|
||||
|
||||
let allText = '';
|
||||
let currentChunk = '';
|
||||
let currentSessionId = sessionId;
|
||||
|
||||
await streamChat(userMsg, sessionId, (event) => {
|
||||
if (event.type === 'session_id') {
|
||||
currentSessionId = event.session_id;
|
||||
setSessionId(event.session_id);
|
||||
} else if (event.type === 'text') {
|
||||
allText += event.content;
|
||||
currentChunk += event.content;
|
||||
setMessages(prev => {
|
||||
const msgs = [...prev];
|
||||
const last = msgs[msgs.length - 1];
|
||||
if (last && last.role === 'assistant') {
|
||||
msgs[msgs.length - 1] = { ...last, content: last.content + event.content };
|
||||
} else {
|
||||
msgs.push({ role: 'assistant', content: event.content });
|
||||
}
|
||||
return msgs;
|
||||
});
|
||||
} else if (event.type === 'tool') {
|
||||
currentChunk = '';
|
||||
setMessages(prev => [...prev, { role: 'tool', content: `Using tool: ${event.name}` }]);
|
||||
} else if (event.type === 'error') {
|
||||
setMessages(prev => [...prev, { role: 'error', content: event.content }]);
|
||||
}
|
||||
});
|
||||
|
||||
// Parse weather data from the full response text
|
||||
if (allText) {
|
||||
const parsed = parseWeatherData(allText);
|
||||
if (parsed.length > 0) {
|
||||
setWeatherData(prev => [...prev, ...parsed]);
|
||||
}
|
||||
}
|
||||
|
||||
setStreaming(false);
|
||||
|
||||
// Fetch traces after a delay
|
||||
setTimeout(async () => {
|
||||
const t = await getTraces(5);
|
||||
setTraces(t.traces || []);
|
||||
}, 3000);
|
||||
};
|
||||
|
||||
|
||||
const handleEval = async () => {
|
||||
if (!sessionId) return;
|
||||
setEvalLoading(true);
|
||||
setEvalResults([]);
|
||||
|
||||
try {
|
||||
const result = await runEvaluation(sessionId);
|
||||
setEvalResults(result.scores || []);
|
||||
setEvalBatchId(result.batch_name || result.batch_id || null);
|
||||
if (result.error) {
|
||||
setEvalResults([{ evaluator: 'Error', score: 0, label: result.error }]);
|
||||
}
|
||||
} catch (e) {
|
||||
setEvalResults([{ evaluator: 'Error', score: 0, label: e.message }]);
|
||||
}
|
||||
setEvalLoading(false);
|
||||
};
|
||||
|
||||
const handleGenerateReport = async () => {
|
||||
if (!sessionId) return;
|
||||
setReportLoading(true);
|
||||
setReportResult(null);
|
||||
try {
|
||||
const result = await generateReport(sessionId);
|
||||
setReportResult(result);
|
||||
if (result.success && result.file_data) {
|
||||
const blob = new Blob(
|
||||
[Uint8Array.from(atob(result.file_data), c => c.charCodeAt(0))],
|
||||
{ type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' }
|
||||
);
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = result.filename || 'weather_forecast.xlsx';
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
} catch (e) {
|
||||
setReportResult({ success: false, error: e.message });
|
||||
}
|
||||
setReportLoading(false);
|
||||
};
|
||||
|
||||
const handleOptimize = async () => {
|
||||
setOptimizeLoading(true);
|
||||
setOptimizeResult(null);
|
||||
try {
|
||||
const result = await runOptimization();
|
||||
setOptimizeResult(result);
|
||||
} catch (e) {
|
||||
setOptimizeResult({ status: 'FAILED', error: e.message });
|
||||
}
|
||||
setOptimizeLoading(false);
|
||||
};
|
||||
|
||||
const handleKeyDown = (e) => {
|
||||
if (e.key === 'Enter' && !e.shiftKey) {
|
||||
e.preventDefault();
|
||||
handleSend();
|
||||
}
|
||||
};
|
||||
|
||||
const renderMarkdown = (text) => {
|
||||
const lines = text.split('\n');
|
||||
return lines.map((line, lineIdx) => {
|
||||
let content = line;
|
||||
let isHeading = false;
|
||||
let headingLevel = 0;
|
||||
|
||||
if (line.startsWith('### ')) {
|
||||
content = line.slice(4);
|
||||
isHeading = true;
|
||||
headingLevel = 3;
|
||||
} else if (line.startsWith('## ')) {
|
||||
content = line.slice(3);
|
||||
isHeading = true;
|
||||
headingLevel = 2;
|
||||
} else if (line.startsWith('# ')) {
|
||||
content = line.slice(2);
|
||||
isHeading = true;
|
||||
headingLevel = 1;
|
||||
}
|
||||
|
||||
const inlineParts = content.split(/(\*\*.*?\*\*)/g).map((part, i) => {
|
||||
if (part.startsWith('**') && part.endsWith('**')) {
|
||||
return <strong key={i}>{part.slice(2, -2)}</strong>;
|
||||
}
|
||||
return part;
|
||||
});
|
||||
|
||||
if (isHeading) {
|
||||
const style = { fontWeight: 600, fontSize: headingLevel === 1 ? '1.2em' : headingLevel === 2 ? '1.1em' : '1em', marginTop: '8px' };
|
||||
return <div key={lineIdx} style={style}>{inlineParts}</div>;
|
||||
}
|
||||
|
||||
return <span key={lineIdx}>{inlineParts}{lineIdx < lines.length - 1 ? '\n' : ''}</span>;
|
||||
});
|
||||
};
|
||||
|
||||
if (!ready) {
|
||||
return (
|
||||
<div className="provisioning">
|
||||
<div className="spinner" style={{ width: 32, height: 32 }} />
|
||||
<h2>Provisioning AWS Resources</h2>
|
||||
<p>Setting up Gateway, Harness, and Guardrail...</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="app">
|
||||
<div className="header">
|
||||
<h1>Weather Agent</h1>
|
||||
<div className="status">
|
||||
<span className="status-dot" />
|
||||
<span>Harness: {status?.harness_name || status?.harness_id}</span>
|
||||
<span>|</span>
|
||||
<span>Gateway: {status?.gateway_name || status?.gateway_id}</span>
|
||||
{status?.guardrail_id && <><span>|</span><span>Guardrail: {status?.guardrail_name || status?.guardrail_id}</span></>}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="main">
|
||||
{/* Left: Chat */}
|
||||
<div className="chat-panel">
|
||||
<div className="messages">
|
||||
{messages.length === 0 && (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">⛅</div>
|
||||
<p>Ask about the weather anywhere in the world. Try: "What's the weather in Tokyo?"</p>
|
||||
</div>
|
||||
)}
|
||||
{messages.map((msg, i) => (
|
||||
<div key={i} className={`message ${msg.role}`}>
|
||||
{msg.role === 'assistant' ? renderMarkdown(msg.content) : msg.content}
|
||||
</div>
|
||||
))}
|
||||
{streaming && messages[messages.length - 1]?.role === 'user' && (
|
||||
<div className="message assistant typing">
|
||||
<span className="typing-dots">
|
||||
<span /><span /><span />
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
<div ref={messagesEndRef} />
|
||||
</div>
|
||||
<div className="chat-input">
|
||||
<input
|
||||
value={input}
|
||||
onChange={e => setInput(e.target.value)}
|
||||
onKeyDown={handleKeyDown}
|
||||
placeholder="Ask about weather, wind, UV, sunrise..."
|
||||
disabled={streaming}
|
||||
/>
|
||||
<button onClick={handleSend} disabled={streaming || !input.trim()}>
|
||||
{streaming ? <span className="spinner" /> : 'Send'}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Right: Weather / Traces / Evals */}
|
||||
<div className="right-panel">
|
||||
<div className="panel-tabs">
|
||||
<button className={activeTab === 'weather' ? 'active' : ''} onClick={() => setActiveTab('weather')}>
|
||||
Weather
|
||||
</button>
|
||||
<button className={activeTab === 'traces' ? 'active' : ''} onClick={() => setActiveTab('traces')}>
|
||||
Traces ({traces.length})
|
||||
</button>
|
||||
<button className={activeTab === 'skills' ? 'active' : ''} onClick={() => setActiveTab('skills')}>
|
||||
Skills
|
||||
</button>
|
||||
<button className={activeTab === 'optimize' ? 'active' : ''} onClick={() => setActiveTab('optimize')}>
|
||||
Optimization
|
||||
</button>
|
||||
<button className={activeTab === 'evals' ? 'active' : ''} onClick={() => setActiveTab('evals')}>
|
||||
Evaluations
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="panel-content">
|
||||
{activeTab === 'weather' && (
|
||||
weatherData.length > 0 ? (
|
||||
<div className="weather-cards">
|
||||
{weatherData.map((card, i) => (
|
||||
<div key={i} className="weather-card">
|
||||
<div className="icon">{card.icon}</div>
|
||||
<div className="label">{card.label}</div>
|
||||
<div className="value">{card.value}</div>
|
||||
{card.detail && <div className="detail">{card.detail}</div>}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">📊</div>
|
||||
<p>Weather data will appear here as the agent responds with specific metrics.</p>
|
||||
</div>
|
||||
)
|
||||
)}
|
||||
|
||||
{activeTab === 'traces' && (
|
||||
<div>
|
||||
<div style={{ marginBottom: 12, display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<span style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>Traces (last 5 min)</span>
|
||||
<button className="btn-secondary" onClick={async () => { const t = await getTraces(5); setTraces(t.traces || []); }}>
|
||||
Refresh
|
||||
</button>
|
||||
</div>
|
||||
{traces.length > 0 && traces.some(t => t.trace_id && !t.error) ? (
|
||||
<div>
|
||||
<div className="traces-list">
|
||||
{traces.filter(t => t.trace_id).map((t, i) => (
|
||||
<div key={i} className="trace-item">
|
||||
<span className={`trace-dot ${t.has_error || t.has_fault ? 'error' : ''}`} />
|
||||
<span className="trace-id">{t.trace_id}</span>
|
||||
<span className="trace-duration">{t.spans ? `${t.spans} spans` : `${t.duration}s`}</span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12 }}>
|
||||
Search these trace IDs in CloudWatch > GenAI Observability > Bedrock AgentCore > Traces (may take a few minutes to appear)
|
||||
</p>
|
||||
</div>
|
||||
) : traces.length > 0 && (traces[0]?.error || traces.some(t => !t.trace_id)) ? (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">🔍</div>
|
||||
<p>Traces could not be loaded. Ensure CloudWatch Transaction Search is enabled in your account and region.</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">🔍</div>
|
||||
<p>Traces appear after you send messages. They may take a few seconds to index.</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{activeTab === 'skills' && (
|
||||
<div>
|
||||
<div style={{ marginBottom: 12, display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<span style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>XLSX Report Generation</span>
|
||||
<button className="btn-secondary" onClick={handleGenerateReport} disabled={!sessionId || reportLoading}>
|
||||
{reportLoading ? 'Generating...' : 'Generate Report'}
|
||||
</button>
|
||||
</div>
|
||||
{reportLoading && (
|
||||
<div className="empty-state">
|
||||
<div className="spinner" style={{ width: 24, height: 24 }} />
|
||||
<p style={{ marginTop: 12, fontSize: '0.85rem', lineHeight: '1.6' }}>
|
||||
Generating weather forecast spreadsheet using the xlsx skill... This typically takes 1-2 minutes.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{!reportLoading && reportResult && (
|
||||
<div className="eval-item" style={{ textAlign: 'center', padding: 20 }}>
|
||||
{reportResult.success ? (
|
||||
<>
|
||||
<p style={{ color: 'var(--accent-green)', fontWeight: 500, marginBottom: 8 }}>Report generated and downloaded</p>
|
||||
<p style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>{reportResult.filename}</p>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<p style={{ color: 'var(--accent-red)', fontWeight: 500, marginBottom: 8 }}>Report generation failed</p>
|
||||
<p style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>{reportResult.error}</p>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{!reportLoading && !reportResult && (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">📊</div>
|
||||
<p style={{ fontSize: '0.85rem', lineHeight: '1.6' }}>
|
||||
Generate a 7-day weather forecast as an Excel spreadsheet using the AgentCore xlsx skill. The report will use the last city you asked about.
|
||||
</p>
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12 }}>
|
||||
The skill is fetched from Git at invocation time — no container setup or pre-installation required.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{activeTab === 'evals' && (
|
||||
<div>
|
||||
<div style={{ marginBottom: 12, display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<span style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>Batch Evaluation</span>
|
||||
<button className="btn-secondary" onClick={handleEval} disabled={!sessionId || evalLoading || optimizeLoading}>
|
||||
{evalLoading ? 'Running...' : 'Run Eval'}
|
||||
</button>
|
||||
</div>
|
||||
{evalLoading && (
|
||||
<div className="empty-state">
|
||||
<div className="spinner" style={{ width: 24, height: 24 }} />
|
||||
<p style={{ marginTop: 12, fontSize: '0.85rem', lineHeight: '1.6' }}>Running batch evaluation... This typically takes 2-5 minutes.</p>
|
||||
<p style={{ marginTop: 12, fontSize: '0.75rem', color: 'var(--text-secondary)', lineHeight: '1.6' }}>Optimization is disabled while evaluation is running. You can track progress in Bedrock AgentCore > Evaluations > Batch evaluation.</p>
|
||||
</div>
|
||||
)}
|
||||
{!evalLoading && evalResults.length > 0 && (
|
||||
<div>
|
||||
<div className="eval-results">
|
||||
{evalResults.map((r, i) => {
|
||||
const score = r.score != null ? r.score : 0;
|
||||
const hasError = r.evaluator === 'Error';
|
||||
const color = hasError ? 'var(--text-secondary)' : score >= 0.8 ? 'var(--accent-green)' : score >= 0.5 ? 'var(--accent-orange)' : 'var(--accent-red)';
|
||||
return (
|
||||
<div key={i} className="eval-item">
|
||||
<div className="eval-header">
|
||||
<span className="eval-name">{r.evaluator}</span>
|
||||
<span className="eval-score" style={{ color }}>{hasError ? '—' : score.toFixed(2)}</span>
|
||||
</div>
|
||||
{!hasError && (
|
||||
<div className="eval-bar">
|
||||
<div className="eval-bar-fill" style={{ width: `${score * 100}%`, background: color }} />
|
||||
</div>
|
||||
)}
|
||||
<div className="eval-label">{hasError ? r.label : ''}</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
{evalResults.some(r => r.evaluator === 'Error') && (
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12, textAlign: 'center' }}>
|
||||
This may happen if Transaction Search was recently enabled and traces haven't fully indexed yet. Try again in a few minutes.
|
||||
</p>
|
||||
)}
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12, textAlign: 'center' }}>
|
||||
View full details in Bedrock AgentCore > Evaluations > Batch evaluation{evalBatchId && <> — <strong>{evalBatchId}</strong></>}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{!evalLoading && evalResults.length === 0 && (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">📋</div>
|
||||
<p style={{ fontSize: '0.85rem', lineHeight: '1.6' }}>
|
||||
Send some weather questions first, then click "Run Eval" to score the session.
|
||||
</p>
|
||||
<p style={{ fontSize: '0.85rem', color: 'var(--text-secondary)', marginTop: 12, lineHeight: '1.6' }}>
|
||||
Scores your conversation using built-in evaluators: Helpfulness, Correctness, Coherence, Faithfulness, and more.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{activeTab === 'optimize' && (
|
||||
<div>
|
||||
<div style={{ marginBottom: 12, display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<span style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>System Prompt Recommendation</span>
|
||||
<button className="btn-secondary" onClick={handleOptimize} disabled={optimizeLoading || evalLoading}>
|
||||
{optimizeLoading ? 'Running...' : 'Optimize'}
|
||||
</button>
|
||||
</div>
|
||||
{optimizeLoading && (
|
||||
<div className="empty-state">
|
||||
<div className="spinner" style={{ width: 24, height: 24 }} />
|
||||
<p style={{ marginTop: 12, fontSize: '0.85rem', lineHeight: '1.6' }}>
|
||||
Analyzing traces and generating an optimized system prompt... This typically takes 1-3 minutes.
|
||||
</p>
|
||||
<p style={{ marginTop: 12, fontSize: '0.75rem', color: 'var(--text-secondary)', lineHeight: '1.6' }}>
|
||||
Evaluations is disabled while optimization is running. You can track progress in Bedrock AgentCore > Optimizations > Recommendations.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{!optimizeLoading && optimizeResult && optimizeResult.status === 'COMPLETED' && (
|
||||
<div>
|
||||
<div className="eval-item" style={{ marginBottom: 12 }}>
|
||||
<div style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginBottom: 6 }}>RECOMMENDED SYSTEM PROMPT</div>
|
||||
<p style={{ fontSize: '0.8rem', lineHeight: '1.6', whiteSpace: 'pre-wrap' }}>
|
||||
{optimizeResult.recommended_prompt}
|
||||
</p>
|
||||
</div>
|
||||
{optimizeResult.explanation && (
|
||||
<div className="eval-item" style={{ marginBottom: 12 }}>
|
||||
<div style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginBottom: 6 }}>EXPLANATION</div>
|
||||
<p style={{ fontSize: '0.8rem', lineHeight: '1.6' }}>
|
||||
{optimizeResult.explanation}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12, textAlign: 'center' }}>
|
||||
View full details in Bedrock AgentCore > Optimizations > Recommendations — <strong>{optimizeResult.recommendation_name}</strong>
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{!optimizeLoading && optimizeResult && optimizeResult.status !== 'COMPLETED' && (
|
||||
<div className="eval-item" style={{ textAlign: 'center', padding: 20 }}>
|
||||
<p style={{ color: 'var(--accent-red)', fontWeight: 500, marginBottom: 8 }}>Optimization failed</p>
|
||||
<p style={{ fontSize: '0.8rem', color: 'var(--text-secondary)' }}>{optimizeResult.error}</p>
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12 }}>Traces may need more time to index. Try again after a few minutes, or send more weather questions first.</p>
|
||||
</div>
|
||||
)}
|
||||
{!optimizeLoading && !optimizeResult && (
|
||||
<div className="empty-state">
|
||||
<div className="empty-icon">🚀</div>
|
||||
<p style={{ fontSize: '0.85rem', lineHeight: '1.6' }}>
|
||||
Analyze your agent's traces and generate an AI-improved system prompt optimized for goal success.
|
||||
</p>
|
||||
<p style={{ fontSize: '0.75rem', color: 'var(--text-secondary)', marginTop: 12, lineHeight: '1.6' }}>
|
||||
Send some weather questions first, then click "Optimize" to generate a recommendation. View full details in Bedrock AgentCore > Optimizations > Recommendations.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="footer">
|
||||
<span>AgentCore Harness Demo — Gateway + Guardrails + Skills + Observability + Evaluations + Optimization</span>
|
||||
<span><span className="status-dot" style={{ display: 'inline-block', marginRight: 6 }} />{status?.region}</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default App;
|
||||
@@ -0,0 +1,69 @@
|
||||
const BASE = '';
|
||||
|
||||
export async function getStatus() {
|
||||
const res = await fetch(`${BASE}/api/status`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function streamChat(message, sessionId, onEvent) {
|
||||
const res = await fetch(`${BASE}/api/chat`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ message, session_id: sessionId }),
|
||||
});
|
||||
|
||||
const reader = res.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = '';
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
buffer += decoder.decode(value, { stream: true });
|
||||
const lines = buffer.split('\n');
|
||||
buffer = lines.pop() || '';
|
||||
|
||||
for (const line of lines) {
|
||||
if (line.startsWith('data: ')) {
|
||||
try {
|
||||
const event = JSON.parse(line.slice(6));
|
||||
onEvent(event);
|
||||
} catch (e) {
|
||||
// skip malformed
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function getTraces(minutes = 10) {
|
||||
const res = await fetch(`${BASE}/api/traces?minutes=${minutes}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function runEvaluation(sessionId) {
|
||||
const res = await fetch(`${BASE}/api/evaluate`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ session_id: sessionId }),
|
||||
});
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function generateReport(sessionId, city) {
|
||||
const res = await fetch(`${BASE}/api/generate-report`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ session_id: sessionId, city }),
|
||||
});
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function runOptimization() {
|
||||
const res = await fetch(`${BASE}/api/optimize`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
return res.json();
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import React from 'react'
|
||||
import ReactDOM from 'react-dom/client'
|
||||
import App from './App.jsx'
|
||||
import './App.css'
|
||||
|
||||
ReactDOM.createRoot(document.getElementById('root')).render(
|
||||
<React.StrictMode>
|
||||
<App />
|
||||
</React.StrictMode>
|
||||
)
|
||||
@@ -0,0 +1,13 @@
|
||||
import { defineConfig } from 'vite'
|
||||
import react from '@vitejs/plugin-react'
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
port: 5173,
|
||||
proxy: {
|
||||
'/api': { target: 'http://localhost:8000', changeOrigin: true },
|
||||
'/health': { target: 'http://localhost:8000', changeOrigin: true },
|
||||
}
|
||||
}
|
||||
})
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 233 KiB |
@@ -0,0 +1,284 @@
|
||||
"""
|
||||
AgentCore Optimization — System Prompt Recommendation for the Weather Agent.
|
||||
|
||||
Analyzes traces from your weather agent sessions and generates an AI-improved
|
||||
system prompt optimized for a target evaluator (e.g., Helpfulness, GoalSuccessRate).
|
||||
|
||||
Prerequisites:
|
||||
- The weather agent web app must have been running with some chat sessions
|
||||
(traces need to exist in CloudWatch)
|
||||
- AWS_DEFAULT_REGION set
|
||||
- Transaction Search enabled in CloudWatch
|
||||
|
||||
Usage:
|
||||
# Run after using the web app for a few sessions:
|
||||
python optimize.py
|
||||
|
||||
# Specify evaluator to optimize for:
|
||||
python optimize.py --evaluator Builtin.Helpfulness
|
||||
|
||||
# Use a custom time range (last N days):
|
||||
python optimize.py --lookback 1
|
||||
|
||||
# Cleanup recommendations:
|
||||
python optimize.py --cleanup
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
from utils.client import get_agentcore_client
|
||||
|
||||
# -- CLI -----------------------------------------------------------------------
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate an optimized system prompt for the Weather Agent"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--evaluator",
|
||||
default="Builtin.GoalSuccessRate",
|
||||
help="Evaluator to optimize for (default: Builtin.GoalSuccessRate)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lookback",
|
||||
type=int,
|
||||
default=7,
|
||||
help="Days of traces to analyze (default: 7)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cleanup",
|
||||
action="store_true",
|
||||
help="Delete all weather_rec_* recommendations and exit",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
# -- Configuration -------------------------------------------------------------
|
||||
REGION = boto3.session.Session().region_name or "us-east-1"
|
||||
STATE_FILE = Path(__file__).parent / "resource_info.json"
|
||||
|
||||
CURRENT_SYSTEM_PROMPT = (
|
||||
"You are a weather assistant. You ONLY answer questions about weather, "
|
||||
"climate, and atmospheric conditions (temperature, wind, humidity, UV index, "
|
||||
"sunrise, sunset, moon phase, forecasts, air quality, precipitation). "
|
||||
"If the user asks about anything unrelated to weather, politely redirect them. "
|
||||
"For example: 'I'm a weather assistant — I can help with forecasts, current conditions, "
|
||||
"UV index, wind, sunrise/sunset, and more. What location would you like weather for?' "
|
||||
"When answering weather questions: always search for real-time data using your tools, "
|
||||
"include specific numbers with units (temperature in F/C, wind in km/h or mph), "
|
||||
"mention the city name in your response, and keep responses concise and well-structured."
|
||||
)
|
||||
|
||||
# -- Clients -------------------------------------------------------------------
|
||||
dp_client = boto3.client("bedrock-agentcore", region_name=REGION)
|
||||
logs_client = boto3.client("logs", region_name=REGION)
|
||||
|
||||
|
||||
# -- Helpers -------------------------------------------------------------------
|
||||
def discover_log_group(harness_name: str) -> tuple[str, str] | None:
|
||||
"""Find the log group ARN and service name for a harness."""
|
||||
prefix = f"/aws/bedrock-agentcore/runtimes/harness_{harness_name}-"
|
||||
resp = logs_client.describe_log_groups(logGroupNamePrefix=prefix, limit=5)
|
||||
groups = resp.get("logGroups", [])
|
||||
if not groups:
|
||||
return None
|
||||
groups.sort(key=lambda g: g.get("creationTime", 0), reverse=True)
|
||||
log_group = groups[0]
|
||||
log_group_name = log_group["logGroupName"]
|
||||
log_group_arn = log_group["arn"]
|
||||
|
||||
# Service name: harness_{name}.DEFAULT (without random suffix)
|
||||
basename = log_group_name.split("/")[-1]
|
||||
parts = basename.rsplit("-", 2)
|
||||
service_name = f"{parts[0]}.DEFAULT" if len(parts) >= 3 else basename.replace("-DEFAULT", ".DEFAULT")
|
||||
|
||||
return log_group_arn, service_name
|
||||
|
||||
|
||||
def cleanup_recommendations():
|
||||
"""Delete all weather_rec_* recommendations."""
|
||||
try:
|
||||
resp = dp_client.list_recommendations()
|
||||
recs = resp.get("recommendationSummaries", resp.get("recommendations", resp.get("items", [])))
|
||||
count = 0
|
||||
for rec in recs:
|
||||
name = rec.get("name", "")
|
||||
rec_id = rec.get("recommendationId", "")
|
||||
if name.startswith("weather_rec_"):
|
||||
try:
|
||||
dp_client.delete_recommendation(recommendationId=rec_id)
|
||||
print(f" Deleted: {name}")
|
||||
count += 1
|
||||
except Exception as e:
|
||||
print(f" Warning: {e}")
|
||||
if count == 0:
|
||||
print(" No weather_rec_* recommendations found")
|
||||
except Exception as e:
|
||||
print(f" Error: {e}")
|
||||
|
||||
|
||||
# -- Main ----------------------------------------------------------------------
|
||||
def main():
|
||||
if args.cleanup:
|
||||
print("Cleaning up recommendations...")
|
||||
cleanup_recommendations()
|
||||
return
|
||||
|
||||
print("=" * 65)
|
||||
print("AgentCore Optimization — System Prompt Recommendation")
|
||||
print("=" * 65)
|
||||
|
||||
# Load state
|
||||
if not STATE_FILE.exists():
|
||||
print("\nError: resource_info.json not found.")
|
||||
print("Run ./start.sh first to create the weather agent, then use it for a few sessions.")
|
||||
sys.exit(1)
|
||||
|
||||
state = json.loads(STATE_FILE.read_text())
|
||||
harness_name = state.get("harness_name")
|
||||
if not harness_name:
|
||||
print("\nError: harness_name not found in resource_info.json")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"\n Harness: {harness_name}")
|
||||
print(f" Region: {REGION}")
|
||||
print(f" Evaluator: {args.evaluator}")
|
||||
print(f" Lookback: {args.lookback} day(s)")
|
||||
|
||||
# Discover log group
|
||||
print("\n Discovering log group...")
|
||||
result = discover_log_group(harness_name)
|
||||
if not result:
|
||||
print(" Error: Could not find log group for this harness.")
|
||||
print(" Make sure the web app has been running and you've sent some messages.")
|
||||
sys.exit(1)
|
||||
|
||||
log_group_arn, service_name = result
|
||||
print(f" Log group: {log_group_arn.split(':log-group:')[-1].rstrip(':*')}")
|
||||
print(f" Service: {service_name}")
|
||||
|
||||
# Time range
|
||||
now = datetime.now(timezone.utc)
|
||||
start_time = now - timedelta(days=args.lookback)
|
||||
|
||||
# Start recommendation
|
||||
rec_name = f"weather_rec_{uuid.uuid4().hex[:8]}"
|
||||
print(f"\n Starting recommendation: {rec_name}")
|
||||
print(f" Analyzing traces from {start_time.strftime('%Y-%m-%d %H:%M')} to {now.strftime('%Y-%m-%d %H:%M')} UTC")
|
||||
print(f" Optimizing for: {args.evaluator}")
|
||||
|
||||
try:
|
||||
resp = dp_client.start_recommendation(
|
||||
name=rec_name,
|
||||
type="SYSTEM_PROMPT_RECOMMENDATION",
|
||||
recommendationConfig={
|
||||
"systemPromptRecommendationConfig": {
|
||||
"systemPrompt": {
|
||||
"text": CURRENT_SYSTEM_PROMPT,
|
||||
},
|
||||
"agentTraces": {
|
||||
"cloudwatchLogs": {
|
||||
"logGroupArns": [log_group_arn],
|
||||
"serviceNames": [service_name],
|
||||
"startTime": start_time,
|
||||
"endTime": now,
|
||||
}
|
||||
},
|
||||
"evaluationConfig": {
|
||||
"evaluators": [
|
||||
{"evaluatorArn": f"arn:aws:bedrock-agentcore:::evaluator/{args.evaluator}"}
|
||||
]
|
||||
},
|
||||
}
|
||||
},
|
||||
clientToken=str(uuid.uuid4()),
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"\n Error starting recommendation: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
rec_id = resp["recommendationId"]
|
||||
print(f" Recommendation ID: {rec_id}")
|
||||
print(f" Status: {resp.get('status', 'PENDING')}")
|
||||
|
||||
# Poll for completion
|
||||
print("\n Waiting for recommendation to complete (typically 2-5 minutes)...")
|
||||
status = "PENDING"
|
||||
for i in range(60):
|
||||
time.sleep(10)
|
||||
try:
|
||||
result = dp_client.get_recommendation(recommendationId=rec_id)
|
||||
status = result.get("status", "UNKNOWN")
|
||||
if i % 3 == 0:
|
||||
print(f" [{i * 10}s] {status}")
|
||||
if status in ("COMPLETED", "FAILED"):
|
||||
break
|
||||
except Exception as e:
|
||||
print(f" Error polling: {e}")
|
||||
|
||||
if status != "COMPLETED":
|
||||
print(f"\n Recommendation did not complete (status: {status})")
|
||||
if status == "FAILED":
|
||||
error_msg = result.get("recommendationResult", {}).get(
|
||||
"systemPromptRecommendationResult", {}
|
||||
).get("errorMessage", "Unknown error")
|
||||
print(f" Error: {error_msg}")
|
||||
sys.exit(1)
|
||||
|
||||
# Extract result
|
||||
rec_result = result.get("recommendationResult", {}).get(
|
||||
"systemPromptRecommendationResult", {}
|
||||
)
|
||||
recommended_prompt = rec_result.get("recommendedSystemPrompt", "")
|
||||
explanation = rec_result.get("explanation", "")
|
||||
|
||||
# Display results
|
||||
print("\n" + "=" * 65)
|
||||
print("RECOMMENDATION RESULT")
|
||||
print("=" * 65)
|
||||
|
||||
print("\n--- Current System Prompt ---")
|
||||
print(CURRENT_SYSTEM_PROMPT[:300])
|
||||
if len(CURRENT_SYSTEM_PROMPT) > 300:
|
||||
print(f" ... ({len(CURRENT_SYSTEM_PROMPT)} chars total)")
|
||||
|
||||
print("\n--- Recommended System Prompt ---")
|
||||
print(recommended_prompt[:500])
|
||||
if len(recommended_prompt) > 500:
|
||||
print(f" ... ({len(recommended_prompt)} chars total)")
|
||||
|
||||
print("\n--- Explanation ---")
|
||||
print(explanation[:500])
|
||||
|
||||
# Save result
|
||||
output_file = Path(__file__).parent / "optimization_result.json"
|
||||
output_data = {
|
||||
"recommendation_id": rec_id,
|
||||
"recommendation_name": rec_name,
|
||||
"evaluator": args.evaluator,
|
||||
"current_system_prompt": CURRENT_SYSTEM_PROMPT,
|
||||
"recommended_system_prompt": recommended_prompt,
|
||||
"explanation": explanation,
|
||||
"timestamp": now.isoformat(),
|
||||
}
|
||||
output_file.write_text(json.dumps(output_data, indent=2))
|
||||
print(f"\n Full result saved to: {output_file.name}")
|
||||
|
||||
print("\n" + "=" * 65)
|
||||
print("Next steps:")
|
||||
print(" 1. Review the recommended prompt above")
|
||||
print(" 2. Update backend/agent.py SYSTEM_PROMPT with the recommendation")
|
||||
print(" 3. Restart the app and compare agent behavior")
|
||||
print(" 4. Run a batch evaluation to measure improvement")
|
||||
print(f"\n View in console: Bedrock AgentCore > Optimizations > Recommendations")
|
||||
print("=" * 65)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Weather Agent — One-Command Runner
|
||||
#
|
||||
# Usage:
|
||||
# ./run.sh Full demo (gateway + guardrail + agent + observability + evals)
|
||||
# ./run.sh --fast Skip evals (no 90s wait)
|
||||
# ./run.sh --keep Keep AWS resources after demo (for console inspection)
|
||||
# ./run.sh --cleanup Delete any leftover resources from a previous --keep run
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${BLUE}"
|
||||
echo "============================================================"
|
||||
echo " Weather Agent — Harness + Gateway + Guardrails + Evals"
|
||||
echo "============================================================"
|
||||
echo -e "${NC}"
|
||||
|
||||
# ── Parse flags ──────────────────────────────────────────────────────────────
|
||||
EXTRA_FLAGS=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--fast) EXTRA_FLAGS="$EXTRA_FLAGS --skip-evals" ;;
|
||||
--keep) EXTRA_FLAGS="$EXTRA_FLAGS --skip-cleanup" ;;
|
||||
--cleanup) EXTRA_FLAGS="--cleanup-only" ;;
|
||||
--help|-h)
|
||||
echo "Usage: ./run.sh [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --fast Skip evaluations (saves ~90 seconds)"
|
||||
echo " --keep Keep AWS resources after demo (inspect in console)"
|
||||
echo " --cleanup Delete leftover resources from a previous --keep run"
|
||||
echo " --help Show this help"
|
||||
echo ""
|
||||
echo "Prerequisites:"
|
||||
echo " - AWS CLI configured (aws sts get-caller-identity should work)"
|
||||
echo " - AWS_DEFAULT_REGION set (or defaults to us-east-1)"
|
||||
echo " - Claude Haiku 4.5 model access enabled in Bedrock console"
|
||||
echo " - CloudWatch Transaction Search enabled (for observability)"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Step 1: Check prerequisites ──────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[1/4] Checking prerequisites...${NC}"
|
||||
|
||||
# Python 3
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
echo -e "${RED} Python 3 is required but not found.${NC}"
|
||||
echo " Install: https://www.python.org/downloads/"
|
||||
exit 1
|
||||
fi
|
||||
PYTHON_VERSION=$(python3 --version 2>&1 | awk '{print $2}')
|
||||
echo -e " ${GREEN}Python:${NC} $PYTHON_VERSION"
|
||||
|
||||
# AWS CLI
|
||||
if ! command -v aws &> /dev/null; then
|
||||
echo -e "${RED} AWS CLI is required but not found.${NC}"
|
||||
echo " Install: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html"
|
||||
exit 1
|
||||
fi
|
||||
echo -e " ${GREEN}AWS CLI:${NC} $(aws --version 2>&1 | awk '{print $1}')"
|
||||
|
||||
# AWS credentials
|
||||
if ! aws sts get-caller-identity &> /dev/null; then
|
||||
echo -e "${RED} AWS credentials not configured or expired.${NC}"
|
||||
echo " Run: aws configure"
|
||||
exit 1
|
||||
fi
|
||||
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
|
||||
CALLER_ARN=$(aws sts get-caller-identity --query Arn --output text)
|
||||
echo -e " ${GREEN}Account:${NC} $ACCOUNT_ID"
|
||||
echo -e " ${GREEN}Identity:${NC} ${CALLER_ARN##*/}"
|
||||
|
||||
# Region
|
||||
REGION="${AWS_DEFAULT_REGION:-us-east-1}"
|
||||
export AWS_DEFAULT_REGION="$REGION"
|
||||
echo -e " ${GREEN}Region:${NC} $REGION"
|
||||
|
||||
echo ""
|
||||
|
||||
# ── Step 2: Install dependencies ─────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[2/4] Installing dependencies...${NC}"
|
||||
|
||||
HARNESS_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
if [ ! -d "$SCRIPT_DIR/venv" ]; then
|
||||
echo " Creating virtual environment..."
|
||||
python3 -m venv "$SCRIPT_DIR/venv"
|
||||
fi
|
||||
|
||||
source "$SCRIPT_DIR/venv/bin/activate"
|
||||
|
||||
# Install quietly, only show errors
|
||||
echo " Installing Python packages..."
|
||||
pip install --upgrade pip -q 2>&1 | grep -i error || true
|
||||
pip install -r "$HARNESS_ROOT/requirements.txt" -q 2>&1 | grep -i error || true
|
||||
pip install -r "$SCRIPT_DIR/backend/requirements.txt" -q 2>&1 | grep -i error || true
|
||||
|
||||
echo -e " ${GREEN}Dependencies ready${NC}"
|
||||
echo ""
|
||||
|
||||
# ── Step 3: Verify Bedrock model access ──────────────────────────────────────
|
||||
echo -e "${YELLOW}[3/4] Verifying Bedrock model access...${NC}"
|
||||
|
||||
python3 -c "
|
||||
import boto3, sys
|
||||
bedrock = boto3.client('bedrock', region_name='$REGION')
|
||||
try:
|
||||
resp = bedrock.get_foundation_model(modelIdentifier='anthropic.claude-haiku-4-5-20251001-v1:0')
|
||||
status = resp['modelDetails'].get('modelLifecycle', {}).get('status', 'ACTIVE')
|
||||
print(f' Claude Haiku 4.5: {status}')
|
||||
except Exception as e:
|
||||
if 'AccessDenied' in str(e) or 'ValidationException' in str(e):
|
||||
print(' Claude Haiku 4.5: access check inconclusive (may still work via inference profile)')
|
||||
else:
|
||||
print(f' Warning: {e}')
|
||||
" 2>&1
|
||||
|
||||
echo ""
|
||||
|
||||
# ── Step 4: Run the weather agent ────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[4/4] Running Weather Agent...${NC}"
|
||||
echo ""
|
||||
|
||||
python3 "$SCRIPT_DIR/weather_agent.py" $EXTRA_FLAGS
|
||||
|
||||
deactivate 2>/dev/null || true
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}============================================================${NC}"
|
||||
echo -e "${GREEN} Done!${NC}"
|
||||
echo -e "${GREEN}============================================================${NC}"
|
||||
@@ -0,0 +1,172 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Weather Agent — One-Command Start
|
||||
# Sets up everything and starts the web app.
|
||||
#
|
||||
# Usage:
|
||||
# ./start.sh Start the full app (provisions AWS resources on first run)
|
||||
# ./start.sh --help Show options
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${BLUE}"
|
||||
echo "=================================================================================="
|
||||
echo " Weather Agent — Harness + Gateway + Guardrails + Skills + Evals + Optimization"
|
||||
echo "=================================================================================="
|
||||
echo -e "${NC}"
|
||||
|
||||
# ── Cleanup on exit ──────────────────────────────────────────────────────────
|
||||
cleanup() {
|
||||
echo ""
|
||||
echo -e "${YELLOW}Stopping servers...${NC}"
|
||||
[ -f backend.pid ] && kill "$(cat backend.pid)" 2>/dev/null && rm -f backend.pid
|
||||
[ -f frontend.pid ] && kill "$(cat frontend.pid)" 2>/dev/null && rm -f frontend.pid
|
||||
lsof -ti:8000 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
lsof -ti:5173 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
echo -e "${GREEN}Stopped.${NC}"
|
||||
echo ""
|
||||
echo " To resume the app: ./start.sh (reuses existing AWS resources)"
|
||||
echo " To delete AWS resources: ./cleanup.sh"
|
||||
exit 0
|
||||
}
|
||||
trap cleanup SIGINT SIGTERM
|
||||
|
||||
# ── Step 1: Check prerequisites ──────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[1/5] Checking prerequisites...${NC}"
|
||||
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
echo -e "${RED} Python 3 is required. Install: https://www.python.org/downloads/${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo -e " ${GREEN}Python:${NC} $(python3 --version 2>&1 | awk '{print $2}')"
|
||||
|
||||
if ! command -v node &> /dev/null; then
|
||||
echo -e "${RED} Node.js is required. Install: https://nodejs.org/${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo -e " ${GREEN}Node.js:${NC} $(node --version)"
|
||||
|
||||
if ! command -v aws &> /dev/null; then
|
||||
echo -e "${RED} AWS CLI is required.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! aws sts get-caller-identity &> /dev/null; then
|
||||
echo -e "${RED} AWS credentials not configured or expired.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
|
||||
CALLER_ARN=$(aws sts get-caller-identity --query Arn --output text)
|
||||
echo -e " ${GREEN}AWS Account:${NC} $ACCOUNT_ID"
|
||||
echo -e " ${GREEN}Identity:${NC} ${CALLER_ARN##*/}"
|
||||
|
||||
REGION="${AWS_DEFAULT_REGION:-$(aws configure get region 2>/dev/null || echo 'us-east-1')}"
|
||||
export AWS_DEFAULT_REGION="$REGION"
|
||||
echo -e " ${GREEN}Region:${NC} $REGION"
|
||||
echo ""
|
||||
|
||||
# ── Step 2: Python virtual environment + deps ────────────────────────────────
|
||||
echo -e "${YELLOW}[2/5] Setting up Python environment...${NC}"
|
||||
|
||||
if [ ! -d "venv" ]; then
|
||||
python3 -m venv venv
|
||||
echo " Created virtual environment"
|
||||
fi
|
||||
|
||||
source venv/bin/activate
|
||||
pip install --upgrade pip -q 2>&1 | grep -i error || true
|
||||
pip install --upgrade -r backend/requirements.txt -q 2>&1 | grep -i error || true
|
||||
BOTO_VERSION=$(python3 -c "import boto3; print(boto3.__version__)" 2>/dev/null)
|
||||
echo -e " ${GREEN}Python dependencies ready${NC} (boto3: $BOTO_VERSION)"
|
||||
echo ""
|
||||
|
||||
# ── Step 3: Frontend dependencies ────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[3/5] Setting up frontend...${NC}"
|
||||
|
||||
if [ ! -d "frontend/node_modules" ]; then
|
||||
cd frontend
|
||||
npm install --silent 2>&1 | tail -1
|
||||
cd ..
|
||||
echo " Installed Node.js packages"
|
||||
else
|
||||
echo " Node.js packages already installed"
|
||||
fi
|
||||
echo -e " ${GREEN}Frontend ready${NC}"
|
||||
echo ""
|
||||
|
||||
# ── Step 4: Start backend ────────────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[4/5] Starting backend (provisions AWS resources on first run)...${NC}"
|
||||
|
||||
lsof -ti:8000 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
(
|
||||
cd backend
|
||||
python3 main.py 2>&1 | tee ../backend.log &
|
||||
echo $! > ../backend.pid
|
||||
)
|
||||
|
||||
# Wait for backend to be ready
|
||||
echo " Waiting for backend (this includes AWS resource provisioning, may take 3-5 minutes)..."
|
||||
MAX_WAIT=360
|
||||
ELAPSED=0
|
||||
while [ $ELAPSED -lt $MAX_WAIT ]; do
|
||||
if curl -s http://localhost:8000/health > /dev/null 2>&1; then
|
||||
echo -e " ${GREEN}Backend ready${NC}"
|
||||
break
|
||||
fi
|
||||
sleep 3
|
||||
ELAPSED=$((ELAPSED + 3))
|
||||
if [ $((ELAPSED % 15)) -eq 0 ]; then
|
||||
echo " Still provisioning... (${ELAPSED}s)"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ELAPSED -ge $MAX_WAIT ]; then
|
||||
echo -e "${RED} Backend failed to start. Check backend.log${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ── Step 5: Start frontend ───────────────────────────────────────────────────
|
||||
echo -e "${YELLOW}[5/5] Starting frontend...${NC}"
|
||||
|
||||
lsof -ti:5173 2>/dev/null | xargs kill -9 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
(
|
||||
cd frontend
|
||||
npm run dev 2>&1 | tee ../frontend.log &
|
||||
echo $! > ../frontend.pid
|
||||
)
|
||||
|
||||
sleep 3
|
||||
echo -e " ${GREEN}Frontend ready${NC}"
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}============================================================${NC}"
|
||||
echo -e "${GREEN} App is running!${NC}"
|
||||
echo -e "${GREEN}============================================================${NC}"
|
||||
echo ""
|
||||
echo -e " ${BLUE}Open:${NC} http://localhost:5173"
|
||||
echo ""
|
||||
echo " Logs:"
|
||||
echo " Backend: tail -f backend.log"
|
||||
echo " Frontend: tail -f frontend.log"
|
||||
echo ""
|
||||
echo " Press Ctrl+C to stop servers"
|
||||
echo " Run ./cleanup.sh to delete AWS resources"
|
||||
echo ""
|
||||
|
||||
# Wait
|
||||
while true; do sleep 1; done
|
||||
@@ -0,0 +1,513 @@
|
||||
"""
|
||||
Weather Agent — AgentCore Harness with Evals, Gateway & Observability.
|
||||
|
||||
An end-to-end use case demonstrating four AgentCore pillars through a weather
|
||||
assistant that provides current conditions, UV index, wind, and sun/moon data:
|
||||
|
||||
Part 1: Create Gateway + Harness (infrastructure)
|
||||
Part 2: Attach Bedrock Guardrail (PII anonymization)
|
||||
Part 3: Invoke agent — multi-turn weather session via Gateway tools
|
||||
Part 4: Observability — query CloudWatch X-Ray traces
|
||||
Part 5: Evaluations — on-demand scoring with built-in + custom evaluators
|
||||
Part 6: Cleanup
|
||||
|
||||
The Gateway proxies to Open-Meteo (free weather API, no key required) via
|
||||
an MCP target, giving the agent access to real-time weather data with
|
||||
centralized auth and observability on the tool traffic.
|
||||
|
||||
Usage:
|
||||
python weather_agent.py
|
||||
|
||||
# Skip evaluations (faster, no 90s wait for span ingestion)
|
||||
python weather_agent.py --skip-evals
|
||||
|
||||
# Skip guardrail creation (use existing or run without)
|
||||
python weather_agent.py --skip-guardrail
|
||||
|
||||
# Keep resources after demo
|
||||
python weather_agent.py --skip-cleanup
|
||||
|
||||
Prerequisites:
|
||||
- AWS CLI configured with credentials
|
||||
- pip install -r ../../requirements.txt
|
||||
- AWS_DEFAULT_REGION environment variable set
|
||||
- CloudWatch Transaction Search enabled (for observability)
|
||||
- Model access enabled for Claude Haiku 4.5 in Amazon Bedrock
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from utils.iam import create_harness_role, delete_harness_role
|
||||
from utils.client import get_agentcore_control_client, get_agentcore_client
|
||||
|
||||
# -- CLI -----------------------------------------------------------------------
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Weather Agent — Harness + Evals + Gateway + Observability"
|
||||
)
|
||||
parser.add_argument("--skip-evals", action="store_true", help="Skip evaluation step")
|
||||
parser.add_argument("--skip-guardrail", action="store_true", help="Skip guardrail creation")
|
||||
parser.add_argument("--skip-cleanup", action="store_true", help="Keep resources after demo")
|
||||
args = parser.parse_args()
|
||||
|
||||
# -- Configuration -------------------------------------------------------------
|
||||
MODEL_ID = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
REGION = boto3.session.Session().region_name or "us-east-1"
|
||||
ACCOUNT_ID = boto3.client("sts").get_caller_identity()["Account"]
|
||||
|
||||
# -- Clients -------------------------------------------------------------------
|
||||
control = get_agentcore_control_client()
|
||||
client = get_agentcore_client()
|
||||
bedrock = boto3.client("bedrock", region_name=REGION)
|
||||
|
||||
# -- Helpers -------------------------------------------------------------------
|
||||
|
||||
|
||||
def poll_status(get_fn, extract_fn, target="READY", timeout=120, interval=5):
|
||||
"""Poll a resource until it reaches target status or times out."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
resp = get_fn()
|
||||
status = extract_fn(resp)
|
||||
print(f" Status: {status}")
|
||||
if status == target:
|
||||
return resp
|
||||
if status in ("FAILED", "CREATE_FAILED", "DELETE_FAILED"):
|
||||
raise RuntimeError(f"Resource failed: {status}")
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError(f"Resource not {target} after {timeout}s")
|
||||
time.sleep(interval)
|
||||
|
||||
|
||||
def stream_response(harness_arn, session_id, message, tools=None):
|
||||
"""Invoke harness and stream the response. Returns accumulated text."""
|
||||
kwargs = dict(
|
||||
harnessArn=harness_arn,
|
||||
runtimeSessionId=session_id,
|
||||
messages=[{"role": "user", "content": [{"text": message}]}],
|
||||
model={"bedrockModelConfig": {"modelId": MODEL_ID}},
|
||||
)
|
||||
if tools:
|
||||
kwargs["tools"] = tools
|
||||
|
||||
response = client.invoke_harness(**kwargs)
|
||||
full_text = ""
|
||||
for event in response["stream"]:
|
||||
if "contentBlockStart" in event:
|
||||
start = event["contentBlockStart"].get("start", {})
|
||||
if "toolUse" in start:
|
||||
print(f"\n [Tool: {start['toolUse'].get('name', '?')}]", flush=True)
|
||||
elif "contentBlockDelta" in event:
|
||||
delta = event["contentBlockDelta"].get("delta", {})
|
||||
if "text" in delta:
|
||||
print(delta["text"], end="", flush=True)
|
||||
full_text += delta["text"]
|
||||
elif "messageStop" in event:
|
||||
print()
|
||||
elif "internalServerException" in event:
|
||||
print(f"\n Error: {event['internalServerException']}")
|
||||
return full_text
|
||||
|
||||
|
||||
# -- Resource tracking ---------------------------------------------------------
|
||||
harness_id = None
|
||||
gateway_id = None
|
||||
target_id = None
|
||||
guardrail_id = None
|
||||
eval_config_id = None
|
||||
|
||||
try:
|
||||
# ==========================================================================
|
||||
# Part 1: Create Gateway + Harness
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 1: Create Gateway + Harness")
|
||||
print("=" * 65)
|
||||
|
||||
# IAM role
|
||||
role_arn = create_harness_role()
|
||||
print(f" Role ARN: {role_arn}")
|
||||
print(" Waiting for IAM propagation...")
|
||||
time.sleep(10)
|
||||
|
||||
# Gateway — manages tool traffic with observability
|
||||
gateway_name = f"WeatherGateway-{uuid.uuid4().hex[:8]}"
|
||||
gw_control = boto3.client("bedrock-agentcore-control", region_name=REGION)
|
||||
|
||||
print(f"\n Creating Gateway: {gateway_name}")
|
||||
resp = gw_control.create_gateway(
|
||||
name=gateway_name,
|
||||
roleArn=role_arn,
|
||||
protocolType="MCP",
|
||||
authorizerType="NONE",
|
||||
)
|
||||
gateway_id = resp["gatewayId"]
|
||||
gateway_arn = resp["gatewayArn"]
|
||||
print(f" Gateway ID: {gateway_id}")
|
||||
print(f" Gateway ARN: {gateway_arn}")
|
||||
|
||||
poll_status(
|
||||
lambda: gw_control.get_gateway(gatewayIdentifier=gateway_id),
|
||||
lambda r: r["status"],
|
||||
)
|
||||
|
||||
# Add MCP target — Exa search for weather data
|
||||
print("\n Adding MCP target (Exa search)...")
|
||||
resp = gw_control.create_gateway_target(
|
||||
gatewayIdentifier=gateway_id,
|
||||
name="exa-weather-search",
|
||||
targetConfiguration={"mcp": {"mcpServer": {"endpoint": "https://mcp.exa.ai/mcp"}}},
|
||||
)
|
||||
target_id = resp["targetId"]
|
||||
print(f" Target ID: {target_id}")
|
||||
|
||||
poll_status(
|
||||
lambda: gw_control.get_gateway_target(
|
||||
gatewayIdentifier=gateway_id, targetId=target_id
|
||||
),
|
||||
lambda r: r["status"],
|
||||
)
|
||||
print(" Gateway ready with Exa MCP target")
|
||||
|
||||
# Harness — the managed agent runtime
|
||||
harness_name = f"WeatherAgent_{uuid.uuid4().hex[:8]}"
|
||||
print(f"\n Creating Harness: {harness_name}")
|
||||
resp = control.create_harness(harnessName=harness_name, executionRoleArn=role_arn)
|
||||
harness = resp["harness"]
|
||||
harness_id = harness["harnessId"]
|
||||
harness_arn = harness["arn"]
|
||||
print(f" Harness ID: {harness_id}")
|
||||
print(f" Harness ARN: {harness_arn}")
|
||||
|
||||
poll_status(
|
||||
lambda: control.get_harness(harnessId=harness_id),
|
||||
lambda r: r["harness"]["status"],
|
||||
)
|
||||
print(" Harness ready")
|
||||
|
||||
# ==========================================================================
|
||||
# Part 2: Attach Bedrock Guardrail
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 2: Attach Bedrock Guardrail (PII anonymization)")
|
||||
print("=" * 65)
|
||||
|
||||
if args.skip_guardrail:
|
||||
print(" Skipped (--skip-guardrail)")
|
||||
else:
|
||||
print(" Creating guardrail with PII filters...")
|
||||
gr_resp = bedrock.create_guardrail(
|
||||
name=f"weather-pii-guard-{uuid.uuid4().hex[:6]}",
|
||||
description="Anonymize PII in weather agent interactions",
|
||||
sensitiveInformationPolicyConfig={
|
||||
"piiEntitiesConfig": [
|
||||
{"type": "EMAIL", "action": "ANONYMIZE"},
|
||||
{"type": "PHONE", "action": "ANONYMIZE"},
|
||||
{"type": "US_SOCIAL_SECURITY_NUMBER", "action": "ANONYMIZE"},
|
||||
{"type": "CREDIT_DEBIT_CARD_NUMBER", "action": "ANONYMIZE"},
|
||||
{"type": "ADDRESS", "action": "ANONYMIZE"},
|
||||
]
|
||||
},
|
||||
blockedInputMessaging="Your message contains restricted content.",
|
||||
blockedOutputsMessaging="The response contains restricted content.",
|
||||
)
|
||||
guardrail_id = gr_resp["guardrailId"]
|
||||
guardrail_version_resp = bedrock.create_guardrail_version(
|
||||
guardrailIdentifier=guardrail_id,
|
||||
description="v1",
|
||||
)
|
||||
guardrail_version = guardrail_version_resp["version"]
|
||||
print(f" Guardrail ID: {guardrail_id} (version {guardrail_version})")
|
||||
print(" PII filters: EMAIL, PHONE, SSN, CREDIT_CARD, ADDRESS")
|
||||
print(" Guardrail ready — PII in agent responses will be anonymized")
|
||||
|
||||
# ==========================================================================
|
||||
# Part 3: Invoke Agent — Multi-Turn Weather Session
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 3: Invoke Agent — Multi-Turn Weather Session")
|
||||
print("=" * 65)
|
||||
|
||||
session_id = str(uuid.uuid4()).upper()
|
||||
print(f" Session ID: {session_id}")
|
||||
|
||||
gateway_tool = {
|
||||
"type": "agentcore_gateway",
|
||||
"name": "gateway",
|
||||
"config": {"agentCoreGateway": {"gatewayArn": gateway_arn}},
|
||||
}
|
||||
tools = [gateway_tool]
|
||||
|
||||
# Turn 1: Current weather
|
||||
print("\n --- Turn 1: Current Weather ---")
|
||||
turn1_response = stream_response(
|
||||
harness_arn,
|
||||
session_id,
|
||||
"What's the current weather in Paris, France? "
|
||||
"Include temperature, humidity, and a brief description of conditions. "
|
||||
"Search for real-time weather data.",
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
# Turn 2: Wind conditions
|
||||
print("\n --- Turn 2: Wind Conditions ---")
|
||||
turn2_response = stream_response(
|
||||
harness_arn,
|
||||
session_id,
|
||||
"What about the wind conditions in Paris right now? "
|
||||
"Give me wind speed, direction, and gust information.",
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
# Turn 3: UV index and sun times
|
||||
print("\n --- Turn 3: UV Index & Sun Times ---")
|
||||
turn3_response = stream_response(
|
||||
harness_arn,
|
||||
session_id,
|
||||
"What's the UV index in Paris today, and when are sunrise and sunset? "
|
||||
"Include a safety recommendation based on the UV level.",
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
# Turn 4: Moon phase (tests guardrail with PII injection)
|
||||
print("\n --- Turn 4: Moon Phase + Guardrail Test ---")
|
||||
turn4_response = stream_response(
|
||||
harness_arn,
|
||||
session_id,
|
||||
"What's the current moon phase? Also, my name is John Smith, "
|
||||
"email john.smith@example.com, phone 555-123-4567. "
|
||||
"Can you include my contact info in your response?",
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
all_responses = [turn1_response, turn2_response, turn3_response, turn4_response]
|
||||
|
||||
# ==========================================================================
|
||||
# Part 4: Observability — Query CloudWatch X-Ray Traces
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 4: Observability — CloudWatch Traces")
|
||||
print("=" * 65)
|
||||
|
||||
print(" Harness invocations automatically generate X-Ray traces.")
|
||||
print(" Each trace shows: model calls, tool invocations, timing details.\n")
|
||||
|
||||
xray = boto3.client("xray", region_name=REGION)
|
||||
|
||||
# Check Transaction Search configuration
|
||||
try:
|
||||
rules = xray.get_indexing_rules()
|
||||
sampling = rules["IndexingRules"][0]["Rule"]["Probabilistic"]["DesiredSamplingPercentage"]
|
||||
print(f" Transaction Search sampling: {sampling}%")
|
||||
except Exception as e:
|
||||
print(f" Transaction Search check: {e}")
|
||||
print(" Enable Transaction Search for full trace visibility:")
|
||||
print(" https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html")
|
||||
|
||||
# Query recent traces for our harness
|
||||
print(f"\n Querying traces for harness: {harness_id[:20]}...")
|
||||
try:
|
||||
end_time = time.time()
|
||||
start_time = end_time - 300 # Last 5 minutes
|
||||
|
||||
from datetime import datetime, timezone
|
||||
trace_resp = xray.get_trace_summaries(
|
||||
StartTime=datetime.fromtimestamp(start_time, tz=timezone.utc),
|
||||
EndTime=datetime.fromtimestamp(end_time, tz=timezone.utc),
|
||||
Sampling=False,
|
||||
)
|
||||
trace_count = len(trace_resp.get("TraceSummaries", []))
|
||||
print(f" Found {trace_count} trace(s) in the last 5 minutes")
|
||||
|
||||
if trace_count > 0:
|
||||
for i, trace in enumerate(trace_resp["TraceSummaries"][:3], 1):
|
||||
duration = trace.get("Duration", 0)
|
||||
has_error = trace.get("HasError", False)
|
||||
status_icon = "x" if has_error else "ok"
|
||||
print(f" Trace {i}: duration={duration:.2f}s status={status_icon}")
|
||||
except Exception as e:
|
||||
print(f" Trace query: {e}")
|
||||
print(" (Traces may take 1-2 minutes to appear after invocation)")
|
||||
|
||||
print("\n View traces in AWS Console:")
|
||||
print(f" CloudWatch > X-Ray > Traces (region: {REGION})")
|
||||
print(" Filter by: service(bedrock-agentcore)")
|
||||
|
||||
# ==========================================================================
|
||||
# Part 5: Evaluations — Batch Evaluation
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 5: Evaluations — Batch Evaluation")
|
||||
print("=" * 65)
|
||||
|
||||
if args.skip_evals:
|
||||
print(" Skipped (--skip-evals)")
|
||||
else:
|
||||
print(" Waiting 60s for CloudWatch trace ingestion...")
|
||||
time.sleep(60)
|
||||
|
||||
# Discover the log group for this harness
|
||||
logs_client = boto3.client("logs", region_name=REGION)
|
||||
prefix = f"/aws/bedrock-agentcore/runtimes/harness_{harness_name}-"
|
||||
log_groups = logs_client.describe_log_groups(logGroupNamePrefix=prefix, limit=5)
|
||||
groups = log_groups.get("logGroups", [])
|
||||
|
||||
if not groups:
|
||||
print(" Could not find log group for harness — skipping evaluation")
|
||||
else:
|
||||
groups.sort(key=lambda g: g.get("creationTime", 0), reverse=True)
|
||||
log_group = groups[0]["logGroupName"]
|
||||
log_group_basename = log_group.split("/")[-1]
|
||||
parts = log_group_basename.rsplit("-", 2)
|
||||
service_name = f"{parts[0]}.DEFAULT" if len(parts) >= 3 else log_group_basename.replace("-DEFAULT", ".DEFAULT")
|
||||
|
||||
print(f" Log group: {log_group}")
|
||||
print(f" Service: {service_name}")
|
||||
|
||||
batch_name = f"weather_eval_{uuid.uuid4().hex[:8]}"
|
||||
evaluator_ids = [
|
||||
"Builtin.InstructionFollowing",
|
||||
"Builtin.Helpfulness",
|
||||
"Builtin.Correctness",
|
||||
"Builtin.Faithfulness",
|
||||
"Builtin.ResponseRelevance",
|
||||
"Builtin.Coherence",
|
||||
"Builtin.Conciseness",
|
||||
"Builtin.Refusal",
|
||||
]
|
||||
|
||||
print(f"\n Starting batch evaluation: {batch_name}")
|
||||
try:
|
||||
resp = client.start_batch_evaluation(
|
||||
batchEvaluationName=batch_name,
|
||||
evaluators=[{"evaluatorId": eid} for eid in evaluator_ids],
|
||||
dataSourceConfig={
|
||||
"cloudWatchLogs": {
|
||||
"serviceNames": [service_name],
|
||||
"logGroupNames": [log_group],
|
||||
"filterConfig": {
|
||||
"sessionIds": [session_id],
|
||||
},
|
||||
}
|
||||
},
|
||||
)
|
||||
batch_id = resp["batchEvaluationId"]
|
||||
print(f" Batch ID: {batch_id}")
|
||||
|
||||
# Poll until complete
|
||||
print(" Polling for results...")
|
||||
for _ in range(30):
|
||||
time.sleep(10)
|
||||
result = client.get_batch_evaluation(batchEvaluationId=batch_id)
|
||||
status = result.get("status", "UNKNOWN")
|
||||
print(f" Status: {status}")
|
||||
if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED"):
|
||||
break
|
||||
|
||||
if status == "COMPLETED":
|
||||
eval_results = result.get("evaluationResults", {})
|
||||
summaries = eval_results.get("evaluatorSummaries", [])
|
||||
print(f"\n Evaluation Results ({len(summaries)} evaluator(s)):")
|
||||
print(f" {'Evaluator':<30} {'Score':<8}")
|
||||
print(" " + "-" * 50)
|
||||
for s in summaries:
|
||||
eid = s.get("evaluatorId", "").replace("Builtin.", "")
|
||||
stats = s.get("statistics", {})
|
||||
avg = stats.get("averageScore")
|
||||
score_str = f"{avg:.2f}" if avg is not None else "N/A"
|
||||
print(f" {eid:<30} {score_str}")
|
||||
else:
|
||||
print(f" Evaluation ended with status: {status}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" Evaluation error: {e}")
|
||||
|
||||
# ==========================================================================
|
||||
# Summary
|
||||
# ==========================================================================
|
||||
print("\n" + "=" * 65)
|
||||
print("Summary")
|
||||
print("=" * 65)
|
||||
print(f" Harness: {harness_id}")
|
||||
print(f" Gateway: {gateway_id} (Exa MCP target)")
|
||||
if guardrail_id:
|
||||
print(f" Guardrail: {guardrail_id} (PII anonymization)")
|
||||
print(f" Session: {session_id}")
|
||||
print(" Turns: 4 (weather, wind, UV/sun, moon+PII test)")
|
||||
print(f" Observability: CloudWatch X-Ray traces (region: {REGION})")
|
||||
if not args.skip_evals:
|
||||
print(" Evaluations: Built-in batch evaluators")
|
||||
print()
|
||||
print(" View traces: CloudWatch > X-Ray > Traces")
|
||||
print(" Filter: service(bedrock-agentcore)")
|
||||
|
||||
finally:
|
||||
# ==========================================================================
|
||||
# Part 6: Cleanup
|
||||
# ==========================================================================
|
||||
if not args.skip_cleanup:
|
||||
print("\n" + "=" * 65)
|
||||
print("Part 6: Cleanup")
|
||||
print("=" * 65)
|
||||
|
||||
if harness_id:
|
||||
try:
|
||||
control.delete_harness(harnessId=harness_id)
|
||||
print(f" Deleted harness: {harness_id}")
|
||||
except Exception as e:
|
||||
print(f" Warning (harness): {e}")
|
||||
|
||||
if gateway_id and target_id:
|
||||
try:
|
||||
gw_control.delete_gateway_target(
|
||||
gatewayIdentifier=gateway_id, targetId=target_id
|
||||
)
|
||||
print(f" Deleted target: {target_id}")
|
||||
time.sleep(10)
|
||||
except Exception as e:
|
||||
print(f" Warning (target): {e}")
|
||||
|
||||
if gateway_id:
|
||||
try:
|
||||
gw_control.delete_gateway(gatewayIdentifier=gateway_id)
|
||||
print(f" Deleted gateway: {gateway_id}")
|
||||
except Exception as e:
|
||||
print(f" Warning (gateway): {e}")
|
||||
|
||||
if guardrail_id:
|
||||
try:
|
||||
bedrock.delete_guardrail(guardrailIdentifier=guardrail_id)
|
||||
print(f" Deleted guardrail: {guardrail_id}")
|
||||
except Exception as e:
|
||||
print(f" Warning (guardrail): {e}")
|
||||
|
||||
# Delete batch evaluations created by this run
|
||||
try:
|
||||
evals = client.list_batch_evaluations()
|
||||
for ev in evals.get("batchEvaluations", evals.get("items", [])):
|
||||
ev_name = ev.get("batchEvaluationName", ev.get("name", ""))
|
||||
ev_id = ev.get("batchEvaluationId", "")
|
||||
if ev_name.startswith("weather_eval_"):
|
||||
try:
|
||||
client.delete_batch_evaluation(batchEvaluationId=ev_id)
|
||||
print(f" Deleted batch evaluation: {ev_name}")
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f" Warning (batch evals): {e}")
|
||||
|
||||
delete_harness_role()
|
||||
print(" Done.")
|
||||
else:
|
||||
print("\n=== Skipping cleanup (--skip-cleanup) ===")
|
||||
print(f" Harness ID: {harness_id}")
|
||||
print(f" Gateway ID: {gateway_id}")
|
||||
if guardrail_id:
|
||||
print(f" Guardrail: {guardrail_id}")
|
||||
@@ -127,6 +127,9 @@ python 02-use-cases/01-travel-agent/travel_agent.py
|
||||
|
||||
# Webapp visual testing
|
||||
python 02-use-cases/02-webapp-visual-testing/webapp_visual_testing.py
|
||||
|
||||
# Weather agent (gateway + guardrails + evals + observability)
|
||||
python 02-use-cases/04-weather-agent/weather_agent.py
|
||||
```
|
||||
|
||||
Run all tests:
|
||||
|
||||
Reference in New Issue
Block a user