feat(harness/14-s3-filesystem): optional script to provision the S3 Files prerequisites (#1881)
Both scripts in this sample mount an existing S3 Files access point, and nothing in the folder creates one — so on a fresh account --access-point-arn, --subnet-ids and --security-group-ids have nothing to point at and the sample cannot be run at all. Adds an optional provision_s3_filesystem.py that creates the prerequisites (versioned bucket, service role, file system, access point, a mount target per subnet, and an NFS security group) and prints the exact command line to paste into either script. The sample scripts themselves are unchanged. Networking is bring-your-own by default: the script discovers private subnets that already have NAT-gateway egress rather than creating a VPC, because a NAT gateway bills hourly. --create-vpc is opt-in. Every resource is recorded in provision_state.json as it is created, and --teardown deletes only what is recorded there.
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
# Written by provision_s3_filesystem.py; records the resources it created so
|
||||
# --teardown knows what to delete. Account-specific, never committed.
|
||||
provision_state.json
|
||||
@@ -23,9 +23,11 @@ sessions.
|
||||
|---|---|
|
||||
| [`s3_filesystem.py`](s3_filesystem.py) | **The mechanism.** Session A writes a file under the mount; Session B (a brand-new microVM) reads it back — only possible because the file lives in S3, not on the VM disk. |
|
||||
| [`s3_llm_wiki.py`](s3_llm_wiki.py) | **The use case: a persistent LLM wiki.** The agent builds and maintains a compounding markdown wiki on the S3 mount across sessions (ingest → query → lint). |
|
||||
| [`provision_s3_filesystem.py`](provision_s3_filesystem.py) | **Optional setup.** Creates the prerequisites below (bucket, file system, access point, mount targets) and prints the command line to paste into either script. `--teardown` removes exactly what it created. |
|
||||
|
||||
The first script proves the persistence boundary; the second shows *why you'd
|
||||
want it*.
|
||||
want it*. Both mount an **existing** access point — if you don't have one yet,
|
||||
[`provision_s3_filesystem.py`](provision_s3_filesystem.py) will make one.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -107,6 +109,61 @@ the wait doesn't read as a hang.
|
||||
|
||||
**IAM scope**: The mount permissions (`ClientMount`/`ClientWrite`) are scoped to the single access point with an `AccessPointArn` condition. The two the runtime checks at create time (`GetAccessPoint`, `ListMountTargets`) have to stay on `"*"`: `ListMountTargets` is authorized against the *file system* ID rather than the access point, so scoping it to the access point ARN denies it.
|
||||
|
||||
## Optional: provisioning the prerequisites
|
||||
|
||||
Both sample scripts mount an **existing** access point — they don't create
|
||||
infrastructure, so on a fresh account there is nothing for `--access-point-arn`
|
||||
to point at. [`provision_s3_filesystem.py`](provision_s3_filesystem.py) closes
|
||||
that gap:
|
||||
|
||||
```bash
|
||||
# Create the S3 Files layer and print the command line to run the sample with
|
||||
python provision_s3_filesystem.py
|
||||
|
||||
# See what it would do first, without creating anything
|
||||
python provision_s3_filesystem.py --dry-run
|
||||
|
||||
# Delete everything it created
|
||||
python provision_s3_filesystem.py --teardown
|
||||
```
|
||||
|
||||
It creates a bucket (versioning enabled), the IAM service role S3 Files assumes,
|
||||
a file system and access point over that bucket, a mount target per subnet, and a
|
||||
security group allowing NFS 2049. Each resource is written to
|
||||
`provision_state.json` as it is created, and `--teardown` deletes **only** what is
|
||||
recorded there — a bucket or VPC you brought yourself is never touched.
|
||||
|
||||
**Networking: bring your own by default.** The script does *not* create a VPC. It
|
||||
looks for private subnets that already have NAT-gateway egress and uses those,
|
||||
because a NAT gateway bills hourly whether or not you're using it and is the
|
||||
resource people forget to delete. Pass `--create-vpc` only if the account has no
|
||||
suitable subnets; it then builds the VPC, subnets, internet gateway and NAT
|
||||
gateway too, and `--teardown` removes them.
|
||||
|
||||
| Flag | What it does |
|
||||
|---|---|
|
||||
| `--bucket NAME` | Reuse a bucket you already have. It must have versioning enabled, and it is **never** deleted on teardown. |
|
||||
| `--prefix PREFIX` | Key prefix the file system is scoped to (default: `harness-sample/`). |
|
||||
| `--subnet-ids` | Use these subnets instead of discovering private ones with NAT egress. |
|
||||
| `--security-group-ids` | Use these security groups instead of creating one that allows NFS 2049. |
|
||||
| `--create-vpc` | Also create a VPC, subnets and a **NAT gateway** (bills hourly). |
|
||||
| `--dry-run` | Report what would be created, and what was discovered, without creating it. Combine with `--teardown` to preview a deletion. |
|
||||
| `--teardown` | Delete everything in `provision_state.json`, then remove the file. |
|
||||
|
||||
Two things worth knowing about teardown:
|
||||
|
||||
- **Expect to run it twice.** AgentCore reclaims the harness microVM's network
|
||||
interfaces on its own schedule *after* the harness is deleted — well over an
|
||||
hour in testing — and the security group can't be deleted until they're
|
||||
released. Rather than make you wait, the first run deletes everything that
|
||||
bills, reports the security group as still held, and exits 0. Re-run it later
|
||||
and it removes just what's left.
|
||||
- It is **safe to re-run** as often as you like. Anything already gone is skipped,
|
||||
and the state file shrinks as each resource is deleted.
|
||||
|
||||
`--teardown --dry-run` lists what would be deleted, and what it would leave alone,
|
||||
without touching anything.
|
||||
|
||||
## Use case: a persistent LLM wiki
|
||||
|
||||
[`s3_llm_wiki.py`](s3_llm_wiki.py) turns the S3 mount into a
|
||||
|
||||
+1179
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user