1
0
mirror of synced 2026-08-05 11:07:15 +00:00

feat(harness/14-s3-filesystem): optional script to provision the S3 Files prerequisites (#1881)

Both scripts in this sample mount an existing S3 Files access point, and nothing
in the folder creates one — so on a fresh account --access-point-arn,
--subnet-ids and --security-group-ids have nothing to point at and the sample
cannot be run at all.

Adds an optional provision_s3_filesystem.py that creates the prerequisites
(versioned bucket, service role, file system, access point, a mount target per
subnet, and an NFS security group) and prints the exact command line to paste
into either script. The sample scripts themselves are unchanged.

Networking is bring-your-own by default: the script discovers private subnets
that already have NAT-gateway egress rather than creating a VPC, because a NAT
gateway bills hourly. --create-vpc is opt-in.

Every resource is recorded in provision_state.json as it is created, and
--teardown deletes only what is recorded there.
This commit is contained in:
Rui Cardoso
2026-08-03 13:18:21 +01:00
committed by GitHub
parent f67f2c041f
commit ee31c1790e
3 changed files with 1240 additions and 1 deletions
@@ -0,0 +1,3 @@
# Written by provision_s3_filesystem.py; records the resources it created so
# --teardown knows what to delete. Account-specific, never committed.
provision_state.json
@@ -23,9 +23,11 @@ sessions.
|---|---|
| [`s3_filesystem.py`](s3_filesystem.py) | **The mechanism.** Session A writes a file under the mount; Session B (a brand-new microVM) reads it back — only possible because the file lives in S3, not on the VM disk. |
| [`s3_llm_wiki.py`](s3_llm_wiki.py) | **The use case: a persistent LLM wiki.** The agent builds and maintains a compounding markdown wiki on the S3 mount across sessions (ingest → query → lint). |
| [`provision_s3_filesystem.py`](provision_s3_filesystem.py) | **Optional setup.** Creates the prerequisites below (bucket, file system, access point, mount targets) and prints the command line to paste into either script. `--teardown` removes exactly what it created. |
The first script proves the persistence boundary; the second shows *why you'd
want it*.
want it*. Both mount an **existing** access point — if you don't have one yet,
[`provision_s3_filesystem.py`](provision_s3_filesystem.py) will make one.
## Configuration
@@ -107,6 +109,61 @@ the wait doesn't read as a hang.
**IAM scope**: The mount permissions (`ClientMount`/`ClientWrite`) are scoped to the single access point with an `AccessPointArn` condition. The two the runtime checks at create time (`GetAccessPoint`, `ListMountTargets`) have to stay on `"*"`: `ListMountTargets` is authorized against the *file system* ID rather than the access point, so scoping it to the access point ARN denies it.
## Optional: provisioning the prerequisites
Both sample scripts mount an **existing** access point — they don't create
infrastructure, so on a fresh account there is nothing for `--access-point-arn`
to point at. [`provision_s3_filesystem.py`](provision_s3_filesystem.py) closes
that gap:
```bash
# Create the S3 Files layer and print the command line to run the sample with
python provision_s3_filesystem.py
# See what it would do first, without creating anything
python provision_s3_filesystem.py --dry-run
# Delete everything it created
python provision_s3_filesystem.py --teardown
```
It creates a bucket (versioning enabled), the IAM service role S3 Files assumes,
a file system and access point over that bucket, a mount target per subnet, and a
security group allowing NFS 2049. Each resource is written to
`provision_state.json` as it is created, and `--teardown` deletes **only** what is
recorded there — a bucket or VPC you brought yourself is never touched.
**Networking: bring your own by default.** The script does *not* create a VPC. It
looks for private subnets that already have NAT-gateway egress and uses those,
because a NAT gateway bills hourly whether or not you're using it and is the
resource people forget to delete. Pass `--create-vpc` only if the account has no
suitable subnets; it then builds the VPC, subnets, internet gateway and NAT
gateway too, and `--teardown` removes them.
| Flag | What it does |
|---|---|
| `--bucket NAME` | Reuse a bucket you already have. It must have versioning enabled, and it is **never** deleted on teardown. |
| `--prefix PREFIX` | Key prefix the file system is scoped to (default: `harness-sample/`). |
| `--subnet-ids` | Use these subnets instead of discovering private ones with NAT egress. |
| `--security-group-ids` | Use these security groups instead of creating one that allows NFS 2049. |
| `--create-vpc` | Also create a VPC, subnets and a **NAT gateway** (bills hourly). |
| `--dry-run` | Report what would be created, and what was discovered, without creating it. Combine with `--teardown` to preview a deletion. |
| `--teardown` | Delete everything in `provision_state.json`, then remove the file. |
Two things worth knowing about teardown:
- **Expect to run it twice.** AgentCore reclaims the harness microVM's network
interfaces on its own schedule *after* the harness is deleted — well over an
hour in testing — and the security group can't be deleted until they're
released. Rather than make you wait, the first run deletes everything that
bills, reports the security group as still held, and exits 0. Re-run it later
and it removes just what's left.
- It is **safe to re-run** as often as you like. Anything already gone is skipped,
and the state file shrinks as each resource is deleted.
`--teardown --dry-run` lists what would be deleted, and what it would leave alone,
without touching anything.
## Use case: a persistent LLM wiki
[`s3_llm_wiki.py`](s3_llm_wiki.py) turns the S3 mount into a