122c408fff
#3: Stored XSS in stats dashboard - escape p[path] with html.escape() #4: Caddy timeout race - increase read/write_timeout 30s -> 60s #5: Missing CSP header - add Content-Security-Policy to Caddyfile
120 lines
4.4 KiB
Markdown
120 lines
4.4 KiB
Markdown
# AGENTS — AO3 Mirror v5
|
||
|
||
This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.
|
||
|
||
## Project Overview
|
||
|
||
AO3 Mirror v5 is a high-performance reverse proxy for archiveofourown.org, redesigned from the ground up with sticky sessions, Service Worker client-side caching, and a full 5000-proxy P2C pool — matching the go3 architecture with modern Python tooling.
|
||
|
||
- **Domain**: agento3.miscs.dev (behind Cloudflare CDN)
|
||
- **Target**: archiveofourown.org
|
||
- **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
|
||
- **Proxy Pool**: 5000 WebShare proxies, P2C selection, state machine (healthy/unstable/blocked/probing)
|
||
- **Client**: Service Worker for static cache + offline fallback
|
||
- **Architecture Reference**: https://tomorin.cv/AO3-CN/go3 (go3 v2 branch)
|
||
|
||
## Architecture v5
|
||
|
||
```
|
||
User Browser
|
||
│ ao3_sessid_proxy cookie = sticky proxy
|
||
│
|
||
├─ Service Worker (sw.js)
|
||
│ ├─ Cache API — static assets cached in browser
|
||
│ ├─ Navigation intercept — offline mirror picker
|
||
│ └─ Request passthrough — for non-static, non-nav
|
||
│
|
||
└─ HTTPS → Caddy → 2× uvicorn workers (8081-8082)
|
||
│
|
||
├─ Proxy Pool (5000, P2C selection)
|
||
│ ├─ State machine: healthy/unstable/blocked/probing
|
||
│ ├─ Concurrency limit: asyncio.Semaphore(400)
|
||
│ └─ Passive probing: 10s interval, 10 concurrency
|
||
│
|
||
├─ Sticky Session (ao3_sessid_proxy)
|
||
│ └─ User pinned to one proxy for session duration
|
||
│
|
||
├─ Service Worker Deploy (/sw.js, /sw-YYYYMMDD.js)
|
||
│ └─ HTML injection: auto-register SW on page load
|
||
│
|
||
├─ CF Challenge → User Browser
|
||
│ └─ All proxies blocked → rewrite challenge → browser solves
|
||
│
|
||
├─ Static Cache (server-side LRU + disk MD5)
|
||
│
|
||
└─ Local routes: /stats, /health, /metrics
|
||
```
|
||
|
||
## Build Commands
|
||
|
||
```bash
|
||
# Start all workers + reload Caddy
|
||
cd /home/ubuntu/ao3-mirror && bash start.sh
|
||
|
||
# Stop everything
|
||
cd /home/ubuntu/ao3-mirror && bash stop.sh
|
||
|
||
# Restart daemon only
|
||
sudo systemctl restart ao3-daemon
|
||
|
||
# Restart Caddy only
|
||
sudo systemctl reload caddy
|
||
|
||
# Syntax check all Python files
|
||
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py
|
||
|
||
# Deploy Service Worker update
|
||
# 1. Edit static/sw.js
|
||
# 2. Update SERVICE_WORKER_DATE in app.py
|
||
# 3. Restart workers
|
||
```
|
||
|
||
### Core Files (v5)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `app.py` | FastAPI backend v5 — proxy handler, sticky sessions, SW deploy, stats |
|
||
| `proxy_pool.py` | P2C proxy pool v5 — 5000 proxies, state machine, semaphore limiter |
|
||
| `ao3_fetcher.py` | Single-request fetcher v5 — no internal retry, CF detection |
|
||
| `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) |
|
||
| `stats.py` | In-memory stats with batch SQLite flush every 60s |
|
||
| `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) |
|
||
| `static/sw.js` | Service Worker — client-side cache, offline fallback |
|
||
| `Caddyfile` | Caddy config — SW routes, round-robin, 60s timeouts |
|
||
| `scripts/daemon.py` | Passive worker supervision (systemd) |
|
||
| `start.sh` | Startup — kills old workers, starts new, reloads Caddy |
|
||
|
||
## Security Baseline
|
||
|
||
- No secrets in code — proxy credentials in `/home/ubuntu/proxy.txt`
|
||
- Workers bound to 127.0.0.1 only
|
||
- Cloudflare CDN terminates TLS at edge
|
||
- CORS headers restrict cross-origin access
|
||
- stats/health/metrics are read-only
|
||
|
||
## Engine Guidance
|
||
|
||
- Complex multi-file changes, architecture evolution → Hermes (here)
|
||
- Quick targeted fixes, single-file changes → Hermes
|
||
- Deploy, monitor, notify, schedule → Hermes
|
||
- Proxy pool refresh → Hermes cron (30min)
|
||
- Not sure? Start with Hermes — everything runs via Hermes
|
||
|
||
## Monitoring
|
||
|
||
- Health: https://agento3.miscs.dev/health
|
||
- Stats: https://agento3.miscs.dev/stats
|
||
- Metrics: https://agento3.miscs.dev/metrics
|
||
- SW: https://agento3.miscs.dev/sw.js → /sw-YYYYMMDD.js
|
||
|
||
## CTO Goal (Persistent Focus)
|
||
|
||
Manage akiba/agento3 as CTO. Triage issues hourly, implement top priority, get founder approval before merging. Never ship without YES.
|
||
|
||
## Commit Conventions
|
||
|
||
- One commit per meaningful change
|
||
- Python files only (no binaries, .pyc, logs, .env)
|
||
- SOUL.md updated for architectural changes
|
||
- ao3-mirror skill updated when workflows change
|