111 lines
4.3 KiB
Markdown
111 lines
4.3 KiB
Markdown
|
# AGENTS — AO3 Mirror
|
|||
|
|
|
|||
|
|
This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.
|
|||
|
|
|
|||
|
|
## Project Overview
|
|||
|
|
|
|||
|
|
AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving.
|
|||
|
|
|
|||
|
|
- **Domain**: agento3.miscs.dev (behind Cloudflare CDN)
|
|||
|
|
- **Target**: archiveofourown.org
|
|||
|
|
- **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
|
|||
|
|
- **QPS**: 720 (cache) | 270 (direct proxy)
|
|||
|
|
|
|||
|
|
## Build Commands
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Start all workers + reload Caddy
|
|||
|
|
cd /home/ubuntu/ao3-mirror && bash start.sh
|
|||
|
|
|
|||
|
|
# Stop everything
|
|||
|
|
cd /home/ubuntu/ao3-mirror && bash stop.sh
|
|||
|
|
|
|||
|
|
# Restart daemon only
|
|||
|
|
sudo systemctl restart ao3-daemon
|
|||
|
|
|
|||
|
|
# Restart Caddy only
|
|||
|
|
sudo systemctl reload caddy
|
|||
|
|
|
|||
|
|
# Scan proxy pool (MUST run in foreground!)
|
|||
|
|
cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py
|
|||
|
|
|
|||
|
|
# Syntax check
|
|||
|
|
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Architecture
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin)
|
|||
|
|
→ 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop]
|
|||
|
|
→ Tiered Proxy Pool:
|
|||
|
|
Fast (50): POST/login → 8s timeout, 1 retry
|
|||
|
|
Main (676): GET/browse → 15s timeout, 2 retries
|
|||
|
|
→ Cookie-aware sessions: per-proxy cf_clearance persistence
|
|||
|
|
→ TLS impersonation: safari15_5, safari17_0, chrome123, chrome124
|
|||
|
|
→ archiveofourown.org
|
|||
|
|
|
|||
|
|
CF Challenge Solving (v4):
|
|||
|
|
1. All proxies hit CF challenge → generate challenge_token
|
|||
|
|
2. Rewrite challenge page → forward to user's browser
|
|||
|
|
3. User browser executes CF JS → challenge solved
|
|||
|
|
4. cf_clearance captured → saved to proxy cookie jar
|
|||
|
|
5. Original request retried → content delivered
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Core Files
|
|||
|
|
|
|||
|
|
| File | Purpose |
|
|||
|
|
|------|---------|
|
|||
|
|
| `app.py` | FastAPI backend v4 — proxy handler, stats, CF challenge solving |
|
|||
|
|
| `proxy_pool.py` | Tiered async proxy pool — cookie jar, weighted selection, sampling |
|
|||
|
|
| `ao3_fetcher.py` | Async fetcher — CF detection, smart retry, cookie injection |
|
|||
|
|
| `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) |
|
|||
|
|
| `stats.py` | In-memory stats with batch SQLite flush every 60s |
|
|||
|
|
| `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) |
|
|||
|
|
| `scripts/daemon.py` | Passive worker supervision (systemd, 30s check, 3-strike restart) |
|
|||
|
|
| `scripts/scan_proxies_cffi.py` | Proxy scanner with TLS fingerprint fallback |
|
|||
|
|
| `start.sh` | Startup script — kills old workers, starts new, reloads Caddy |
|
|||
|
|
| `Caddyfile` | Caddy config — TLS, round-robin, 60s timeouts |
|
|||
|
|
|
|||
|
|
## Security Baseline
|
|||
|
|
|
|||
|
|
- No secrets in code — proxy credentials are in `/home/ubuntu/proxy.txt`
|
|||
|
|
- Worker processes bound to 127.0.0.1 only (not exposed publicly)
|
|||
|
|
- Cloudflare CDN terminates TLS and provides DDoS protection at edge
|
|||
|
|
- CORS headers restrict cross-origin access
|
|||
|
|
- stats/health/metrics endpoints are read-only, no mutation
|
|||
|
|
- All proxied content is user-facing; no admin endpoints exposed
|
|||
|
|
|
|||
|
|
## Engine Guidance
|
|||
|
|
|
|||
|
|
- Complex multi-file changes, architecture evolution → Hermes (here)
|
|||
|
|
- Quick targeted fixes, single-file changes → Hermes
|
|||
|
|
- Deploy, monitor, notify, schedule → Hermes
|
|||
|
|
- Proxy scanning, proxy pool refresh → Hermes cron (30min)
|
|||
|
|
- Not sure? Start with Hermes — everything runs via Hermes
|
|||
|
|
|
|||
|
|
## Monitoring
|
|||
|
|
|
|||
|
|
- Health endpoint: https://agento3.miscs.dev/health
|
|||
|
|
- Stats dashboard: https://agento3.miscs.dev/stats
|
|||
|
|
- Metrics (Prometheus): https://agento3.miscs.dev/metrics
|
|||
|
|
- Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log
|
|||
|
|
- Daemon log: /home/ubuntu/ao3-mirror/daemon.log
|
|||
|
|
- Systemd: `systemctl status ao3-daemon`, `journalctl -u ao3-daemon`
|
|||
|
|
|
|||
|
|
## Deployment
|
|||
|
|
|
|||
|
|
- Single server (current host)
|
|||
|
|
- Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev
|
|||
|
|
- CF CDN fronts the domain (Orange Cloud = on)
|
|||
|
|
- Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron
|
|||
|
|
- No CI/CD — manual deploy via start.sh
|
|||
|
|
|
|||
|
|
## Commit Conventions
|
|||
|
|
|
|||
|
|
- One commit per meaningful change
|
|||
|
|
- Python files only (no binaries, no .pyc, no logs, no .env)
|
|||
|
|
- SOUL.md updated to reflect architectural changes
|
|||
|
|
- ao3-mirror skill updated when workflows change
|