085cc7a140
Cookie-aware proxy pool with CF challenge solving - Tiered proxy pool (fast 50 + main 676) - Per-proxy cf_clearance cookie persistence - CF challenge detection + user-browser solving - Safari + Chrome TLS fingerprint rotation - Async FastAPI backend with LRU cache - Passive daemon with systemd supervision - Stats dashboard + Prometheus metrics
111 lines
4.3 KiB
Markdown
111 lines
4.3 KiB
Markdown
# AGENTS — AO3 Mirror
|
||
|
||
This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.
|
||
|
||
## Project Overview
|
||
|
||
AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving.
|
||
|
||
- **Domain**: agento3.miscs.dev (behind Cloudflare CDN)
|
||
- **Target**: archiveofourown.org
|
||
- **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
|
||
- **QPS**: 720 (cache) | 270 (direct proxy)
|
||
|
||
## Build Commands
|
||
|
||
```bash
|
||
# Start all workers + reload Caddy
|
||
cd /home/ubuntu/ao3-mirror && bash start.sh
|
||
|
||
# Stop everything
|
||
cd /home/ubuntu/ao3-mirror && bash stop.sh
|
||
|
||
# Restart daemon only
|
||
sudo systemctl restart ao3-daemon
|
||
|
||
# Restart Caddy only
|
||
sudo systemctl reload caddy
|
||
|
||
# Scan proxy pool (MUST run in foreground!)
|
||
cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py
|
||
|
||
# Syntax check
|
||
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py
|
||
```
|
||
|
||
## Architecture
|
||
|
||
```
|
||
User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin)
|
||
→ 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop]
|
||
→ Tiered Proxy Pool:
|
||
Fast (50): POST/login → 8s timeout, 1 retry
|
||
Main (676): GET/browse → 15s timeout, 2 retries
|
||
→ Cookie-aware sessions: per-proxy cf_clearance persistence
|
||
→ TLS impersonation: safari15_5, safari17_0, chrome123, chrome124
|
||
→ archiveofourown.org
|
||
|
||
CF Challenge Solving (v4):
|
||
1. All proxies hit CF challenge → generate challenge_token
|
||
2. Rewrite challenge page → forward to user's browser
|
||
3. User browser executes CF JS → challenge solved
|
||
4. cf_clearance captured → saved to proxy cookie jar
|
||
5. Original request retried → content delivered
|
||
```
|
||
|
||
### Core Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `app.py` | FastAPI backend v4 — proxy handler, stats, CF challenge solving |
|
||
| `proxy_pool.py` | Tiered async proxy pool — cookie jar, weighted selection, sampling |
|
||
| `ao3_fetcher.py` | Async fetcher — CF detection, smart retry, cookie injection |
|
||
| `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) |
|
||
| `stats.py` | In-memory stats with batch SQLite flush every 60s |
|
||
| `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) |
|
||
| `scripts/daemon.py` | Passive worker supervision (systemd, 30s check, 3-strike restart) |
|
||
| `scripts/scan_proxies_cffi.py` | Proxy scanner with TLS fingerprint fallback |
|
||
| `start.sh` | Startup script — kills old workers, starts new, reloads Caddy |
|
||
| `Caddyfile` | Caddy config — TLS, round-robin, 60s timeouts |
|
||
|
||
## Security Baseline
|
||
|
||
- No secrets in code — proxy credentials are in `/home/ubuntu/proxy.txt`
|
||
- Worker processes bound to 127.0.0.1 only (not exposed publicly)
|
||
- Cloudflare CDN terminates TLS and provides DDoS protection at edge
|
||
- CORS headers restrict cross-origin access
|
||
- stats/health/metrics endpoints are read-only, no mutation
|
||
- All proxied content is user-facing; no admin endpoints exposed
|
||
|
||
## Engine Guidance
|
||
|
||
- Complex multi-file changes, architecture evolution → Hermes (here)
|
||
- Quick targeted fixes, single-file changes → Hermes
|
||
- Deploy, monitor, notify, schedule → Hermes
|
||
- Proxy scanning, proxy pool refresh → Hermes cron (30min)
|
||
- Not sure? Start with Hermes — everything runs via Hermes
|
||
|
||
## Monitoring
|
||
|
||
- Health endpoint: https://agento3.miscs.dev/health
|
||
- Stats dashboard: https://agento3.miscs.dev/stats
|
||
- Metrics (Prometheus): https://agento3.miscs.dev/metrics
|
||
- Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log
|
||
- Daemon log: /home/ubuntu/ao3-mirror/daemon.log
|
||
- Systemd: `systemctl status ao3-daemon`, `journalctl -u ao3-daemon`
|
||
|
||
## Deployment
|
||
|
||
- Single server (current host)
|
||
- Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev
|
||
- CF CDN fronts the domain (Orange Cloud = on)
|
||
- Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron
|
||
- No CI/CD — manual deploy via start.sh
|
||
|
||
## Commit Conventions
|
||
|
||
- One commit per meaningful change
|
||
- Python files only (no binaries, no .pyc, no logs, no .env)
|
||
- SOUL.md updated to reflect architectural changes
|
||
- ao3-mirror skill updated when workflows change
|