# AGENTS — AO3 Mirror

This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.

## Project Overview

AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving.

- **Domain**: agento3.miscs.dev (behind Cloudflare CDN)
- **Target**: archiveofourown.org
- **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
- **QPS**: 720 (cache) | 270 (direct proxy)

## Build Commands

```bash
# Start all workers + reload Caddy
cd /home/ubuntu/ao3-mirror && bash start.sh

# Stop everything
cd /home/ubuntu/ao3-mirror && bash stop.sh

# Restart daemon only
sudo systemctl restart ao3-daemon

# Restart Caddy only
sudo systemctl reload caddy

# Scan proxy pool (MUST run in foreground!)
cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py

# Syntax check
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py
```

## Architecture

```
User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin)
  → 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop]
    → Tiered Proxy Pool:
        Fast (50): POST/login → 8s timeout, 1 retry
        Main (676): GET/browse → 15s timeout, 2 retries
    → Cookie-aware sessions: per-proxy cf_clearance persistence
    → TLS impersonation: safari15_5, safari17_0, chrome123, chrome124
    → archiveofourown.org

CF Challenge Solving (v4):
  1. All proxies hit CF challenge → generate challenge_token
  2. Rewrite challenge page → forward to user's browser
  3. User browser executes CF JS → challenge solved
  4. cf_clearance captured → saved to proxy cookie jar
  5. Original request retried → content delivered
```

### Core Files

| File | Purpose |
|------|---------|
| `app.py` | FastAPI backend v4 — proxy handler, stats, CF challenge solving |
| `proxy_pool.py` | Tiered async proxy pool — cookie jar, weighted selection, sampling |
| `ao3_fetcher.py` | Async fetcher — CF detection, smart retry, cookie injection |
| `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) |
| `stats.py` | In-memory stats with batch SQLite flush every 60s |
| `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) |
| `scripts/daemon.py` | Passive worker supervision (systemd, 30s check, 3-strike restart) |
| `scripts/scan_proxies_cffi.py` | Proxy scanner with TLS fingerprint fallback |
| `start.sh` | Startup script — kills old workers, starts new, reloads Caddy |
| `Caddyfile` | Caddy config — TLS, round-robin, 60s timeouts |

## Security Baseline

- No secrets in code — proxy credentials are in `/home/ubuntu/proxy.txt`
- Worker processes bound to 127.0.0.1 only (not exposed publicly)
- Cloudflare CDN terminates TLS and provides DDoS protection at edge
- CORS headers restrict cross-origin access
- stats/health/metrics endpoints are read-only, no mutation
- All proxied content is user-facing; no admin endpoints exposed

## Engine Guidance

- Complex multi-file changes, architecture evolution → Hermes (here)
- Quick targeted fixes, single-file changes → Hermes
- Deploy, monitor, notify, schedule → Hermes
- Proxy scanning, proxy pool refresh → Hermes cron (30min)
- Not sure? Start with Hermes — everything runs via Hermes

## Monitoring

- Health endpoint: https://agento3.miscs.dev/health
- Stats dashboard: https://agento3.miscs.dev/stats
- Metrics (Prometheus): https://agento3.miscs.dev/metrics
- Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log
- Daemon log: /home/ubuntu/ao3-mirror/daemon.log
- Systemd: `systemctl status ao3-daemon`, `journalctl -u ao3-daemon`

## Deployment

- Single server (current host)
- Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev
- CF CDN fronts the domain (Orange Cloud = on)
- Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron
- No CI/CD — manual deploy via start.sh

## Commit Conventions

- One commit per meaningful change
- Python files only (no binaries, no .pyc, no logs, no .env)
- SOUL.md updated to reflect architectural changes
- ao3-mirror skill updated when workflows change