agento3/AGENTS.md

AGENTS — AO3 Mirror

This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.

Project Overview

AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving.

• Domain: agento3.miscs.dev (behind Cloudflare CDN)
• Target: archiveofourown.org
• Stack: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
• QPS: 720 (cache) | 270 (direct proxy)

Build Commands

# Start all workers + reload Caddy
cd /home/ubuntu/ao3-mirror && bash start.sh

# Stop everything
cd /home/ubuntu/ao3-mirror && bash stop.sh

# Restart daemon only
sudo systemctl restart ao3-daemon

# Restart Caddy only
sudo systemctl reload caddy

# Scan proxy pool (MUST run in foreground!)
cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py

# Syntax check
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py

Architecture

User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin)
  → 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop]
    → Tiered Proxy Pool:
        Fast (50): POST/login → 8s timeout, 1 retry
        Main (676): GET/browse → 15s timeout, 2 retries
    → Cookie-aware sessions: per-proxy cf_clearance persistence
    → TLS impersonation: safari15_5, safari17_0, chrome123, chrome124
    → archiveofourown.org

CF Challenge Solving (v4):
  1. All proxies hit CF challenge → generate challenge_token
  2. Rewrite challenge page → forward to user's browser
  3. User browser executes CF JS → challenge solved
  4. cf_clearance captured → saved to proxy cookie jar
  5. Original request retried → content delivered

Core Files

File	Purpose
app.py	FastAPI backend v4 — proxy handler, stats, CF challenge solving
proxy_pool.py	Tiered async proxy pool — cookie jar, weighted selection, sampling
ao3_fetcher.py	Async fetcher — CF detection, smart retry, cookie injection
cache.py	Per-worker LRU cache (5000 entries, path-differentiated TTL)
stats.py	In-memory stats with batch SQLite flush every 60s
url_rewriter.py	URL/header rewriting (ao3 → mirror domain)
scripts/daemon.py	Passive worker supervision (systemd, 30s check, 3-strike restart)
scripts/scan_proxies_cffi.py	Proxy scanner with TLS fingerprint fallback
start.sh	Startup script — kills old workers, starts new, reloads Caddy
Caddyfile	Caddy config — TLS, round-robin, 60s timeouts

Security Baseline

• No secrets in code — proxy credentials are in /home/ubuntu/proxy.txt
• Worker processes bound to 127.0.0.1 only (not exposed publicly)
• Cloudflare CDN terminates TLS and provides DDoS protection at edge
• CORS headers restrict cross-origin access
• stats/health/metrics endpoints are read-only, no mutation
• All proxied content is user-facing; no admin endpoints exposed

Engine Guidance

• Complex multi-file changes, architecture evolution → Hermes (here)
• Quick targeted fixes, single-file changes → Hermes
• Deploy, monitor, notify, schedule → Hermes
• Proxy scanning, proxy pool refresh → Hermes cron (30min)
• Not sure? Start with Hermes — everything runs via Hermes

Monitoring

• Health endpoint: https://agento3.miscs.dev/health
• Stats dashboard: https://agento3.miscs.dev/stats
• Metrics (Prometheus): https://agento3.miscs.dev/metrics
• Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log
• Daemon log: /home/ubuntu/ao3-mirror/daemon.log
• Systemd: systemctl status ao3-daemon, journalctl -u ao3-daemon

Deployment

• Single server (current host)
• Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev
• CF CDN fronts the domain (Orange Cloud = on)
• Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron
• No CI/CD — manual deploy via start.sh

Commit Conventions

• One commit per meaningful change
• Python files only (no binaries, no .pyc, no logs, no .env)
• SOUL.md updated to reflect architectural changes
• ao3-mirror skill updated when workflows change