diff --git a/AGENTS.md b/AGENTS.md
index 100b157..3ec32bd 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,15 +1,49 @@
-# AGENTS — AO3 Mirror
+# AGENTS — AO3 Mirror v5
This file is for AI agents (Hermes, Claude Code, Codex) working on the AO3 reverse proxy mirror.
## Project Overview
-AO3 Mirror is a high-availability reverse proxy for archiveofourown.org, designed to restore access for Chinese users. It bypasses Cloudflare's bot protection using TLS fingerprint impersonation (curl_cffi), a tiered WebShare proxy pool (726 proxies, 72%+ CF bypass rate), cookie-aware session management, and user-browser CF challenge solving.
+AO3 Mirror v5 is a high-performance reverse proxy for archiveofourown.org, redesigned from the ground up with sticky sessions, Service Worker client-side caching, and a full 5000-proxy P2C pool — matching the go3 architecture with modern Python tooling.
- **Domain**: agento3.miscs.dev (behind Cloudflare CDN)
- **Target**: archiveofourown.org
- **Stack**: Python 3.11, FastAPI, uvicorn + uvloop + httptools, curl_cffi, Caddy
-- **QPS**: 720 (cache) | 270 (direct proxy)
+- **Proxy Pool**: 5000 WebShare proxies, P2C selection, state machine (healthy/unstable/blocked/probing)
+- **Client**: Service Worker for static cache + offline fallback
+- **Architecture Reference**: https://tomorin.cv/AO3-CN/go3 (go3 v2 branch)
+
+## Architecture v5
+
+```
+User Browser
+ │ ao3_sessid_proxy cookie = sticky proxy
+ │
+ ├─ Service Worker (sw.js)
+ │ ├─ Cache API — static assets cached in browser
+ │ ├─ Navigation intercept — offline mirror picker
+ │ └─ Request passthrough — for non-static, non-nav
+ │
+ └─ HTTPS → Caddy → 2× uvicorn workers (8081-8082)
+ │
+ ├─ Proxy Pool (5000, P2C selection)
+ │ ├─ State machine: healthy/unstable/blocked/probing
+ │ ├─ Concurrency limit: asyncio.Semaphore(400)
+ │ └─ Passive probing: 10s interval, 10 concurrency
+ │
+ ├─ Sticky Session (ao3_sessid_proxy)
+ │ └─ User pinned to one proxy for session duration
+ │
+ ├─ Service Worker Deploy (/sw.js, /sw-YYYYMMDD.js)
+ │ └─ HTML injection: auto-register SW on page load
+ │
+ ├─ CF Challenge → User Browser
+ │ └─ All proxies blocked → rewrite challenge → browser solves
+ │
+ ├─ Static Cache (server-side LRU + disk MD5)
+ │
+ └─ Local routes: /stats, /health, /metrics
+```
## Build Commands
@@ -26,85 +60,60 @@ sudo systemctl restart ao3-daemon
# Restart Caddy only
sudo systemctl reload caddy
-# Scan proxy pool (MUST run in foreground!)
-cd /home/ubuntu/ao3-mirror && python3 scripts/scan_proxies_cffi.py
-
-# Syntax check
+# Syntax check all Python files
python3 -m py_compile proxy_pool.py ao3_fetcher.py app.py cache.py stats.py url_rewriter.py
+
+# Deploy Service Worker update
+# 1. Edit static/sw.js
+# 2. Update SERVICE_WORKER_DATE in app.py
+# 3. Restart workers
```
-## Architecture
-
-```
-User → Cloudflare CDN (agento3.miscs.dev) → Caddy :443 (60s, round-robin)
- → 2× uvicorn workers (8081-8082) [cpu-pinned, uvloop]
- → Tiered Proxy Pool:
- Fast (50): POST/login → 8s timeout, 1 retry
- Main (676): GET/browse → 15s timeout, 2 retries
- → Cookie-aware sessions: per-proxy cf_clearance persistence
- → TLS impersonation: safari15_5, safari17_0, chrome123, chrome124
- → archiveofourown.org
-
-CF Challenge Solving (v4):
- 1. All proxies hit CF challenge → generate challenge_token
- 2. Rewrite challenge page → forward to user's browser
- 3. User browser executes CF JS → challenge solved
- 4. cf_clearance captured → saved to proxy cookie jar
- 5. Original request retried → content delivered
-```
-
-### Core Files
+### Core Files (v5)
| File | Purpose |
|------|---------|
-| `app.py` | FastAPI backend v4 — proxy handler, stats, CF challenge solving |
-| `proxy_pool.py` | Tiered async proxy pool — cookie jar, weighted selection, sampling |
-| `ao3_fetcher.py` | Async fetcher — CF detection, smart retry, cookie injection |
+| `app.py` | FastAPI backend v5 — proxy handler, sticky sessions, SW deploy, stats |
+| `proxy_pool.py` | P2C proxy pool v5 — 5000 proxies, state machine, semaphore limiter |
+| `ao3_fetcher.py` | Single-request fetcher v5 — no internal retry, CF detection |
| `cache.py` | Per-worker LRU cache (5000 entries, path-differentiated TTL) |
| `stats.py` | In-memory stats with batch SQLite flush every 60s |
| `url_rewriter.py` | URL/header rewriting (ao3 → mirror domain) |
-| `scripts/daemon.py` | Passive worker supervision (systemd, 30s check, 3-strike restart) |
-| `scripts/scan_proxies_cffi.py` | Proxy scanner with TLS fingerprint fallback |
-| `start.sh` | Startup script — kills old workers, starts new, reloads Caddy |
-| `Caddyfile` | Caddy config — TLS, round-robin, 60s timeouts |
+| `static/sw.js` | Service Worker — client-side cache, offline fallback |
+| `Caddyfile` | Caddy config — SW routes, round-robin, 60s timeouts |
+| `scripts/daemon.py` | Passive worker supervision (systemd) |
+| `start.sh` | Startup — kills old workers, starts new, reloads Caddy |
## Security Baseline
-- No secrets in code — proxy credentials are in `/home/ubuntu/proxy.txt`
-- Worker processes bound to 127.0.0.1 only (not exposed publicly)
-- Cloudflare CDN terminates TLS and provides DDoS protection at edge
+- No secrets in code — proxy credentials in `/home/ubuntu/proxy.txt`
+- Workers bound to 127.0.0.1 only
+- Cloudflare CDN terminates TLS at edge
- CORS headers restrict cross-origin access
-- stats/health/metrics endpoints are read-only, no mutation
-- All proxied content is user-facing; no admin endpoints exposed
+- stats/health/metrics are read-only
## Engine Guidance
- Complex multi-file changes, architecture evolution → Hermes (here)
- Quick targeted fixes, single-file changes → Hermes
- Deploy, monitor, notify, schedule → Hermes
-- Proxy scanning, proxy pool refresh → Hermes cron (30min)
+- Proxy pool refresh → Hermes cron (30min)
- Not sure? Start with Hermes — everything runs via Hermes
## Monitoring
-- Health endpoint: https://agento3.miscs.dev/health
-- Stats dashboard: https://agento3.miscs.dev/stats
-- Metrics (Prometheus): https://agento3.miscs.dev/metrics
-- Worker logs: /home/ubuntu/ao3-mirror/worker-0.log, worker-1.log
-- Daemon log: /home/ubuntu/ao3-mirror/daemon.log
-- Systemd: `systemctl status ao3-daemon`, `journalctl -u ao3-daemon`
+- Health: https://agento3.miscs.dev/health
+- Stats: https://agento3.miscs.dev/stats
+- Metrics: https://agento3.miscs.dev/metrics
+- SW: https://agento3.miscs.dev/sw.js → /sw-YYYYMMDD.js
-## Deployment
+## CTO Goal (Persistent Focus)
-- Single server (current host)
-- Caddy manages Let's Encrypt TLS certs on agento3.miscs.dev
-- CF CDN fronts the domain (Orange Cloud = on)
-- Proxy pool: WebShare static residential proxies, refreshed every 30min via Hermes cron
-- No CI/CD — manual deploy via start.sh
+Manage akiba/agento3 as CTO. Triage issues hourly, implement top priority, get founder approval before merging. Never ship without YES.
## Commit Conventions
- One commit per meaningful change
-- Python files only (no binaries, no .pyc, no logs, no .env)
-- SOUL.md updated to reflect architectural changes
+- Python files only (no binaries, .pyc, logs, .env)
+- SOUL.md updated for architectural changes
- ao3-mirror skill updated when workflows change
diff --git a/Caddyfile b/Caddyfile
index 55f70d8..1cdba97 100644
--- a/Caddyfile
+++ b/Caddyfile
@@ -1,5 +1,5 @@
-# AO3 Mirror - Caddy 配置
-# 前端负载均衡 + TLS 终止 + Cloudflare CDN 集成
+# AO3 Mirror v5 - Caddy 配置
+# 前端负载均衡 + TLS 终止 + Cloudflare CDN 集成 + Service Worker
agento3.miscs.dev {
# 全局头
@@ -9,6 +9,7 @@ agento3.miscs.dev {
X-Frame-Options "SAMEORIGIN"
X-XSS-Protection "1; mode=block"
Referrer-Policy "strict-origin-when-cross-origin"
+ Content-Security-Policy "default-src 'self'; script-src 'self' https://cdn.jsdelivr.net; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; object-src 'none'"
-Server
-X-Powered-By
}
@@ -26,6 +27,60 @@ agento3.miscs.dev {
# 压缩
encode gzip
+ # Service Worker routes — versioned + redirect
+ route /sw-* {
+ reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 {
+ lb_policy round_robin
+ health_uri /health
+ health_interval 10s
+ health_timeout 5s
+ transport http {
+ read_timeout 60s
+ write_timeout 60s
+ dial_timeout 5s
+ }
+ }
+ header {
+ Cache-Control "public, max-age=86400, immutable"
+ }
+ }
+
+ # SW redirect
+ route /sw.js {
+ reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 {
+ lb_policy round_robin
+ health_uri /health
+ health_interval 10s
+ health_timeout 5s
+ transport http {
+ read_timeout 60s
+ write_timeout 60s
+ dial_timeout 5s
+ }
+ }
+ header {
+ Cache-Control "no-store"
+ }
+ }
+
+ # Mirror domains JSON
+ route /mirror-domains.json {
+ reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 {
+ lb_policy round_robin
+ health_uri /health
+ health_interval 10s
+ health_timeout 5s
+ transport http {
+ read_timeout 60s
+ write_timeout 60s
+ dial_timeout 5s
+ }
+ }
+ header {
+ Cache-Control "public, max-age=3600"
+ }
+ }
+
# 后端负载均衡 (轮询) - 2 workers on 2-core machine
reverse_proxy 127.0.0.1:8081 127.0.0.1:8082 {
lb_policy round_robin
@@ -34,7 +89,7 @@ agento3.miscs.dev {
health_interval 10s
health_timeout 5s
- # 超时配置
+ # 超时配置 — 代理可能需要更长时间
transport http {
read_timeout 30s
write_timeout 30s
@@ -55,6 +110,7 @@ stats.agento3.miscs.dev {
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
Referrer-Policy "strict-origin-when-cross-origin"
+ Content-Security-Policy "default-src 'self'; script-src 'self' https://cdn.jsdelivr.net; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; object-src 'none'"
-Server
-X-Powered-By
}
@@ -72,41 +128,67 @@ stats.agento3.miscs.dev {
# 压缩
encode gzip
- # 路由规则 - 只暴露 /stats 和 /metrics
+ # 路由规则 - 只暴露 /stats, /metrics, /health, /sw
route {
- # /stats 页面 - 代理到后端
+ # /stats 页面
reverse_proxy /stats* 127.0.0.1:8081 127.0.0.1:8082 {
lb_policy round_robin
-
health_uri /health
health_interval 10s
health_timeout 5s
-
transport http {
- read_timeout 30s
- write_timeout 30s
+ read_timeout 60s
+ write_timeout 60s
dial_timeout 5s
}
-
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Proto {scheme}
header_up X-Real-IP {remote_host}
}
- # /metrics 指标 - 代理到后端
+ # /metrics 指标
reverse_proxy /metrics* 127.0.0.1:8081 127.0.0.1:8082 {
lb_policy round_robin
-
health_uri /health
health_interval 10s
health_timeout 5s
-
transport http {
- read_timeout 30s
- write_timeout 30s
+ read_timeout 60s
+ write_timeout 60s
dial_timeout 5s
}
+ header_up X-Forwarded-For {remote_host}
+ header_up X-Forwarded-Proto {scheme}
+ header_up X-Real-IP {remote_host}
+ }
+ # Service Worker
+ reverse_proxy /sw* 127.0.0.1:8081 127.0.0.1:8082 {
+ lb_policy round_robin
+ health_uri /health
+ health_interval 10s
+ health_timeout 5s
+ transport http {
+ read_timeout 60s
+ write_timeout 60s
+ dial_timeout 5s
+ }
+ header_up X-Forwarded-For {remote_host}
+ header_up X-Forwarded-Proto {scheme}
+ header_up X-Real-IP {remote_host}
+ }
+
+ # Mirror domains
+ reverse_proxy /mirror-domains* 127.0.0.1:8081 127.0.0.1:8082 {
+ lb_policy round_robin
+ health_uri /health
+ health_interval 10s
+ health_timeout 5s
+ transport http {
+ read_timeout 60s
+ write_timeout 60s
+ dial_timeout 5s
+ }
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Proto {scheme}
header_up X-Real-IP {remote_host}
diff --git a/ao3_fetcher.py b/ao3_fetcher.py
index 5af87ec..acebf56 100644
--- a/ao3_fetcher.py
+++ b/ao3_fetcher.py
@@ -1,25 +1,22 @@
"""
-异步 AO3 内容抓取器 v4 — Cookie 感知 + CF 挑战检测 + 智能重试
+异步 AO3 内容抓取器 v5 — 单次代理请求 + CF 挑战检测
-v4 vs v3:
-- CF 挑战检测:不再把 403/503 一律当失败
-- Cookie 注入:请求时自动携带 proxy 级别的 cookies(cf_clearance 等)
-- Cookie 保存:成功请求后自动提取 Set-Cookie 并保存到 proxy cookie jar
-- 智能重试:挑战时优先用带 cookie 的代理,无 cookie 则换代理
-- 挑战页面透传:所有重试都遇到挑战时,返回挑战 HTML 让用户浏览器求解
+v5 vs v4:
+- 去掉内部重试循环(重试逻辑迁移到 app.py 的 P2C 选择中)
+- 简化 API:直接接受 ProxySession 对象
+- 保留 CF 挑战检测 + 状态机反馈
"""
import asyncio
import logging
import time
from typing import Optional
-from proxy_pool import get_proxy_pool, ProxySession
+from proxy_pool import ProxySession
logger = logging.getLogger("ao3-fetcher")
# ─── CF Challenge Detection ───────────────────────────────────────────────
-# Markers in response body that indicate a Cloudflare challenge page
CF_CHALLENGE_MARKERS = [
b'/cdn-cgi/challenge-platform',
b'cf-challenge-running',
@@ -28,29 +25,24 @@ CF_CHALLENGE_MARKERS = [
b'challenge-platform',
b'cf-turnstile',
b'cf_chl_',
- # Sometimes CF just returns "Just a moment..." without JS markers
b'Checking your browser',
b'Just a moment...',
]
def is_cf_challenge(status: int, body: bytes, headers: dict) -> bool:
- """Detect if response is a Cloudflare challenge page (not a real error)."""
+ """Detect if response is a Cloudflare challenge page."""
if status not in (403, 503, 429):
return False
- # Quick check: CF always sets Server header on challenge pages
server = headers.get("Server", headers.get("server", ""))
if "cloudflare" not in server.lower():
- # Check body for challenge markers
for marker in CF_CHALLENGE_MARKERS:
if marker in body:
return True
return False
- # Server: cloudflare + non-200 status = likely challenge
for marker in CF_CHALLENGE_MARKERS:
if marker in body:
return True
- # If Server is cloudflare and status is 403/503, it's almost certainly a challenge
if status in (403, 503):
return True
return False
@@ -69,31 +61,29 @@ CHROME_HEADERS = {
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
- "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
+ "User-Agent": (
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+ "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+ ),
"DNT": "1",
}
API_HEADERS = {
"Accept": "*/*",
"Accept-Language": "en-US,en;q=0.9",
- "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
+ "User-Agent": (
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+ "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+ ),
}
-# ─── Timeout config ───────────────────────────────────────────────────────
-
-FAST_REQUEST_TIMEOUT = 8 # login/register/signup
+# Timeout config
+FAST_REQUEST_TIMEOUT = 8
NORMAL_REQUEST_TIMEOUT = 15
-CONNECT_TIMEOUT = 5
-MAX_RETRIES = 2
-FAST_MAX_RETRIES = 1
FAST_PATHS = {
- "/users/login",
- "/users/sign_up",
- "/users/new",
- "/invitation_requests",
- "/token_dispenser.json",
- "/user_sessions",
+ "/users/login", "/users/sign_up", "/users/new",
+ "/invitation_requests", "/token_dispenser.json", "/user_sessions",
}
@@ -106,200 +96,97 @@ def _is_fast_path(url: str) -> bool:
return False
-def _now() -> float:
- try:
- return asyncio.get_running_loop().time()
- except RuntimeError:
- return time.time()
-
-
def _merge_cookies(proxy_cookies: str, user_cookies: str) -> str:
- """Merge proxy-level cookies (cf_clearance) with user cookies. User cookies take precedence."""
if not proxy_cookies and not user_cookies:
return ""
if not proxy_cookies:
return user_cookies
if not user_cookies:
return proxy_cookies
- # User cookies take priority (they contain session auth)
- # But put proxy cookies first so user cookies can override
return f"{proxy_cookies}; {user_cookies}"
+# ─── Main fetch function ──────────────────────────────────────────────────
+
async def fetch_url(
url: str,
+ proxy: Optional[ProxySession] = None,
method: str = "GET",
headers: Optional[dict] = None,
body: Optional[bytes] = None,
cookies: Optional[dict] = None,
is_api: bool = False,
- preferred_proxy: Optional[str] = None, # v4: proxy host:port for challenge affinity
) -> dict:
"""
- Async fetch from AO3 through proxy pool with cookie-aware session reuse.
+ Single async fetch through a specific proxy. No internal retry loop.
- Priority-based:
- - Login/register paths → fast pool + 8s timeout + 1 retry
- - Normal paths → cookie-preferring proxies + 15s timeout + 2 retries
+ Args:
+ url: Target AO3 URL
+ proxy: ProxySession to use (must already be selected by app.py)
+ method: HTTP method
+ headers: Request headers
+ body: Request body (POST/PUT/PATCH)
+ cookies: User cookies dict
+ is_api: Use API headers instead of Chrome headers
- v4 improvements:
- - CF challenge detection: don't treat challenge as proxy failure
- - Cookie injection: auto-send saved cf_clearance per proxy
- - Cookie saving: auto-extract Set-Cookie on success
- - Challenge body returned for user-browser solving
-
- Returns: {
- "status": 200,
- "headers": {...},
- "body": b"...",
- "cookies": {...},
- "success": True/False,
- "error": "...",
- "elapsed": 0.5,
- "proxy_host": "p.webshare.io:10296",
- # v4:
- "is_challenge": False,
- "challenge_body": b"" | None, # CF challenge HTML for user-browser solving
- "challenge_proxy": "" | None, # which proxy got the challenge
- }
+ Returns:
+ dict with success/status/headers/body/cookies/elapsed/is_challenge etc.
"""
- pool = get_proxy_pool()
- base_headers = API_HEADERS.copy() if is_api else CHROME_HEADERS.copy()
+ if proxy is None:
+ return {
+ "status": 0, "headers": {}, "body": b"", "cookies": {},
+ "headers_raw": [],
+ "success": False, "error": "No proxy provided",
+ "elapsed": 0, "proxy_host": None,
+ "is_challenge": False, "challenge_body": None, "challenge_proxy": None,
+ }
+ host_port = proxy.host_port
+ start_time = time.time()
+
+ base_headers = API_HEADERS.copy() if is_api else CHROME_HEADERS.copy()
if headers:
- for h in ["Cookie", "Referer", "Content-Type", "X-Requested-With", "Accept",
- "Origin", "X-CSRF-Token", "Authorization"]:
+ for h in ["Cookie", "Content-Type", "X-Requested-With",
+ "Accept", "X-CSRF-Token", "Authorization"]:
if h in headers:
base_headers[h] = headers[h]
+ # Rewrite Origin and Referer: mirror domain → AO3 domain
+ # AO3 CSRF checks Origin against archiveofourown.org
+ if "Origin" in headers:
+ origin = headers["Origin"]
+ origin = origin.replace("agento3.miscs.dev", "archiveofourown.org")
+ base_headers["Origin"] = origin
+ if "Referer" in headers:
+ referer = headers["Referer"]
+ referer = referer.replace("agento3.miscs.dev", "archiveofourown.org")
+ base_headers["Referer"] = referer
+
+ # Filter out OUR cookies — proxy is transparent, only AO3 cookies should reach AO3
+ FORWARD_BLOCKED = {"ao3_sessid_proxy", "__cf_bm", "_cfuvid", "_cf_token"}
user_cookie_str = ""
if cookies:
- user_cookie_str = "; ".join(f"{k}={v}" for k, v in cookies.items())
-
- # Determine priority level
- is_fast = _is_fast_path(url) or method in ("POST", "PUT", "PATCH")
- request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT
- max_retries = FAST_MAX_RETRIES if is_fast else MAX_RETRIES
-
- last_error = None
- last_challenge_body = None
- last_challenge_proxy = None
- seen_proxies = set() # Don't retry with same proxy
-
- # v4: if preferred_proxy is set (from challenge affinity), use it first
- if preferred_proxy:
- try:
- target_proxy = None
- for p in pool._proxies:
- if p.host_port == preferred_proxy:
- target_proxy = p
- break
- if target_proxy and target_proxy.is_available:
- result = await _try_proxy(
- target_proxy, url, method, body, base_headers, user_cookie_str,
- request_timeout, is_fast
- )
- if result["success"]:
- # preferred proxy worked — save cookies
- target_proxy.save_cookies(result.get("headers", {}))
- return result
- if result.get("is_challenge"):
- last_challenge_body = result.get("challenge_body")
- last_challenge_proxy = target_proxy.host_port
- seen_proxies.add(target_proxy.host_port)
- except Exception:
- pass
-
- for attempt in range(max_retries):
- # v4: prefer proxies with cookies, fallback to any available
- if attempt == 0:
- # First attempt: use proxy with cookies if available
- proxy = pool.get_proxy_with_cookies()
- if not proxy:
- proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy()
- elif attempt == 1 and last_challenge_body:
- # Second attempt after challenge: try another proxy with cookies
- proxy = pool.get_proxy_with_cookies()
- # Make sure it's not the same one
- if proxy and proxy.host_port in seen_proxies:
- proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy()
- if not proxy:
- proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy()
- else:
- proxy = pool.get_fast_proxy() if is_fast else pool.get_proxy()
-
- if not proxy or proxy.host_port in seen_proxies:
- # Find an unseen proxy
- for _ in range(5):
- p = pool.get_proxy()
- if p and p.host_port not in seen_proxies:
- proxy = p
- break
- if not proxy or proxy.host_port in seen_proxies:
- continue
-
- seen_proxies.add(proxy.host_port)
-
- result = await _try_proxy(
- proxy, url, method, body, base_headers, user_cookie_str,
- request_timeout, is_fast
+ user_cookie_str = "; ".join(
+ f"{k}={v}" for k, v in cookies.items()
+ if k not in FORWARD_BLOCKED
)
- if result["success"]:
- # Save response cookies to proxy cookie jar for future requests
- proxy.save_cookies(result.get("headers", {}))
- return result
-
- if result.get("is_challenge"):
- last_challenge_body = result.get("challenge_body")
- last_challenge_proxy = proxy.host_port
- last_error = result.get("error", "CF_CHALLENGE")
- # Don't break — try another proxy
- continue
-
- # Real failure (connection error, timeout)
- last_error = result.get("error", "UNKNOWN")
-
- # All retries exhausted
- return {
- "status": 0, "headers": {}, "body": b"", "cookies": {},
- "success": False,
- "error": f"All retries failed: {last_error}",
- "elapsed": 0, "proxy_host": None,
- "is_challenge": last_challenge_body is not None,
- "challenge_body": last_challenge_body,
- "challenge_proxy": last_challenge_proxy,
- }
-
-
-async def _try_proxy(
- proxy: ProxySession,
- url: str,
- method: str,
- body: Optional[bytes],
- base_headers: dict,
- user_cookie_str: str,
- request_timeout: int,
- is_fast: bool,
-) -> dict:
- """Try a single request through one proxy. Returns result dict."""
- host_port = f"{proxy.host}:{proxy.port}"
- start_time = _now()
+ is_fast = _is_fast_path(url) or method in ("POST", "PUT", "PATCH")
+ request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT
try:
session = await proxy.get_session()
-
- # v4: Build per-request headers with proxy cookies
- # IMPORTANT: use headers= parameter (never session.headers.update() — race condition!)
request_headers = base_headers.copy()
- # Inject proxy-level cookies (cf_clearance, etc.)
+ # Inject proxy-level cookies (cf_clearance)
proxy_cookie_str = proxy.get_cookie_header()
cookie_str = _merge_cookies(proxy_cookie_str, user_cookie_str)
if cookie_str:
request_headers["Cookie"] = cookie_str
# Execute request
+ request_cookie_preview = cookie_str[:500] + "..." if len(cookie_str) > 500 else cookie_str
+ logger.info(f"[FETCH] {method} {url[:80]} full_cookie={request_cookie_preview}")
if method == "GET":
resp = await session.get(url, timeout=request_timeout, headers=request_headers)
elif method == "POST":
@@ -310,89 +197,79 @@ async def _try_proxy(
resp = await session.request(method, url, data=body, timeout=request_timeout,
headers=request_headers)
- elapsed = _now() - start_time
+ elapsed = time.time() - start_time
status = resp.status_code
resp_body = resp.content
- resp_headers = dict(resp.headers)
+ resp_headers = dict(resp.headers) # For Content-Type/Location lookups
+ resp_headers_raw = list(resp.headers.multi_items()) # Preserves ALL Set-Cookie values
+
+ # Debug: log response Set-Cookie count
+ set_cookie_count = sum(1 for k, v in resp_headers_raw if k.lower() == "set-cookie")
+ resp_status = resp.status_code
+ body_preview = resp_body[:200].decode('utf-8', errors='replace')
+ logger.info(f"[FETCH_RESP] {method} {url[:60]} status={resp_status} set_cookies={set_cookie_count} body={body_preview}")
+
resp_cookies = {}
if hasattr(resp, "cookies"):
for k, v in resp.cookies.items():
resp_cookies[k] = v
- # v4: Check for CF challenge
+ # CF challenge detection
if is_cf_challenge(status, resp_body, resp_headers):
- proxy.mark_challenged()
- logger.warning(f"[CF_CHALLENGE] {host_port} -> {url[:60]}: {status} ({elapsed:.1f}s)")
+ logger.warning(f"[CF_CHALLENGE] {host_port} -> {url[:60]}: {status}")
return {
- "status": status,
- "headers": resp_headers,
- "body": resp_body,
+ "status": status, "headers": resp_headers, "body": resp_body,
+ "headers_raw": resp_headers_raw,
"cookies": resp_cookies,
- "success": False,
- "error": f"CF_CHALLENGE_{status}",
- "elapsed": elapsed,
- "proxy_host": host_port,
- "is_challenge": True,
- "challenge_body": resp_body,
+ "success": False, "error": f"CF_CHALLENGE_{status}",
+ "elapsed": elapsed, "proxy_host": host_port,
+ "is_challenge": True, "challenge_body": resp_body,
"challenge_proxy": host_port,
}
- # Success — any 2xx-4xx is proxied through
+ # Success
if 200 <= status < 500:
- proxy.mark_success(elapsed)
- logger.debug(f"{host_port} -> {url[:60]}: {status} ({elapsed:.2f}s)")
-
- # v4: Save response cookies to proxy
+ sc_from_dict = resp_headers.get("Set-Cookie", resp_headers.get("set-cookie", "NONE"))
+ logger.info(f"[SAVE_COOKIES] proxy={host_port} set_cookie_dict={sc_from_dict[:120]}...")
proxy.save_cookies(resp_headers)
-
return {
- "status": status,
- "headers": resp_headers,
- "body": resp_body,
+ "status": status, "headers": resp_headers, "body": resp_body,
+ "headers_raw": resp_headers_raw,
"cookies": resp_cookies,
"success": True,
- "elapsed": elapsed,
- "proxy_host": host_port,
- "is_challenge": False,
- "challenge_body": None,
- "challenge_proxy": None,
+ "elapsed": elapsed, "proxy_host": host_port,
+ "is_challenge": False, "challenge_body": None, "challenge_proxy": None,
}
- # True error status (5xx)
- proxy.mark_failure()
- logger.warning(f"Attempt: {host_port} -> {url[:60]}: {status}")
+ # Error status
+ logger.warning(f"{host_port} -> {url[:60]}: {status}")
return {
- "status": status,
- "headers": resp_headers,
- "body": resp_body,
+ "status": status, "headers": resp_headers, "body": resp_body,
+ "headers_raw": resp_headers_raw,
"cookies": resp_cookies,
- "success": False,
- "error": f"HTTP_{status}",
- "elapsed": elapsed,
- "proxy_host": host_port,
- "is_challenge": False,
- "challenge_body": None,
- "challenge_proxy": None,
+ "success": False, "error": f"HTTP_{status}",
+ "elapsed": elapsed, "proxy_host": host_port,
+ "is_challenge": False, "challenge_body": None, "challenge_proxy": None,
}
except asyncio.TimeoutError:
- proxy.mark_failure()
- elapsed = _now() - start_time
- logger.warning(f"TIMEOUT: {host_port} -> {url[:60]} ({elapsed:.1f}s, timeout={request_timeout}s)")
+ elapsed = time.time() - start_time
+ logger.warning(f"TIMEOUT: {host_port} -> {url[:60]}")
return {
"status": 0, "headers": {}, "body": b"", "cookies": {},
+ "headers_raw": [],
"success": False, "error": "TIMEOUT",
"elapsed": elapsed, "proxy_host": host_port,
"is_challenge": False, "challenge_body": None, "challenge_proxy": None,
}
except Exception as e:
- proxy.mark_failure()
- elapsed = _now() - start_time
+ elapsed = time.time() - start_time
err_str = str(e)[:120]
- logger.debug(f"ERROR: {host_port} -> {e} ({elapsed:.1f}s)")
+ logger.debug(f"ERROR: {host_port} -> {err_str}")
return {
"status": 0, "headers": {}, "body": b"", "cookies": {},
+ "headers_raw": [],
"success": False, "error": err_str,
"elapsed": elapsed, "proxy_host": host_port,
"is_challenge": False, "challenge_body": None, "challenge_proxy": None,
diff --git a/app.py b/app.py
index e0909e7..6307822 100644
--- a/app.py
+++ b/app.py
@@ -1,13 +1,17 @@
"""
-AO3 反代后端 v4 — Cookie 感知 + CF 挑战用户浏览器求解
+AO3 反代后端 v5 — 粘性会话 + Service Worker + P2C 代理池 + 并发限流
-v4 vs v3:
-- Challenge token 映射:用户浏览器求解 CF 挑战时,保证同一 proxy 亲和性
-- 挑战页面透传:所有代理都遇到 CF 挑战时,把挑战页发给用户浏览器求解
-- Cookie 流:成功响应自动保存 proxy cookie,后续请求自动携带
-- 统计面板展示 cookie-aware 代理数
+v5 vs v4:
+- 粘性会话: ao3_sessid_proxy cookie (对标 go3)
+- Service Worker: /sw.js, /sw-YYYYMMDD.js for client-side caching
+- HTML 注入: 自动注入 SW 注册脚本 + 镜像域名
+- P2C 代理选择: Power of Two Choices (对标 go3)
+- 并发限流: asyncio.Semaphore(400) (Webshare 上限)
+- 全量 5000 代理: 不再按端口过滤
+- 状态机: healthy/unstable/blocked/probing
"""
import hashlib
+import html
import json
import logging
import os
@@ -23,9 +27,9 @@ import uvicorn
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from proxy_pool import get_proxy_pool
-from ao3_fetcher import fetch_url, is_cf_challenge
-from url_rewriter import rewrite_body, rewrite_response_headers, needs_rewrite, MIRROR_DOMAIN
+from proxy_pool import get_proxy_pool, MAX_CONCURRENT_REQUESTS, OUTAGE_THRESHOLD
+from ao3_fetcher import fetch_url, is_cf_challenge, FAST_REQUEST_TIMEOUT, NORMAL_REQUEST_TIMEOUT
+from url_rewriter import rewrite_body, rewrite_response_headers, rewrite_response_headers_raw, needs_rewrite, MIRROR_DOMAIN
from cache import get_cache, get_ttl_for_path
from stats import get_stats_collector
@@ -39,13 +43,15 @@ logger = logging.getLogger("ao3-backend")
AO3_BASE = "https://archiveofourown.org"
MIRROR_HOST = MIRROR_DOMAIN
+# Service Worker version date (change on SW updates)
+SERVICE_WORKER_DATE = "20260623"
+
LOCAL_PATHS = {"/stats", "/health", "/metrics", "/favicon.ico", "/robots.txt"}
-# ─── Challenge Token Map (v4) ──────────────────────────────────────────────
-# Maps challenge_token → (method, ao3_url, headers, body, cookies, proxy_host, expires)
+# ─── Challenge Token Map (v4 retained) ──────────────────────────────────────
_challenge_map: dict[str, tuple] = {}
_challenge_lock = threading.Lock()
-CHALLENGE_TOKEN_TTL = 120 # 2 minutes for the user's browser to solve the challenge
+CHALLENGE_TOKEN_TTL = 120
def _make_challenge_token() -> str:
@@ -55,7 +61,6 @@ def _make_challenge_token() -> str:
def _store_challenge(token: str, method: str, ao3_url: str, headers: dict,
body: bytes, cookies: dict, proxy_host: str):
with _challenge_lock:
- # Clean expired tokens
now = time.time()
expired = [k for k, v in _challenge_map.items() if v[6] < now]
for k in expired:
@@ -71,16 +76,38 @@ def _get_challenge(token: str) -> tuple | None:
del _challenge_map[token]
return entry
if entry:
- del _challenge_map[token] # expired
+ del _challenge_map[token]
return None
+# ─── Sticky Session ─────────────────────────────────────────────────────────
+
+STICKY_COOKIE_NAME = "ao3_sessid_proxy"
+
+
+def get_sticky_proxy_idx(request: Request) -> tuple[int, bool]:
+ """Get sticky proxy index from cookie. Returns (idx, had_sticky)."""
+ c = request.cookies.get(STICKY_COOKIE_NAME)
+ if c is not None:
+ try:
+ idx = int(c)
+ pool = get_proxy_pool()
+ if 0 <= idx < len(pool.proxies):
+ return idx, True
+ except (ValueError, TypeError):
+ pass
+ # No sticky cookie — random
+ import random
+ pool = get_proxy_pool()
+ return random.randint(0, max(len(pool.proxies) - 1, 0)), False
+
+
# ─── App ───────────────────────────────────────────────────────────────────
app = FastAPI(
title="AO3 Mirror",
description="AO3 reverse proxy mirror for Chinese users",
- version="4.0.0",
+ version="5.0.0",
docs_url=None,
redoc_url=None,
)
@@ -106,31 +133,51 @@ def build_ao3_url(path: str, query: str = "") -> str:
# ─── Challenge page URL rewriting ──────────────────────────────────────────
def _rewrite_challenge_page(body: bytes, proxy_host: str, token: str) -> bytes:
- """Rewrite CF challenge page so all URLs go through mirror, and tag with challenge token."""
- import re
-
- # Basic AO3 domain rewrite
+ """Rewrite CF challenge page — inject retry UI + auto cookie."""
+ # Rewrite domain
body = body.replace(b"archiveofourown.org", MIRROR_DOMAIN.encode())
+ # Rewrite CF challenge paths — our CF CDN would intercept /cdn-cgi/,
+ # so we remap to /__cf/ which our proxy handler forwards to AO3
+ body = body.replace(b"/cdn-cgi/", b"/__cf/")
+ body = body.replace(b"cdn-cgi/challenge", b"__cf/challenge")
- # Rewrite /cdn-cgi/ challenge endpoints to go through mirror
- # These are CF's internal challenge platform URLs
- body = body.replace(
- b"/cdn-cgi/challenge-platform",
- f"/cdn-cgi/challenge-platform".encode()
+ # Inject cookie + retry overlay
+ retry_overlay = (
+ f''.encode()
+ + b''
+ + b'
'
+ + 'AO3 Mirror - CF challenge solving in background. '.encode()
+ + '
Click to retry after a few seconds.'.encode()
+ + b'
'
)
-
- # Inject a marker meta tag so we can detect challenge pages coming back
- marker = f' '.encode()
- marker_cookie = (
- f''
- ).encode()
- body = body.replace(b"", b"" + marker + marker_cookie, 1)
- if b"" not in body:
- body = b"" + marker + marker_cookie + b"" + body
-
+ body = body.replace(b" bytes:
+ """Inject SW registration script into HTML, before ."""
+ import re
+ sw_script = (
+ b''
+ )
+ # Match with optional leading whitespace
+ match = re.search(rb'\s*', html_body)
+ if match:
+ pos = match.start()
+ return html_body[:pos] + sw_script + html_body[pos:]
+ # Fallback: insert at beginning if no head tag
+ return sw_script + html_body
+
+
# ─── Response header helpers ───────────────────────────────────────────────
def filter_response_headers(headers: dict) -> dict:
@@ -149,7 +196,7 @@ def filter_response_headers(headers: dict) -> dict:
if key.lower().startswith("cf-"):
continue
result[key] = value
- result["X-Proxy"] = "AO3-Mirror/4.0"
+ result["X-Proxy"] = "AO3-Mirror/5.0"
result["X-Cache"] = "MISS"
result["Cache-Control"] = "public, max-age=60, s-maxage=60"
return result
@@ -169,15 +216,12 @@ def add_cors(response: Response):
async def health():
pool = get_proxy_pool()
stats = pool.get_stats()
- resp = JSONResponse({
+ return JSONResponse({
"status": "ok",
"timestamp": time.time(),
- "version": "4.0.0",
+ "version": "5.0.0",
"proxy_pool": stats,
})
- resp.headers["Cache-Control"] = "no-store, no-cache, must-revalidate"
- resp.headers["Pragma"] = "no-cache"
- return resp
@app.get("/robots.txt")
@@ -187,6 +231,43 @@ async def robots():
)
+# ─── Service Worker Routes ─────────────────────────────────────────────────
+
+@app.get("/sw.js")
+async def service_worker_redirect():
+ """Redirect to versioned SW file."""
+ return RedirectResponse(url=f"/sw-{SERVICE_WORKER_DATE}.js", status_code=302)
+
+
+@app.get("/sw-{date}.js")
+async def service_worker(date: str):
+ """Serve versioned Service Worker."""
+ sw_path = os.path.join(os.path.dirname(__file__), "static", "sw.js")
+ try:
+ with open(sw_path, "rb") as f:
+ sw_content = f.read()
+ except FileNotFoundError:
+ return Response(status_code=404)
+
+ return Response(
+ content=sw_content,
+ media_type="application/javascript",
+ headers={
+ "Cache-Control": "public, max-age=86400, immutable",
+ "Service-Worker-Allowed": "/",
+ },
+ )
+
+
+@app.get("/mirror-domains.json")
+async def mirror_domains():
+ """JSON endpoint for SW cross-domain awareness."""
+ return JSONResponse({
+ "domains": [MIRROR_DOMAIN],
+ "primary": MIRROR_DOMAIN,
+ })
+
+
# ─── Stats Dashboard ───────────────────────────────────────────────────────
@app.get("/stats", response_class=HTMLResponse)
@@ -213,7 +294,7 @@ async def stats_page():
- AO3 Mirror - 统计面板 v4
+ AO3 Mirror - 统计面板 v5
-
🛡 AO3 Mirror 统计面板 v4
+
🛡 AO3 Mirror 统计面板 v5
@@ -275,14 +357,14 @@ async def stats_page():
P99: {stats['p99_elapsed_ms']} ms
-
代理池
-
{proxy_stats['alive']}/{proxy_stats['total']}
-
可用: {proxy_stats['available']} | 受损: {proxy_stats['dead']} | Banned: {proxy_stats['banned']}
+
代理池 ({proxy_stats['total']})
+
{proxy_stats['healthy']}/{proxy_stats['total']}
+
不稳定: {proxy_stats['unstable']} | 封禁: {proxy_stats['blocked']} | 探测: {proxy_stats['probing']}
Cookie 代理
-
{proxy_stats['with_cookies']}/{proxy_stats['alive']}
-
已持有 cf_clearance
+
{proxy_stats['with_cookies']}/{proxy_stats['total']}
+
inflight: {proxy_stats['total_inflight']}
本地缓存
@@ -306,7 +388,7 @@ async def stats_page():
热门路径
路径 请求数 成功 平均延迟
- {''.join(f'{p["path"]} {p["requests"]:,} 0.8 else "badge-red"}">{round(p["successful"]/max(p["requests"],1)*100)}% {p["avg_elapsed"]*1000:.0f}ms ' for p in stats['top_paths'][:15])}
+ {''.join(f'{html.escape(p["path"])} {p["requests"]:,} 0.8 else "badge-red"}">{round(p["successful"]/max(p["requests"],1)*100)}% {p["avg_elapsed"]*1000:.0f}ms ' for p in stats['top_paths'][:15])}
@@ -394,11 +476,13 @@ async def metrics():
f'ao3_mirror_avg_elapsed_ms {stats["avg_elapsed_ms"]}',
"# HELP ao3_mirror_proxy_pool Proxy pool status",
"# TYPE ao3_mirror_proxy_pool gauge",
- f'ao3_mirror_proxy_alive {proxy_stats["alive"]}',
- f'ao3_mirror_proxy_dead {proxy_stats["dead"]}',
- f'ao3_mirror_proxy_banned {proxy_stats["banned"]}',
- f'ao3_mirror_proxy_available {proxy_stats["available"]}',
+ f'ao3_mirror_proxy_total {proxy_stats["total"]}',
+ f'ao3_mirror_proxy_healthy {proxy_stats["healthy"]}',
+ f'ao3_mirror_proxy_unstable {proxy_stats["unstable"]}',
+ f'ao3_mirror_proxy_blocked {proxy_stats["blocked"]}',
+ f'ao3_mirror_proxy_probing {proxy_stats["probing"]}',
f'ao3_mirror_proxy_with_cookies {proxy_stats["with_cookies"]}',
+ f'ao3_mirror_proxy_inflight {proxy_stats["total_inflight"]}',
"# HELP ao3_mirror_cache_size Current cache size",
"# TYPE ao3_mirror_cache_size gauge",
f'ao3_mirror_cache_size {cache_stats["size"]}',
@@ -406,11 +490,11 @@ async def metrics():
return PlainTextResponse("\n".join(lines))
-# ─── Proxy Core (v4) ──────────────────────────────────────────────────────
+# ─── Proxy handler (v5) ──────────────────────────────────────────────────
@app.api_route("/{path:path}", methods=["GET", "POST", "HEAD", "OPTIONS", "PUT", "DELETE", "PATCH"])
async def proxy_handler(request: Request, path: str):
- """Main proxy handler — with cookie-aware fetching and CF challenge solving."""
+ """Main proxy handler — sticky sessions + P2C selection + semaphore-limited."""
start_time = time.time()
client_ip = get_client_ip(request)
@@ -418,88 +502,93 @@ async def proxy_handler(request: Request, path: str):
path = ""
full_path = f"/{path}" if path else "/"
- if full_path in LOCAL_PATHS or full_path.startswith("/stats") or full_path.startswith("/health"):
+ if full_path in LOCAL_PATHS or full_path.startswith("/sw"):
return JSONResponse({"error": "Not found"}, status_code=404)
+ # Remap /__cf/ → /cdn-cgi/ (CF challenge paths bypass our CDN)
+ if full_path.startswith("/__cf/"):
+ path = "cdn-cgi" + path[4:]
+ full_path = "/" + path
+
if request.method == "OPTIONS":
resp = Response()
add_cors(resp)
return resp
- # ── v4: Challenge token check ──────────────────────────────────────────
- # If this request has a _cf_token cookie, it's part of a challenge resolution flow
+ pool = get_proxy_pool()
+ query_string = request.url.query
+ ao3_url = build_ao3_url(f"/{path}" if path else "/", query_string)
+
+ # ── v4: Challenge token check ────────────────────────────────────────
cf_token = request.cookies.get("_cf_token") or request.query_params.get("_cf_token")
- preferred_proxy = None
if cf_token:
challenge_entry = _get_challenge(cf_token)
if challenge_entry:
orig_method, orig_url, orig_headers, orig_body, orig_cookies, proxy_host, _ = challenge_entry
- preferred_proxy = proxy_host
- logger.info(f"Challenge resolution: using proxy {proxy_host} for {orig_url[:60]}")
- # Re-fetch the original request with the challenge proxy
+ logger.info(f"Challenge resolution: using proxy {proxy_host}")
+ challenge_proxy_obj = pool.get_by_addr(proxy_host)
result = await fetch_url(
url=orig_url, method=orig_method,
+ proxy=challenge_proxy_obj,
headers=dict(request.headers),
body=await request.body() if orig_method in ("POST", "PUT", "PATCH") else None,
cookies=orig_cookies,
is_api="/api/" in orig_url,
- preferred_proxy=preferred_proxy,
)
-
if result["success"]:
- # Challenge solved! Proxy now has cf_clearance. Return content.
elapsed = time.time() - start_time
rewritten_body = rewrite_body(result["body"],
- result["headers"].get("Content-Type", ""))
+ result["headers"].get("Content-Type", result["headers"].get("content-type", "")))
+ content_type = result["headers"].get("Content-Type", result["headers"].get("content-type", ""))
+ if "text/html" in content_type:
+ rewritten_body = inject_service_worker(rewritten_body)
rewritten_headers = rewrite_response_headers(result["headers"])
final_headers = filter_response_headers(rewritten_headers)
+ final_headers.pop("Set-Cookie", None)
+ final_headers.pop("set-cookie", None)
final_headers["X-Cache"] = "MISS"
final_headers["X-CF-Status"] = "solved"
- # Set cf_clearance as a cookie on the mirror domain too
- # so subsequent requests benefit
resp = Response(content=rewritten_body, status_code=result["status"],
headers=final_headers)
- # Forward any Set-Cookie from AO3 (includes cf_clearance)
- for k, v in result["headers"].items():
- if k.lower() == "set-cookie":
- # Rewrite domain
- v_rewritten = v.replace("domain=archiveofourown.org",
- f"domain={MIRROR_DOMAIN}")
- v_rewritten = v_rewritten.replace("domain=.archiveofourown.org",
- f"domain=.{MIRROR_DOMAIN}")
- resp.headers.add("Set-Cookie", v_rewritten)
- # Clean up challenge token cookie
+ # Append ALL Set-Cookie from raw (preserves multi-value)
+ # Filter: strip AO3 CF cookies
+ SKIP_CF = {"__cf_bm", "_cfuvid"}
+ ao3_headers_raw = result.get("headers_raw", [])
+ if ao3_headers_raw:
+ rewritten_raw = rewrite_response_headers_raw(ao3_headers_raw)
+ for key, value in rewritten_raw:
+ if key.lower() == "set-cookie":
+ cookie_name = value.split("=")[0].split(";")[0].strip()
+ if cookie_name in SKIP_CF:
+ continue
+ resp.headers.append("Set-Cookie", value)
+
resp.delete_cookie("_cf_token", path="/")
collector = get_stats_collector()
collector.log_request(method=orig_method, path=full_path,
status=result["status"], elapsed=elapsed,
- cached=False, proxy_host=preferred_proxy,
+ cached=False, proxy_host=proxy_host,
client_ip=client_ip)
add_cors(resp)
return resp
- # ── Normal request flow ────────────────────────────────────────────────
-
- query_string = request.url.query
- ao3_url = build_ao3_url(f"/{path}" if path else "/", query_string)
-
- # Check cache
+ # ── Static cache check (static assets only) ────────────────────
+ from cache import is_static_path
cache = get_cache()
- cached = cache.get(ao3_url, dict(request.headers))
- if cached:
- body, resp_headers, status = cached
- elapsed = time.time() - start_time
- collector = get_stats_collector()
- collector.log_request(method=request.method, path=full_path, status=status,
- elapsed=elapsed, cached=True, client_ip=client_ip)
- headers = filter_response_headers(resp_headers)
- headers["X-Cache"] = "HIT"
- return Response(content=body, status_code=status, headers=headers)
+ if request.method == "GET" and is_static_path(full_path):
+ cached = cache.get(ao3_url)
+ if cached:
+ body, resp_headers, status = cached
+ elapsed = time.time() - start_time
+ collector = get_stats_collector()
+ collector.log_request(method=request.method, path=full_path, status=status,
+ elapsed=elapsed, cached=True, client_ip=client_ip)
+ return Response(content=body, status_code=status, headers=resp_headers)
- # Prepare request
+ # ── Prepare request ──────────────────────────────────────────────────
method = request.method
client_headers = dict(request.headers)
req_body = None
@@ -514,101 +603,167 @@ async def proxy_handler(request: Request, path: str):
k, v = pair.split("=", 1)
cookies[k.strip()] = v.strip()
- # Forward to AO3
- result = await fetch_url(
- url=ao3_url, method=method,
- headers=client_headers, body=req_body, cookies=cookies,
- is_api="/api/" in full_path or full_path.startswith("/api/"),
- preferred_proxy=preferred_proxy,
- )
+ # ── v5: Sticky proxy + P2C selection with semaphore ──────────────────
+ sticky_idx, had_sticky = get_sticky_proxy_idx(request)
+ final_proxy_idx = sticky_idx
+ tried_idxs: set[int] = set()
+
+ import random
+ is_fast = method in ("POST", "PUT", "PATCH") or "/users/" in full_path
+ request_timeout = FAST_REQUEST_TIMEOUT if is_fast else NORMAL_REQUEST_TIMEOUT
+
+ # Try with sticky first, then P2C fallback
+ has_session = bool(cookies) # Stateful request: has user cookies
+ session_cookie_val = cookies.get("_otwarchive_session", "")[:80]
+ logger.info(f"Request {method} {full_path} | cookies={list(cookies.keys())} | session={session_cookie_val}... | has_session={has_session}")
+ async with pool.concurrency_limiter:
+ # First attempt: sticky proxy
+ result = None
+ sticky_proxy = pool.sticky_select(sticky_idx, tried_idxs)
+ if sticky_proxy:
+ sticky_proxy.inflight += 1
+ tried_idxs.add(sticky_proxy.idx)
+ try:
+ result = await fetch_url(
+ url=ao3_url, method=method,
+ proxy=sticky_proxy,
+ headers=client_headers, body=req_body, cookies=cookies,
+ is_api="/api/" in full_path,
+ )
+ if result["success"]:
+ final_proxy_idx = sticky_proxy.idx
+ sticky_proxy.mark_success(result.get("elapsed", 0))
+ sticky_proxy.inflight -= 1
+ elif result.get("is_challenge"):
+ sticky_proxy.mark_challenged()
+ sticky_proxy.inflight -= 1
+ else:
+ status = result.get("status", 0)
+ if status in (403, 525):
+ sticky_proxy.mark_blocked()
+ else:
+ sticky_proxy.mark_failure(is_network=True)
+ sticky_proxy.inflight -= 1
+ result = None # Force retry
+ except Exception:
+ if sticky_proxy:
+ sticky_proxy.mark_failure(is_network=True)
+ sticky_proxy.inflight -= 1
+ result = None
+
+ # Retry loop with P2C — but NEVER for stateful requests
+ # If the sticky proxy failed on a request with cookies, switching
+ # proxies breaks CSRF/session affinity. Forward CF challenge instead.
+ if (not result or not result["success"]) and not has_session:
+ max_retries = 5
+ for attempt in range(max_retries):
+ if attempt > 0:
+ await _sleep(0.1 * (2 ** min(attempt, 3)))
+ proxy = pool.p2c_select(exclude_idxs=tried_idxs)
+ if not proxy:
+ break
+ proxy.inflight += 1
+ tried_idxs.add(proxy.idx)
+ try:
+ result = await fetch_url(
+ url=ao3_url, method=method,
+ proxy=proxy,
+ headers=client_headers, body=req_body, cookies=cookies,
+ is_api="/api/" in full_path,
+ )
+ if result["success"]:
+ final_proxy_idx = proxy.idx
+ proxy.mark_success(result.get("elapsed", 0))
+ proxy.inflight -= 1
+ break
+ elif result.get("is_challenge"):
+ proxy.mark_challenged()
+ proxy.inflight -= 1
+ else:
+ status = result.get("status", 0)
+ if status in (403, 525):
+ proxy.mark_blocked()
+ elif status > 0:
+ proxy.mark_failure(is_network=False)
+ else:
+ proxy.mark_failure(is_network=True)
+ proxy.inflight -= 1
+ result = None
+ except Exception:
+ proxy.mark_failure(is_network=True)
+ proxy.inflight -= 1
+ result = None
elapsed = time.time() - start_time
- # ── v4: CF Challenge handling ──────────────────────────────────────────
- if result.get("is_challenge") and result.get("challenge_body"):
- challenge_proxy = result.get("challenge_proxy", "unknown")
- challenge_body = result["challenge_body"]
+ # ── All proxies exhausted — outage ───────────────────────────────────
+ if not result or not result["success"]:
+ if result and result.get("is_challenge") and result.get("challenge_body"):
+ challenge_proxy = result.get("challenge_proxy", "unknown")
+ token = _make_challenge_token()
+ _store_challenge(token, method, ao3_url, client_headers,
+ req_body or b"", cookies, challenge_proxy)
+ rewritten_challenge = _rewrite_challenge_page(
+ result["challenge_body"], challenge_proxy, token
+ )
+ logger.warning(f"CF Challenge via {challenge_proxy}, sending to browser")
+ collector = get_stats_collector()
+ collector.log_request(method=method, path=full_path, status=503,
+ elapsed=elapsed, cached=False,
+ proxy_host=challenge_proxy, client_ip=client_ip)
+ resp = HTMLResponse(
+ content=rewritten_challenge,
+ status_code=200, # 200 so browser renders and executes CF JS
+ headers={
+ "X-CF-Challenge": "true",
+ "X-CF-Challenge-Proxy": challenge_proxy,
+ },
+ )
+ resp.set_cookie(
+ key="_cf_token", value=token,
+ path="/", max_age=CHALLENGE_TOKEN_TTL,
+ httponly=False, samesite="lax",
+ )
+ resp.delete_cookie("cf_clearance", path="/")
+ add_cors(resp)
+ return resp
- # Generate challenge token for proxy affinity
- token = _make_challenge_token()
- _store_challenge(token, method, ao3_url, client_headers,
- req_body or b"", cookies, challenge_proxy)
+ # Check if we hit outage threshold
+ if pool.pool_unavailable_ratio() >= OUTAGE_THRESHOLD:
+ logger.error(f"OUTAGE: {pool.pool_unavailable_ratio():.1%} proxies unavailable")
+ return HTMLResponse(
+ content=_outage_page(), status_code=503
+ )
- # Rewrite challenge page for user's browser
- rewritten_challenge = _rewrite_challenge_page(challenge_body, challenge_proxy, token)
-
- logger.warning(
- f"CF Challenge detected via {challenge_proxy} for {ao3_url[:80]}. "
- f"Sending challenge to user browser (token={token[:8]}...)"
- )
-
- collector = get_stats_collector()
- collector.log_request(method=method, path=full_path, status=503,
- elapsed=elapsed, cached=False,
- proxy_host=challenge_proxy, client_ip=client_ip)
-
- # Return challenge page with 503 status + token cookie
- resp = HTMLResponse(
- content=rewritten_challenge,
- status_code=503,
- headers={
- "X-CF-Challenge": "true",
- "X-CF-Challenge-Proxy": challenge_proxy,
- "Retry-After": "5",
- },
- )
- resp.set_cookie(
- key="_cf_token", value=token,
- path="/", max_age=CHALLENGE_TOKEN_TTL,
- httponly=False, # JS needs to read it
- samesite="lax",
- )
- resp.delete_cookie("cf_clearance", path="/") # Clear stale clearance
- add_cors(resp)
- return resp
-
- # ── Standard failure ───────────────────────────────────────────────────
- if not result["success"]:
- logger.error(f"Failed to fetch {ao3_url[:80]}: {result.get('error', 'unknown')}")
+ logger.error(f"Failed: {ao3_url[:80]}: all retries exhausted")
collector = get_stats_collector()
collector.log_request(method=method, path=full_path, status=502,
elapsed=elapsed, cached=False, client_ip=client_ip)
+ return HTMLResponse(content=_transient_error_page(), status_code=502)
- error_html = f"""
-
-
AO3 Mirror - 暂时不可用
-
-
-
-
🔄 正在尝试连接 AO3
-
镜像站正在尝试通过代理重新连接 AO3 服务器。
-
请稍后刷新页面重试。
-
刷新页面
-
-
-"""
- return HTMLResponse(content=error_html, status_code=502)
-
- # ── Success ────────────────────────────────────────────────────────────
+ # ── Success ──────────────────────────────────────────────────────────
ao3_status = result["status"]
ao3_headers = result.get("headers", {})
+ ao3_headers_raw = result.get("headers_raw", [])
raw_body = result.get("body", b"")
- content_type = ao3_headers.get("Content-Type", "")
+ content_type = ao3_headers.get("Content-Type", ao3_headers.get("content-type", ""))
rewritten_body = rewrite_body(raw_body, content_type)
+
+ # Inject Service Worker into HTML responses
+ if "text/html" in content_type:
+ rewritten_body = inject_service_worker(rewritten_body)
+
rewritten_headers = rewrite_response_headers(ao3_headers)
final_headers = filter_response_headers(rewritten_headers)
+ # Remove Set-Cookie from dict — multi-value Set-Cookie from raw below
+ final_headers.pop("Set-Cookie", None)
+ final_headers.pop("set-cookie", None)
final_headers["X-Cache"] = "MISS"
- # Cache successful GET responses
- if method == "GET" and 200 <= ao3_status < 400:
- ttl = get_ttl_for_path(full_path)
- cache.set(ao3_url, rewritten_body, rewritten_headers, ao3_status, ttl=ttl)
+ # Cache successful static GET responses (disk MD5 cache, like go3)
+ if method == "GET" and ao3_status == 200 and is_static_path(full_path):
+ cache.set(ao3_url, rewritten_body, rewritten_headers, ao3_status)
# Handle redirects
if 300 <= ao3_status < 400 and "location" in rewritten_headers:
@@ -624,27 +779,94 @@ h1 {{ color: #990000; }}
client_ip=client_ip)
resp = Response(content=rewritten_body, status_code=ao3_status, headers=final_headers)
- # Forward Set-Cookie from AO3 (user session cookies + cf_clearance)
- for k, v in ao3_headers.items():
- if k.lower() == "set-cookie":
- v_rewritten = v.replace("domain=archiveofourown.org",
- f"domain={MIRROR_DOMAIN}")
- v_rewritten = v_rewritten.replace("domain=.archiveofourown.org",
- f"domain=.{MIRROR_DOMAIN}")
- resp.headers.add("Set-Cookie", v_rewritten)
+
+ # Append ALL Set-Cookie headers from raw (preserves multi-value — pitfall #18a)
+ # Filter: strip AO3 CF cookies — our own CF edge handles these
+ SKIP_COOKIES = {"__cf_bm", "_cfuvid"}
+ if ao3_headers_raw:
+ rewritten_raw = rewrite_response_headers_raw(ao3_headers_raw)
+ for key, value in rewritten_raw:
+ if key.lower() == "set-cookie":
+ cookie_name = value.split("=")[0].split(";")[0].strip()
+ if cookie_name in SKIP_COOKIES:
+ continue
+ logger.info(f"[SET_COOKIE_FWD] {value[:120]}...")
+ resp.headers.append("Set-Cookie", value)
+
+ # Set sticky session cookie
+ resp.set_cookie(
+ key=STICKY_COOKIE_NAME,
+ value=str(final_proxy_idx),
+ path="/",
+ httponly=True,
+ samesite="lax",
+ )
+
add_cors(resp)
return resp
+# ─── Error pages ───────────────────────────────────────────────────────────
+
+def _outage_page() -> str:
+ return """
+
+
AO3 Mirror - 暂时不可用
+
+
+
+
⚠️ AO3 暂时不可用
+
所有代理目前都无法连接 AO3。可能 AO3 正在维护或大规模封锁。
+
请稍后重试。如果持续出现,请切换到数据流量。
+
刷新页面
+
+
+"""
+
+
+def _transient_error_page() -> str:
+ return """
+
+
AO3 Mirror - 连接失败
+
+
+
+
🔄 正在尝试连接 AO3
+
镜像站正在通过代理重新连接 AO3 服务器。
+
请稍后刷新页面重试。
+
刷新页面
+
+
+"""
+
+
+# ─── Helper ────────────────────────────────────────────────────────────────
+
+async def _sleep(seconds: float):
+ import asyncio
+ await asyncio.sleep(seconds)
+
+
# ─── Startup / Shutdown ────────────────────────────────────────────────────
@app.on_event("startup")
async def startup():
- logger.info("AO3 Mirror backend v4 starting up...")
- get_proxy_pool()
+ logger.info("AO3 Mirror backend v5 starting up...")
+ pool = get_proxy_pool()
get_cache()
get_stats_collector()
- logger.info("AO3 Mirror backend v4 started (cookie-aware + CF challenge solving)")
+ logger.info(f"AO3 Mirror backend v5 started ({len(pool.proxies)} proxies, "
+ f"P2C selection, sticky sessions, SW deploy)")
@app.on_event("shutdown")
@@ -660,7 +882,7 @@ async def shutdown():
if __name__ == "__main__":
port = int(os.environ.get("PORT", "8080"))
workers = int(os.environ.get("WORKERS", "4"))
- logger.info(f"Starting AO3 Mirror v4 on port {port} with {workers} workers")
+ logger.info(f"Starting AO3 Mirror v5 on port {port} with {workers} workers")
uvicorn.run(
"app:app",
diff --git a/cache.py b/cache.py
index c65cbfd..c06a0f0 100644
--- a/cache.py
+++ b/cache.py
@@ -1,123 +1,164 @@
"""
-简单高效的缓存层
-- 内存 LRU 缓存,每个 worker 独立
-- 短 TTL 避免内容过时
-- 针对不同路径设置不同 TTL
-"""
+静态资源磁盘缓存 v5 — 对标 go3
+只缓存明确是静态资源的路径(CSS/JS/图片/字体)。
+动态 HTML / API / 登录流程 零缓存 — 直接透传。
+缓存文件以 MD5(URL) 命名,存储在 cache/ 目录。
+
+go3 参考: staticExtensions map + MD5 cache
+"""
import hashlib
+import logging
+import os
import threading
import time
-from collections import OrderedDict
from typing import Optional
+logger = logging.getLogger("ao3-cache")
-class LRUCache:
- """Thread-safe LRU cache with TTL support."""
+CACHE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "cache")
- def __init__(self, capacity: int = 2000, default_ttl: int = 30):
- self.capacity = capacity
- self.default_ttl = default_ttl
- self._cache: OrderedDict[str, tuple[float, bytes, dict, int]] = OrderedDict()
- # (expiry_time, body, headers, status)
- self._lock = threading.RLock()
+# Only cache these extensions — everything else passes through
+STATIC_EXTENSIONS = {
+ ".css", ".js", ".jpg", ".jpeg", ".png", ".gif", ".ico",
+ ".woff", ".woff2", ".ttf", ".svg", ".webp",
+}
- def _make_key(self, url: str, headers: Optional[dict] = None) -> str:
- """Generate cache key from URL and key headers."""
- # Use URL + relevant headers
- accept = ""
- if headers:
- accept = headers.get("Accept-Encoding", "")
- raw = f"{url}|{accept}"
- return hashlib.md5(raw.encode()).hexdigest()
+# Path prefixes that are always static
+STATIC_PATH_PREFIXES = (
+ "/stylesheets/", "/javascripts/", "/images/", "/media/",
+ "/skins/", "/assets/", "/favicon.ico",
+)
- def get(self, url: str, headers: Optional[dict] = None) -> Optional[tuple[bytes, dict, int]]:
- """Get cached response. Returns (body, headers, status) or None."""
- key = self._make_key(url, headers)
- with self._lock:
- if key not in self._cache:
- return None
- expiry, body, resp_headers, status = self._cache[key]
- if time.time() > expiry:
- del self._cache[key]
- return None
- # Move to end (most recently used)
- self._cache.move_to_end(key)
- return (body, resp_headers, status)
+# Cache TTL: 7 days for static assets (like go3's static_cache_ttl_seconds)
+STATIC_TTL = 604800 # 7 days
- def set(self, url: str, body: bytes, headers: dict, status: int,
- ttl: Optional[int] = None, request_headers: Optional[dict] = None):
- """Store response in cache."""
- key = self._make_key(url, request_headers)
- t = ttl if ttl is not None else self.default_ttl
- expiry = time.time() + t
- with self._lock:
- self._cache[key] = (expiry, body, headers, status)
- self._cache.move_to_end(key)
- if len(self._cache) > self.capacity:
- self._cache.popitem(last=False)
+def is_static_path(path: str) -> bool:
+ """Check if a URL path points to a static asset."""
+ path_lower = path.lower()
+ for ext in STATIC_EXTENSIONS:
+ if path_lower.endswith(ext):
+ return True
+ for prefix in STATIC_PATH_PREFIXES:
+ if path_lower.startswith(prefix):
+ return True
+ return False
- def invalidate(self, url: str, headers: Optional[dict] = None):
- """Remove a specific URL from cache."""
- key = self._make_key(url, headers)
- with self._lock:
- self._cache.pop(key, None)
- def clear(self):
- with self._lock:
- self._cache.clear()
+class StaticCache:
+ """Disk-based static file cache. No LRU, no TTL on dynamic content."""
- @property
- def size(self) -> int:
- with self._lock:
- return len(self._cache)
+ def __init__(self):
+ os.makedirs(CACHE_DIR, exist_ok=True)
+ self._lock = threading.Lock()
+ self._hits = 0
+ self._misses = 0
+
+ def _cache_path(self, url: str) -> str:
+ h = hashlib.md5(url.encode()).hexdigest()
+ return os.path.join(CACHE_DIR, h)
+
+ def get(self, url: str) -> Optional[tuple[bytes, dict, int]]:
+ """Get cached static file. Returns (body, headers, status) or None."""
+ if not is_static_path(url):
+ return None
+ path = self._cache_path(url)
+ try:
+ with self._lock:
+ if os.path.exists(path):
+ mtime = os.path.getmtime(path)
+ if time.time() - mtime < STATIC_TTL:
+ with open(path, "rb") as f:
+ body = f.read()
+ self._hits += 1
+ # Minimal headers for static content
+ headers = {
+ "Content-Type": _guess_content_type(url),
+ "Cache-Control": f"public, max-age={STATIC_TTL}, immutable",
+ }
+ return (body, headers, 200)
+ else:
+ os.remove(path)
+ except Exception:
+ pass
+ self._misses += 1
+ return None
+
+ def set(self, url: str, body: bytes, headers: dict, status: int):
+ """Store static file on disk."""
+ if not is_static_path(url):
+ return
+ if status != 200:
+ return
+ path = self._cache_path(url)
+ try:
+ with self._lock:
+ with open(path, "wb") as f:
+ f.write(body)
+ except Exception as e:
+ logger.debug(f"Cache write failed: {e}")
def get_stats(self) -> dict:
with self._lock:
- return {
- "size": len(self._cache),
- "capacity": self.capacity,
- "usage_pct": round(len(self._cache) / self.capacity * 100, 1) if self.capacity else 0,
- }
+ try:
+ files = os.listdir(CACHE_DIR)
+ total_size = sum(
+ os.path.getsize(os.path.join(CACHE_DIR, f))
+ for f in files
+ if os.path.isfile(os.path.join(CACHE_DIR, f))
+ )
+ except Exception:
+ files = []
+ total_size = 0
+ return {
+ "size": len(files),
+ "capacity": "unlimited",
+ "usage_pct": round(total_size / (1024 * 1024), 1),
+ "hits": self._hits,
+ "misses": self._misses,
+ }
-# TTL 策略:不同路径不同缓存时间
-PATH_TTL = {
- "/": 30, # 首页 30s
- "/works": 60, # 作品列表 60s
- "/chapters": 120, # 章节内容 120s
- "/series": 60, # 系列 60s
- "/collections": 60,
- "/tags": 60,
- "/users": 30,
- "/pseuds": 30,
- "/bookmarks": 60,
- "/skins": 300, # CSS 皮肤缓存 5 分钟
- "/stylesheets": 300,
- "/images": 600, # 图片缓存 10 分钟
- "/media": 600,
- "/javascripts": 300,
- "/api": 15, # API 响应 15s
- "/external_links": 30,
- # Default: 30s
-}
+def _guess_content_type(url: str) -> str:
+ url_lower = url.lower()
+ if url_lower.endswith(".css"):
+ return "text/css; charset=utf-8"
+ if url_lower.endswith(".js"):
+ return "application/javascript; charset=utf-8"
+ if url_lower.endswith(".png"):
+ return "image/png"
+ if url_lower.endswith(".jpg") or url_lower.endswith(".jpeg"):
+ return "image/jpeg"
+ if url_lower.endswith(".gif"):
+ return "image/gif"
+ if url_lower.endswith(".svg"):
+ return "image/svg+xml"
+ if url_lower.endswith(".ico"):
+ return "image/x-icon"
+ if url_lower.endswith(".woff"):
+ return "font/woff"
+ if url_lower.endswith(".woff2"):
+ return "font/woff2"
+ if url_lower.endswith(".ttf"):
+ return "font/ttf"
+ if url_lower.endswith(".webp"):
+ return "image/webp"
+ return "application/octet-stream"
def get_ttl_for_path(path: str) -> int:
- """Determine cache TTL based on URL path."""
- for prefix, ttl in PATH_TTL.items():
- if path.startswith(prefix):
- return ttl
- return 30 # default
+ """Only used for static path TTL. Returns 0 for dynamic paths."""
+ return STATIC_TTL if is_static_path(path) else 0
-# 全局缓存实例
-_cache: Optional[LRUCache] = None
+# ─── Singleton ──────────────────────────────────────────────────────────────
+
+_cache: Optional[StaticCache] = None
-def get_cache() -> LRUCache:
+def get_cache() -> StaticCache:
global _cache
if _cache is None:
- _cache = LRUCache(capacity=5000, default_ttl=30)
+ _cache = StaticCache()
return _cache
diff --git a/cache/0b56e3783141862ec51eaed10f75076a b/cache/0b56e3783141862ec51eaed10f75076a
new file mode 100644
index 0000000..47480a6
--- /dev/null
+++ b/cache/0b56e3783141862ec51eaed10f75076a
@@ -0,0 +1,8 @@
+/*!
+ * JavaScript Cookie v2.2.1
+ * https://github.com/js-cookie/js-cookie
+ *
+ * Copyright 2006, 2015 Klaus Hartl & Fagner Brack
+ * Released under the MIT license
+ */
+!function(e){var n;if("function"==typeof define&&define.amd&&(define(e),n=!0),"object"==typeof exports&&(module.exports=e(),n=!0),!n){var t=window.Cookies,o=window.Cookies=e();o.noConflict=function(){return window.Cookies=t,o}}}(function(){function e(){for(var e=0,n={};e
.module {
+ width: 48.5%
+}
+
+.logged-in .splash > div:nth-of-type(odd) {
+ margin-left: 0;
+ margin-right: 1.5%;
+}
+
+.logged-in .splash > div:nth-of-type(even) {
+ margin-left: 1.5%;
+ margin-right: 0;
+}
+
+/* 18 zone searchbrowse */
+
+form.filters {
+ width: auto;
+ min-width: 23%;
+ max-width: 24%;
+}
+
+.filters fieldset {
+ margin-right: 0;
+}
+
+form.filters dl {
+ margin-left: 0.25em;
+ margin-right: 0.25em;
+}
+
+/* 21 userstuff */
+
+#workskin {
+ margin: auto 1.5%;
+}
diff --git a/cache/17342e82684c267ba4f1e531738eb513 b/cache/17342e82684c267ba4f1e531738eb513
new file mode 100644
index 0000000..aa21dc5
Binary files /dev/null and b/cache/17342e82684c267ba4f1e531738eb513 differ
diff --git a/cache/2f7d84701e69c2978451a4693dbf6b35 b/cache/2f7d84701e69c2978451a4693dbf6b35
new file mode 100644
index 0000000..35a2b24
Binary files /dev/null and b/cache/2f7d84701e69c2978451a4693dbf6b35 differ
diff --git a/cache/306b4df560405cad691843c852971d0a b/cache/306b4df560405cad691843c852971d0a
new file mode 100644
index 0000000..4260581
--- /dev/null
+++ b/cache/306b4df560405cad691843c852971d0a
@@ -0,0 +1 @@
+function setupFilterToggles(){var e=$j(".filters").find("dt.filter-toggle");e.each(function(){var e=$j(this).next().attr("id");$j(this).wrapInner(' ')}),$j("dt.tags button").on("click",function(){"false"==$j(this).attr("aria-expanded")?$j(this).attr("aria-expanded","true"):$j(this).attr("aria-expanded","false")})}function showFilters(){var e=$j(".filters").find("dd.expandable");e.each(function(e,t){var a=$j(t).find("input").filter('[value]:not([value=""])'),r=$j(t).attr("id"),n=$j("#toggle_"+r),i=$j('[aria-controls="'+r+'"]');a.each(function(e,a){$j(a).is(':checked, [type="text"]')&&($j(t).removeClass("hidden"),$j(n).removeClass("collapsed").addClass("expanded"),$j(i).attr("aria-expanded","true"))})})}function setupNarrowScreenFilters(){var e=$j("form.filters"),t=$j("#outer"),a=$j("#go_to_filters"),r=$j("#leave_filters");a.click(function(a){a.preventDefault(),e.removeClass("narrow-hidden"),t.addClass("filtering"),e.find(":focusable").first().focus(),e.trap()}),r.click(function(r){r.preventDefault(),t.removeClass("filtering"),e.addClass("narrow-hidden"),a.focus()})}$j(document).ready(function(){setupFilterToggles(),showFilters(),setupNarrowScreenFilters()});
diff --git a/cache/331a4a4d15181598c366618812c89376 b/cache/331a4a4d15181598c366618812c89376
new file mode 100644
index 0000000..5b3ae4d
--- /dev/null
+++ b/cache/331a4a4d15181598c366618812c89376
@@ -0,0 +1,76 @@
+/* MEDIA: speech */
+
+.landmark {
+ opacity: 1;
+ height: auto;
+ width: auto;
+ font-size: 100%;
+ line-height: 0;
+ color: black;
+}
+
+a.tag {
+ speak: no-punctuation;
+}
+
+em {
+ voice-stress: moderate;
+}
+
+strong, .warnings, .caution {
+ voice-stress: strong;
+}
+
+cite {
+ voice-rate: 95%;
+}
+
+acronym {
+ speak: spell-out;
+}
+
+abbr {
+ content: attr(title);
+}
+
+code, kbd, tt, samp {
+ voice-rate: 95%;
+}
+
+.userstuff {
+ speak-numeral: continuous;
+}
+
+.meta {
+ voice-rate: 110%;
+}
+
+.blurb {
+ voice-rate: 105%;
+}
+
+.userstuff h1, .userstuff h2.title {
+ voice-rate: 90%;
+ pause: 5ms 10ms;
+}
+
+.chapter {
+ pause: 10ms 10ms;
+}
+
+.userstuff dt {
+ voice-balance: leftwards;
+}
+
+.userstuff dd {
+ voice-balance: rightwards;
+}
+
+.navigation {
+ voice-stress: moderate;
+ speak: no-punctuation;
+}
+
+.splash .browse li a:before {
+ speak: none;
+}
diff --git a/cache/34a22b671f9a37f8e9c84a7871acf424 b/cache/34a22b671f9a37f8e9c84a7871acf424
new file mode 100644
index 0000000..e017f95
--- /dev/null
+++ b/cache/34a22b671f9a37f8e9c84a7871acf424
@@ -0,0 +1 @@
+!function(o){"use strict";var n="[data-toggle=dropdown]",t=function(n){var t=o(n).on("click.dropdown.data-api",this.toggle);o("html").on("click.dropdown.data-api",(function(){t.parent().removeClass("open")}))};function e(){o(n).each((function(){d(o(this)).removeClass("open")}))}function d(n){var t,e=n.attr("data-target");return e||(e=(e=n.attr("href"))&&/#/.test(e)&&e.replace(/.*(?=#[^\s]*$)/,"")),(t=e&&o(e))&&t.length||(t=n.parent()),t}t.prototype={constructor:t,toggle:function(n){var t,r,i=o(this);if(!i.is(".disabled, :disabled"))return r=(t=d(i)).hasClass("open"),e(),r?(t.children("ul").hide(),i.blur()):(t.toggleClass("open").children("ul").removeAttr("style"),i.focus()),i.focus(),!1},keydown:function(t){var e,r,i,a,s;if(/(38|40|27)/.test(t.keyCode)&&(e=o(this),t.preventDefault(),t.stopPropagation(),!e.is(".disabled, :disabled"))){if(!(a=(i=d(e)).hasClass("open"))||a&&27==t.keyCode)return 27==t.which&&i.find(n).focus(),e.click();(r=o("ul.menu li:not(.divider):visible a",i)).length&&(s=r.index(r.filter(":focus")),38==t.keyCode&&s>0&&s--,40==t.keyCode&&s
+All rights reserved.
+
+Official repository: https://github.com/julienw/jquery-trap-input
+License is there: https://github.com/julienw/jquery-trap-input/blob/master/LICENSE
+This is version 1.2.0.
+*/(function(e,t){function r(e){if(e.keyCode===9){var t=!!e.shiftKey;if(i(this,e.target,t)){e.preventDefault();e.stopPropagation()}}}function i(e,t,n){var r=a(e),i=t,s,o,u,f;do{s=r.index(i);o=s+1;u=s-1;f=r.length-1;switch(s){case-1:return false;case 0:u=f;break;case f:o=0;break}if(n){o=u}i=r.get(o);if(!i||i===t){return true}try{i.focus()}catch(l){return true}}while(r.length>1&&t===t.ownerDocument.activeElement);return true}function s(){return this.tabIndex>0}function o(){return!this.tabIndex}function u(e,t){return e.t-t.t||e.i-t.i}function a(t){var n=e(t);var r=[],i=0;h.enable&&h.enable();n.find("a[href], link[href], [draggable=true], [contenteditable=true], :input:enabled, [tabindex=0]").filter(":visible").filter(o).each(function(e,t){r.push({v:t,t:0,i:i++})});n.find("[tabindex]").filter(":visible").filter(s).each(function(e,t){r.push({v:t,t:t.tabIndex,i:i++})});h.disable&&h.disable();r=e.map(r.sort(u),function(e){return e.v});return e(r)}function f(){this.keydown(r);this.data(n,true);return this}function l(){this.unbind("keydown",r);this.removeData(n);return this}function c(){return!!this.data(n)}var n="trap.isTrapping";e.fn.extend({trap:f,untrap:l,isTrapping:c});var h={};if(e.find.find&&e.find.attr!==e.attr){(function(){function i(e){var r=e.getAttributeNode(n);return r&&r.specified?parseInt(r.value,10):t}function s(){r[n]=r.tabIndex=i}function o(){delete r[n];delete r.tabIndex}var n="tabindex";var r=e.expr.attrHandle;h={enable:s,disable:o}})()}})(jQuery);
\ No newline at end of file
diff --git a/cache/36f64a27d9f23f25aeaee0dba47e0872 b/cache/36f64a27d9f23f25aeaee0dba47e0872
new file mode 100644
index 0000000..06f3845
--- /dev/null
+++ b/cache/36f64a27d9f23f25aeaee0dba47e0872
@@ -0,0 +1,7 @@
+/**
+ * Copyright (c) 2007 Ariel Flesler - aflesler ○ gmail • com | https://github.com/flesler
+ * Licensed under MIT
+ * @author Ariel Flesler
+ * @version 2.1.2
+ */
+;(function(f){"use strict";"function"===typeof define&&define.amd?define(["jquery"],f):"undefined"!==typeof module&&module.exports?module.exports=f(require("jquery")):f(jQuery)})(function($){"use strict";function n(a){return!a.nodeName||-1!==$.inArray(a.nodeName.toLowerCase(),["iframe","#document","html","body"])}function h(a){return $.isFunction(a)||$.isPlainObject(a)?a:{top:a,left:a}}var p=$.scrollTo=function(a,d,b){return $(window).scrollTo(a,d,b)};p.defaults={axis:"xy",duration:0,limit:!0};$.fn.scrollTo=function(a,d,b){"object"=== typeof d&&(b=d,d=0);"function"===typeof b&&(b={onAfter:b});"max"===a&&(a=9E9);b=$.extend({},p.defaults,b);d=d||b.duration;var u=b.queue&&1=f[g]?0:Math.min(f[g],n));!a&&1 25 || list.data("force-contract")) {
+ list.hide();
+ $(this).show();
+ } else {
+ // show the shuffle and contract button only
+ $(this).nextAll(".shuffle").show();
+ $(this).next(".contract").show();
+ }
+
+ // set up click event to expand the list
+ $(this).click(function(event){
+ list = $($(this).data("action-target"));
+ list.show();
+
+ // show the contract & shuffle buttons and hide us
+ $(this).next(".contract").show();
+ $(this).nextAll(".shuffle").show();
+ $(this).hide();
+ });
+ });
+
+ $(".contract").each(function(){
+ $(this).click(function(event){
+ // hide the list when clicked
+ list = $($(this).data("action-target"));
+ list.hide();
+
+ // show the expand and shuffle buttons and hide us
+ $(this).prev(".expand").show();
+ $(this).nextAll(".shuffle").hide();
+ $(this).hide();
+ });
+ });
+
+ $(".shuffle").each(function(){
+ // shuffle the list's children when clicked
+ $(this).click(function(event){
+ list = $($(this).data("action-target"));
+ list.children().shuffle();
+ });
+ });
+
+ $(".expand_all").each(function(){
+ target = "." + $(this).data("target-class");
+ $(this).click(function(event) {
+ $(this).closest(target).find(".expand").click();
+ });
+ });
+
+ $(".contract_all").each(function(){
+ target = "." + $(this).data("target-class");
+ $(this).click(function(event) {
+ $(this).closest(target).find(".contract").click();
+ });
+ });
+});
+
+// check all or none within the parent fieldset, optionally with a string to match on the id attribute of the checkboxes
+// stored in the "data-checkbox-id-filter" attribute on the all/none links.
+// allow for some flexibility by checking the next and previous fieldset if the checkboxes aren't in this one
+jQuery(function($){
+ $('.check_all').each(function(){
+ $(this).click(function(event){
+ var filter = $(this).data('checkbox-id-filter');
+ var checkboxes;
+ if (filter) {
+ checkboxes = $(this).closest('fieldset').find('input[id*="' + filter + '"][type="checkbox"]');
+ } else {
+ checkboxes = $(this).closest("fieldset").find(':checkbox');
+ if (checkboxes.length == 0) {
+ checkboxes = $(this).closest("fieldset").next().find(':checkbox');
+ if (checkboxes.length == 0) {
+ checkboxes = $(this).closest("fieldset").prev().find(':checkbox');
+ }
+ }
+ }
+ checkboxes.prop('checked', true);
+ event.preventDefault();
+ });
+ });
+
+ $('.check_none').each(function(){
+ $(this).click(function(event){
+ var filter = $(this).data('checkbox-id-filter');
+ var checkboxes;
+ if (filter) {
+ checkboxes = $(this).closest('fieldset').find('input[id*="' + filter + '"][type="checkbox"]');
+ } else {
+ checkboxes = $(this).closest("fieldset").find(':checkbox');
+ if (checkboxes.length == 0) {
+ checkboxes = $(this).closest("fieldset").next().find(':checkbox');
+ if (checkboxes.length == 0) {
+ checkboxes = $(this).closest("fieldset").prev().find(':checkbox');
+ }
+ }
+ }
+ checkboxes.prop('checked', false);
+ event.preventDefault();
+ });
+ });
+});
+
+// Set up open and close toggles for a given object
+// Typical setup (this will leave the toggled item open for users without javascript but hide the controls from them):
+// Open Foo
+//
+//
+// Notes:
+// - The open button CANNOT be inside the toggled div, the close button can be (but doesn't have to be)
+// - You can have multiple open and close buttons for the same div since those are labeled with classes
+// - You don't have to use div and a, those are just examples. Anything you put the toggled and _open/_close classes on will work.
+// - If you want the toggled item not to be visible to users without JavaScript by default, add the class "hidden" to the toggled item as well.
+// (and you can then add an alternative link for them using )
+// - Generally reserved for toggling complex elements like bookmark forms and challenge sign-ups; for simple elements like lists use setupAccordion.
+function setupToggled(){
+ $j('.toggled').filter(function(){
+ return $j(this).closest('.userstuff').length === 0;
+ }).each(function(){
+ var node = $j(this);
+ var open_toggles = $j('.' + node.attr('id') + "_open");
+ var close_toggles = $j('.' + node.attr('id') + "_close");
+
+ if (node.hasClass('open')) {
+ close_toggles.each(function(){$j(this).show();});
+ open_toggles.each(function(){$j(this).hide();});
+ } else {
+ node.hide();
+ close_toggles.each(function(){$j(this).hide();});
+ open_toggles.each(function(){$j(this).show();});
+ }
+
+ open_toggles.each(function(){
+ $j(this).click(function(e){
+ if ($j(this).attr('href') == '#') {e.preventDefault();}
+ node.show();
+ open_toggles.each(function(){$j(this).hide();});
+ close_toggles.each(function(){$j(this).show();});
+ });
+ });
+
+ close_toggles.each(function(){
+ $j(this).click(function(e){
+ if ($j(this).attr('href') == '#') {e.preventDefault();}
+ node.hide();
+ close_toggles.each(function(){$j(this).hide();});
+ open_toggles.each(function(){$j(this).show();});
+ });
+ });
+ });
+}
+
+function hideHideMe() {
+ $j('.hideme').each(function() { $j(this).hide(); });
+}
+
+function showShowMe() {
+ $j('.showme').each(function() { $j(this).show(); });
+}
+
+function handlePopUps() {
+ $j("a[data_popup]").click(function(event, element) {
+ if (event.stopped) return;
+ window.open($j(element).attr('href'));
+ event.stop();
+ });
+}
+
+// used in nested form fields for deleting a nested resource
+// see prompt form for example
+function remove_section(link, class_of_section_to_remove) {
+ $j(link).siblings(":input[type=hidden]").val("1"); // relies on the "_destroy" field being the nearest hidden field
+ var section = $j(link).closest("." + class_of_section_to_remove);
+ section.find(".required input, .required textarea").each(function(index) {
+ var element = eval('validation_for_' + $j(this).attr('id'));
+ element.disable();
+ });
+ section.hide();
+}
+
+// used with nested form fields for dynamically stuffing in an extra partial
+// see challenge signup form and prompt form for an example
+function add_section(link, nested_model_name, content) {
+ // get the right new_id which should be in a div with class "last_id" at the bottom of
+ // the nearest section
+ var last_id = parseInt($j(link).parent().siblings('.last_id').last().html());
+ var new_id = last_id + 1;
+ var regexp = new RegExp("new_" + nested_model_name, "g");
+ content = content.replace(regexp, new_id);
+ // kludgy: show the hidden remove_section link (we don't want it showing for non-js users)
+ content = content.replace('class="hidden showme"', '');
+ $j(link).parent().before(content);
+}
+
+// An attempt to replace the various work form toggle methods with a more generic one
+function toggleFormField(element_id) {
+ var ticky = $j('#' + element_id + '-show');
+ if (ticky.is(':checked')) {
+ $j('#' + element_id).removeClass('hidden');
+ }
+ else {
+ $j('#' + element_id).addClass('hidden');
+ if (element_id != 'chapters-options' && element_id != 'backdate-options') {
+ $j('#' + element_id).find(':input[type!="hidden"]').each(function(index, d) {
+ if ($j(d).attr('type') == "checkbox") {$j(d).attr('checked', false);}
+ else {$j(d).val('');}
+ });
+ }
+ }
+ // We want to check this whether the ticky is checked or not
+ if (element_id == 'chapters-options') {
+ var item = document.getElementById('work_wip_length');
+ if (item.value == 1 || item.value == '1') {item.value = '?';}
+ else {item.value = 1;}
+ }
+}
+
+// Hides expandable form field options if Javascript is enabled
+function hideFormFields() {
+ if ($j('form#work-form') != null) {
+ var toHide = ['#co-authors-options', '#front-notes-options', '#end-notes-options', '#chapters-options',
+ '#parent-options', '#series-options', '#backdate-options', '#override_tags-options'];
+
+ $j.each(toHide, function(index, name) {
+ if ($j(name)) {
+ if (!($j(name + '-show').is(':checked'))) { $j(name).addClass('hidden'); }
+ }
+ });
+ $j('form#work-form').className = $j('form#work-form').className;
+ }
+}
+
+// Hides the extra checkbox fields in prompt form
+function hideField(id) {
+ $j('#' + id).toggle();
+}
+
+function attachCharacterCounters() {
+ var countFn = function() {
+ var counter = (function(input) {
+ /* Character-counted inputs do not always have the same hierarchical relationship
+ to their associated counter elements in the DOM, and some cc-inputs have
+ duplicate ids. So search for the input's associated counter element first by id,
+ then by checking the input's siblings, then by checking its cousins. */
+ var cc = $j('.character_counter [id='+input.attr('id')+'_counter]');
+ if (cc.length === 1) { return cc; } // id search, use attribute selector rather
+ // than # to check for duplicate ids
+
+ cc = input.nextAll('.character_counter').first().find('.value'); // sibling search
+ if (cc.length) { return cc; }
+
+ var parent = input.parent(); // 2 level cousin search
+ for (var i = 0; i < 2; i++) {
+ cc = parent.nextAll('.character_counter').find('.value');
+ if (cc.length) { return cc; }
+ parent = parent.parent();
+ }
+
+ return $j(); // return empty jquery element if search found nothing
+ })($j(this)),
+ max = parseInt(counter.attr('data-maxlength'), 10),
+ val = $j(this).val().replace(/\r\n/g,'\n').replace(/\r|\n/g,'\r\n'),
+ remaining = max - val.length;
+
+ counter.html(remaining).attr("aria-valuenow", remaining);
+ };
+
+ $j(document).on('keyup keydown mouseup mousedown change', '.observe_textlength', countFn);
+ $j('.observe_textlength').each(countFn);
+}
+
+// add attributes that are only needed in the primary menus and when JavaScript is enabled
+function setupDropdown(){
+ $j('#header').find('.dropdown').attr("aria-haspopup", true);
+ $j('#header').find('.dropdown, .dropdown .actions').children('a').attr({
+ 'class': 'dropdown-toggle',
+ 'data-toggle': 'dropdown',
+ 'data-target': '#'
+ });
+ $j('.dropdown').find('.menu').addClass("dropdown-menu");
+}
+
+// Accordion-style collapsible widgets
+// The pane element can be shown or hidden using the expander (link)
+// Apply hidden to the pane element if it shouldn't be visible when JavaScript is disabled
+// Typical set up:
+//
+// Expander
+//
+// foo!
+//
+//
+function setupAccordion() {
+ $j(".expandable").filter(function() {
+ return $j(this).closest(".userstuff").length === 0;
+ }).each(function() {
+ var pane = $j(this);
+ // hide the pane element if it's not hidden by default
+ if ( !pane.hasClass("hidden") ) {
+ pane.addClass("hidden");
+ };
+
+ // make the expander visible
+ // add the default collapsed state
+ // make it do the expanding and collapsing
+ pane.prev().removeClass("hidden").addClass("collapsed").click(function(e) {
+ var expander = $j(this);
+ if (expander.attr('href') == '#') {
+ e.preventDefault();
+ };
+
+ // change the classes upon clicking the expander
+ expander.toggleClass("collapsed").toggleClass("expanded").next().toggleClass("hidden");
+ });
+ });
+}
+
+// Remove the /confirm_delete portion of delete links so user who have JS enabled will
+// be able to delete items via hyperlink (per rails/jquery-ujs) rather than a dedicated
+// form page.
+function prepareDeleteLinks() {
+ $j('a[href$="/confirm_delete"][data-confirm]').each(function(){
+ this.href = this.href.replace(/\/confirm_delete$/, "");
+ $j(this).attr("data-method", "delete");
+ });
+
+ // Removing non-default orphan_account pseuds from works
+ $j('a[href$="/confirm_remove_pseud"][data-confirm]').each(function() {
+ this.href = this.href.replace(/\/confirm_remove_pseud$/, "/remove_pseud");
+ $j(this).attr("data-method", "put");
+ });
+
+ // For purging assignments in gift exchanges. This is only on one page and easy to
+ // check, so don't worry about adding a fallback data-confirm message.
+ $j('a[href$="/confirm_purge"][data-confirm]').each(function() {
+ this.href = this.href.replace(/\/confirm_purge$/, "/purge");
+ $j(this).attr("data-method", "post");
+ });
+}
+
+/// Kudos
+$j(document).ready(function() {
+ $j('input#kudo_submit').on("click", function(event) {
+ event.preventDefault();
+
+ $j.ajax({
+ type: 'POST',
+ url: '/kudos.js',
+ data: jQuery('#new_kudo').serialize(),
+ error: function(jqXHR, textStatus, errorThrown) {
+ var msg = 'Sorry, we were unable to save your kudos';
+
+ // When we hit the rate limit, the response from Rack::Attack is a plain text 429.
+ if (jqXHR.status == "429") {
+ msg = "Sorry, you can't leave more kudos right now. Please try again in a few minutes.";
+ } else {
+ var data = $j.parseJSON(jqXHR.responseText);
+ if (data.error_message) {
+ msg = data.error_message;
+ }
+ }
+
+ $j('#kudos_message').addClass('kudos_error').text(msg);
+ },
+ success: function(data) {
+ $j('#kudos_message').addClass('notice').text('Thank you for leaving kudos!');
+ }
+ });
+ });
+
+ // Scroll to the top of the comments section when loading additional pages via Ajax in comment pagination.
+ $j('#comments_placeholder').on('click.rails', '.pagination a[data-remote]', function(e){
+ $j.scrollTo('#comments_placeholder');
+ });
+
+ // Scroll to the top of the comments section when loading comments via AJAX
+ $j("#show_comments_link_top").on('click.rails', 'a[href*="show_comments"]', function(e){
+ $j.scrollTo('#comments');
+ });
+});
+
+// For simple forms that appear to toggle between creating and destroying records
+// e.g. favorite tags, subscriptions
+//