Files
BadNote/docs/plans/2026-06-21-badnote-full-refactor.md
Akiba So 3295018ee3
All checks were successful
CI / Windows build (push) Successful in 11m34s
feat(pen): pressure-responsive width, configurable thinning, native Windows pen (tilt/buttons)
W1 — Custom pen width + pressure sensitivity (Saber-style):
- Root cause of "压感没用": perfect_freehand 1.0.4 IGNORES real stylus pressure
  (hardcodes radius=size/2 when simulatePressure=false) — width never tracked pen
  force. Upgraded perfect_freehand ^1.0.0 -> ^2.0.0 (honors real pressure); migrated
  all 5 getStroke call sites to the 2.x API (PointVector / StrokeOptions / Offset).
- De-hardcoded `thinning` into `kDefaultPenThinning` (0.85), single source shared by
  the on-screen painter and the PDF export path; exposed as PenConfig.pressureSensitivity
  with a Pressure Sensitivity slider; live-applies via a config listener.

W3 — Native Windows pen plugin (tilt + barrel/eraser buttons):
- windows/runner/pen_channel.{h,cpp}: observe WM_POINTER at the TOP of MessageHandler
  (before HandleTopLevelWindowProc, which Flutter uses to consume pen events), read
  GetPointerPenInfo penFlags + tilt, stream over EventChannel('badnote/pen'); non-consuming.
- PenInputService: single latched hardware state (no Win32-pointerId<->event.pointer
  correlation); graceful no-op off-Windows.
- pen_canvas maps barrel/inverted/eraser through PenConfig.sideButton/eraserEnd
  (eraser/undo/toggleTool/pan) and captures tilt into PenPoint.tilt -> EditorPoint.tilt.

W2 — Zoom flicker: page raster isolated in its own RepaintBoundary (safe interim);
definitive crisp-on-zoom fix gated on the on-device root-cause probe (plan M3).

Plans: ralplan-consensus plan at docs/plans/2026-06-22-badnote-pen-polish.md
(Architect APPROVE-WITH-MUST-FIX M1-M4 + Critic ITERATE->APPROVE).

Tests: 58/58 pass incl. shared-thinning invariant + thinning-affects-outline +
tilt-adapter round-trip. flutter analyze clean; linux debug build OK.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 02:10:05 +08:00

447 lines
62 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# BadNote — Full Refactor + Feature Roadmap (own-canvas engine)
**Status:** PLAN (ralplan consensus — Architect APPROVE-WITH-MUST-FIX + Critic-ITERATE fixes applied 2026-06-21: MF1MF3 + P0.5 slice + SF1SF5 + cache-split / diff-write-durability / erase-is-new-behavior / render-handle-lifecycle; pending final Critic confirm)
**Date:** 2026-06-21
**Mode:** DELIBERATE (high-risk: engine generalization, 60fps continuous/double-page, OCR/CAS feasibility, server/AI scope)
**Owner plan file:** `docs/plans/2026-06-21-badnote-full-refactor.md`
**Supersedes (in part):** `docs/plans/2026-06-21-badnote-phase1.md`
> **Grounding.** Written after reading the live code, not from memory. Verified sources:
> live editor `lib/editor/canvas/{pen_canvas,ink_painters,pen_stroke,pen_editor_screen}.dart`,
> `lib/editor/pdf/pen_capture_region.dart`, `lib/main.dart` (dynamic_color + `pdfrxFlutterInitialize` + `PenCaptureBinding`),
> `lib/services/{database_service,pdf_service,stroke_rasterizer,undo_manager}.dart`,
> `lib/services/ocr/onnx_recognition_backend.dart`, `lib/models/{ink_stroke,ink_point,pressure_curve}.dart`,
> `lib/providers/settings_provider.dart`, `lib/screens/home_screen.dart`,
> `server/badnote_server/{main,models,routers/sync_router}.py`, `server/README.md`,
> `pubspec.yaml`, existing `test/` + `integration_test/` + `tool/` assets.
>
> **CRITICAL DIRECTION CHANGE captured here (vs phase1 plan):** the live `lib/editor/canvas/` has **already abandoned** the phase-1 "pdfrx `pageOverlaysBuilder`-hosted ink + RenderProxyBox arena bypass" architecture. The live `PenCanvas` owns **one `InteractiveViewer` + a single `Listener`** that draws a `PdfPageView` *bitmap* and the ink in the **same** child subtree (Saber clean-room model). pdfrx is used only as a **page renderer / document API** (`PdfDocument.openFile`, `PdfPageView`), never for gestures. `PenCaptureRegion`/`PenCaptureBinding` are installed in `main()` but are **NOT wired into the live canvas** and are slated for retirement (§6, §9). This plan builds the full product on the **own-canvas** model and records that decision in the ADR (§10).
---
## 0. Vision & Scope
**BadNote** is a pen-first, performant note-taking app. Primary device: **Windows Tablet + Surface Pen** (Flutter ≥ 3.44 for WM_POINTER stylus/multitouch); desktop for review/复习/search. Differentiators the user explicitly wants to win on: **(1)** library-wide full-text search over PDF text + typed text + handwriting OCR with jump-to-location; **(2)** modern cohesive **Material You** UI; **(3)** truly **book-like reading** (page-flip, two-page spread, thumbnail grid, reader-vs-annotate modes). Plus a **双链 sticky-note infinite board**, **one-notebook-per-PDF** with insertable blank pages and never-rasterized source, and a later **server sync + AI refinement** and **ink CAS**.
**Existing user data MAY be reset** — no migration burden; prefer the cleanest schema. **Stack stays Flutter** (own gesture pipeline; pdfrx only renders pages). GFW-aware CI already solved (pdfium pre-fetch + Dev Mode + vendored sqlite3 + onnxruntime pre-fetch).
### 0.1 The 11 feature areas → phase map (full vision; nothing dropped, everything sequenced)
| # | Feature | Phase | Risk |
| --- | --- | --- | --- |
| F1 | Pen-first editor core (harden): pressure, palm rejection, perfect_freehand quality, eraser, tools, undo/redo, **DB persistence of strokes per doc/page** | **P0** | LowMed |
| F2 | Page layout modes: continuous-single/-double, paged-single/-double + switcher; lazy render @60fps | **P1** | **High** |
| F3 | Book-like reading: page-flip, two-page spread, reader vs annotate mode, quick jump | **P1** | Med |
| F4 | Thumbnail-grid navigation (Drawboard-style), slider, no keyboard input | **P1** | Low |
| F5 | Configurable pen: side-button + eraser-end mapping, pressure curve, palm sensitivity, finger-drawing toggle — Pen settings page | **P2** | Med |
| F6 | One-notebook-per-PDF: page-level binding, insert blank pages between PDF pages, keep PDF vector, portable bundle + relink | **P2** | Med |
| F7 | Infinite 便利贴 board + **双链** bidirectional links / backlinks | **P3** | MedHigh |
| F8 | Library-wide full-text search (PDF text + typed text + handwriting OCR) with snippets + jump | **P3** | **High (handwriting OCR)** |
| F9 | Server sync + AI refinement (llm_wiki, VLM/LLM organize) | **P4** | High (scope) |
| F10 | Ink CAS / formula recognition → searchable + solve toggle | **P5** | **Very High** |
| F11 | Modern UX polish: Material You, subtoolbars, drag-reorder thumbnails, hover pre-warm | **woven across P0P3** | Low |
### 0.2 Non-goals per phase
- **P0:** no layout modes, no board, no text boxes, no search UI, no settings page beyond what exists, no server. Single-page editor + persistence + eraser/undo only.
- **P1:** no 双链, no OCR-search, no server, no CAS. Layout/reading/thumbnails only.
- **P2:** no server, no CAS, no handwriting OCR. Pen-config + notebook/bundle only.
- **P3:** no CAS, no server-side AI. Board+双链 and **local** search (incl. best-effort handwriting OCR via existing text-line model; formula deferred to P5).
- **P4:** sync + AI organize; CAS still out.
- **P5:** formula OCR + CAS.
- **All phases:** never rasterize the source PDF; no on-screen keyboard reliance (slider/grid nav).
---
## 1. Target Architecture
### 1.1 Layered module decomposition (target `lib/editor/`)
```
lib/editor/
├─ engine/ # host-agnostic ink engine (pure-ish, testable)
│ ├─ coordinate_space_host.dart # CoordinateSpaceHost + NormalizedPageHost + BoardHost
│ ├─ stroke_model.dart # EditorStroke/EditorPoint (canonical; replaces PenStroke split — §3, §4)
│ ├─ stroke_store.dart # StrokeStore { committed, int revision } (O(1) shouldRepaint gate)
│ ├─ stroke_geometry.dart # buildStrokeOutline(getStroke) — single source for screen+export (kills hairline bug)
│ ├─ stroke_eraser.dart # pure splitStroke / eraseHits (extract from pen_canvas._eraseAt)
│ └─ undo_stack.dart # global, commit-time-ordered, host-tagged (generalize undo_manager)
├─ input/
│ ├─ input_arbiter.dart # pure state machine: pointerCount>=2→pan/zoom, stylus→draw, palm reject
│ └─ pen_config.dart # button/eraser-end action mapping, pressure curve, palm sensitivity (F5)
├─ render/
│ ├─ annotation_layer.dart # Stack(StaticInkPainter, LiveInkPainter, TextBoxLayer) over a host
│ ├─ static_ink_painter.dart # reads ink_picture_cache, keyed by revision
│ ├─ live_ink_painter.dart # in-progress stroke only
│ └─ ink_picture_cache.dart # LRU<revision, ui.Picture> for STATIC INK — resolution-INDEPENDENT, NO DPI bucket (vector ink re-rasterizes crisp at composite)
├─ layout/ # F2/F3 — page layout + reading
│ ├─ page_layout.dart # enum {continuousSingle, continuousDouble, pagedSingle, pagedDouble}
│ ├─ page_viewport.dart # lazy windowed page hosting (visible + cache extent), recenter
│ └─ reader_controller.dart # reader vs annotate mode, page-flip, spread, quick-jump
├─ pdf/
│ ├─ pdf_document_source.dart # wraps PdfDocument (open/dispose/pageSize/render), notebook page-map (F6)
│ ├─ page_tile.dart # PdfPageView host widget (bitmap) sized to host rect; drives tile DPI from transform scale (R11)
│ └─ page_tile_cache.dart # LRU<TileKey{hostId, dpiBucket}, ui.Image> for rendered PAGE BITMAPS — DPI-bucketed (tiles blur on upscale); owns native-handle dispose (R11)
├─ board/ # F7
│ ├─ board_host_pane.dart # infinite InteractiveViewer board, reuses engine
│ └─ link_graph.dart # 双链 backlink model + queries
├─ text/ # OneNote-like editable text boxes (P3+, optional in P2 notebook)
│ ├─ text_box_model.dart
│ └─ text_box_layer.dart
├─ persistence/
│ ├─ editor_repository.dart # batched load/save per document; maps DB ↔ engine
│ └─ save_scheduler.dart # debounced, synchronous-snapshot-before-await
├─ search/ # F8 (P3)
│ ├─ search_indexer.dart # PDF text + typed text + OCR → document_fts/library index
│ └─ ocr_ingest.dart # bridge to lib/services/ocr (text-line; formula later)
├─ sync/ # F9 (P4)
│ └─ sync_client.dart # talks to server/ (already has push/pull/auth)
└─ ui/
├─ editor_screen.dart # top-level editor (replaces pen_editor_screen + pdf_annotator_screen)
├─ editor_toolbar.dart # Material You subtoolbars (F11)
├─ thumbnail_grid.dart # F4 nav
└─ editor_shortcuts.dart # tool numbers, Ctrl+Z/Y/F/S (mouse/desktop)
```
### 1.2 Dependency graph (acyclic)
```
ui/ ─────────────► layout/ ─► pdf/ ─► (pdfrx PdfDocument/PdfPageView)
│ │
│ ├──────► render/ ─► engine/ (geometry, store, hosts)
│ └──────► input/ ─► engine/
├──► board/ ─────────────► render/, engine/
├──► text/ ─────────────► engine/(hosts)
├──► search/ ────────────► persistence/, services/ocr/
├──► sync/ ────────────► persistence/, server API
└──► persistence/ ───────► services/database_service.dart, engine/(stroke_model)
```
**engine/** depends on nothing in editor/ except itself (and `perfect_freehand`). **render/input/layout/** depend on engine. **ui/** is the only place wiring Riverpod controllers. This is the seam that lets PDF-page host, infinite board, and (P5) CAS overlay all reuse one renderer.
### 1.3 Coordinate model — single source of truth
**Truth = host content coordinates** (already the live convention; generalize it):
- **NormalizedPageHost:** points ∈ `[0,1]` of the page rect (today `PenPoint(nx, ny)`); width as fraction of page width (today `PenStroke.width`). Stored per `(documentId, pageIndex)`.
- **BoardHost:** absolute logical px on an unbounded canvas; `InteractiveViewer` supplies pan/zoom.
`CoordinateSpaceHost` exposes `toContent(deviceLocal, deviceSize)`, `toDevice(content, deviceSize)`, `applyContentToCanvas(canvas, deviceSize)`. The live painters already scale `nx*pageSize.width` at paint time — formalize that into `applyContentToCanvas` (a `canvas.scale`) so no transformed-point copies are ever allocated. Text boxes are a **sanctioned widget-space exception** (Positioned widgets multiply by current zoom; stored truth stays content coords for search).
### 1.4 How today's `lib/editor/canvas/` evolves into this (no rewrite)
| Today (live) | Becomes | Action |
| --- | --- | --- |
| `pen_stroke.dart` `PenStroke`/`PenPoint` (in-memory only) | `engine/stroke_model.dart` `EditorStroke`/`EditorPoint` **persistable** (freezed/JSON, normalized) | **generalize + add JSON**; converge with `InkStroke` (§4) |
| `ink_painters.dart` `StaticInkPainter`/`LiveInkPainter` + `buildStrokePath` | `render/static_ink_painter.dart` (+ `render/ink_picture_cache.dart`), `render/live_ink_painter.dart`, `engine/stroke_geometry.dart` | **move + add ink `ui.Picture` cache (resolution-independent, revision-keyed) + revision gate**; `buildStrokePath``buildStrokeOutline` shared with export |
| `pen_canvas.dart` `_PenCanvasState` gesture logic (`_activePointers`, `_shouldDraw`, palm reject, `_eraseAt`) | `input/input_arbiter.dart` (pure SM) + `render/annotation_layer.dart` (widget) | **extract** SM out of the widget; eraser→`engine/stroke_eraser.dart` |
| `pen_editor_screen.dart` single-page + `_strokesByPage` Map + `_transform` + slider | `ui/editor_screen.dart` + `layout/page_viewport.dart` + `persistence/editor_repository.dart` | **generalize** single-page → windowed multi-page; `_strokesByPage` → repository-backed `HostState`s |
| `pen_capture_region.dart` + `PenCaptureBinding` (unused by live canvas) | — | **retire** after P0 confirms the `Listener`-in-shared-transform model holds on device (§6, §9) |
| `pen_editor_screen` single `PdfPageView` bitmap inside the `InteractiveViewer` (1× layout-sized bitmap, matrix-scaled by pinch) | `pdf/page_tile.dart` **multi-resolution tile** whose render DPI tracks the transform scale | **NEW work (not relocation): own multi-resolution tiling** — pinch GPU-upscales a 1× bitmap → blurry at high zoom (R11). On zoom-settle, re-instantiate the tile (or `PdfPage.render()`) at a DPI matching the current scale into `pdf/page_tile_cache.dart` (DPI-bucketed page-bitmap cache, distinct from the ink cache). Budgeted P1/P0.5. |
Net: the four live files are **promoted, not thrown away**. P0 is mostly *relocation + persistence + revision-gated Picture cache*, which is low risk and immediately shippable. The one genuinely **new** burden the own-canvas choice imposes (beyond layout/windowing) is **multi-resolution page-tile rendering** — see R11 (§7) and the P0.5 gate (§5/§9): we own crisp-on-zoom re-rasterization that pdfrx's own viewer would have given for free.
---
## 2. Per-feature design (file-level)
### F1 — Pen-first editor core + persistence (P0)
**Add/modify:**
- `engine/stroke_model.dart` **[ADD]** — `EditorStroke { id, List<EditorPoint> points, EditorTool tool, int color, double width, bool filled, String? textContent, double fontSize }`, `EditorPoint { double x, y; double? pressure; double? tilt; int? timestamp; InputDeviceKind? pointerDeviceKind }`. freezed + `toJson/fromJson`. (Normalized; `width` = fraction of page width, matching live `PenStroke`.) **SF1 — non-lossy superset of `InkStroke`/`InkPoint`:** `InkPoint` carries `tilt`/`timestamp`/`pointerDeviceKind` (database_service.dart:346362) that the live `PenPoint` drops; `EditorPoint` includes them now (nullable, cheap with freezed) so the `InkStroke``EditorStroke` adapter (for OCR/export reuse) is **lossless in both directions**. The live `PenCanvas` simply leaves the extra fields null at capture; existing `InkStroke` data round-trips intact. (Avoids a silent data-loss footgun for OCR, which keys on `pointerDeviceKind`/`pressure`.)
- `engine/stroke_store.dart` **[ADD]** — `class StrokeStore { List<EditorStroke> committed; int revision; add/removeAt/replace bump revision; }`. Fixes the live `pen_editor_screen._commitStroke` "new list identity" hack by making revision explicit.
- `engine/stroke_geometry.dart` **[ADD]** — `Path buildStrokeOutline(EditorStroke, Size, {bool isComplete})` lifted verbatim from `ink_painters.buildStrokePath` (the proven `getStroke(thinning: hl?0:0.7, smoothing:.5, streamline:.5, simulatePressure: !hasRealPressure && !hl)` recipe). **Single source** for screen + export.
- `engine/stroke_eraser.dart` **[ADD]** — pure `eraseHits(strokes, point, radius)` (extract live `_eraseAt`) + `splitStroke` for partial/segment erase. **Critic note — `splitStroke` is NEW behavior, not an extraction:** the live `pen_canvas._eraseAt` is **whole-stroke** (it removes the entire stroke on the first proximity hit and returns). There is no segment-split logic in the live canvas to port — `undo_manager.removeStroke` only *records* replacements, it does not compute them. So `splitStroke` (point-run splitting → 0/1/2 sub-strokes) is greenfield code; its tests (§8) exercise new functionality, not a regression port. `eraseHits` (whole-stroke) IS an extraction of the live behavior and stays available as the default erase mode.
- `engine/undo_stack.dart` **[ADD]** — generalize `lib/services/undo_manager.dart` to host-tagged, commit-time-ordered global stack (entries carry `hostId`+`pageIndex`).
- `render/{annotation_layer,static_ink_painter,live_ink_painter,ink_picture_cache}.dart` **[ADD]** — relocate live painters; `StaticInkPainter` reads `ink_picture_cache` (`LRU<revision, ui.Picture>`) keyed by `revision` (`shouldRepaint = old.revision != revision` — O(1)); per-host `RepaintBoundary`. **`ink_picture_cache` is for STATIC VECTOR INK only — resolution-INDEPENDENT, NO DPI bucket** (a `ui.Picture` of vector strokes re-rasterizes crisp at composite time at any zoom). It is a **different cache from the page-bitmap `page_tile_cache`** (§2/F2, R11), which IS DPI-bucketed because raster page tiles blur on upscale. Do not conflate the two.
- `input/input_arbiter.dart` **[ADD]** — pure SM extracted from `pen_canvas` (`idle→inking→erasing`, `pointerCount>=2`→cancel+pan, palm reject = touch dropped while stylus active, eraser = `kSecondaryButton || invertedStylus`). Keeps the live pressure-normalization logic (`_normalizedPressure`).
- `persistence/editor_repository.dart` **[ADD]** — `loadDocument(documentId) → Map<int,StrokeStore>` (one batched query), `saveHost(documentId, hostId, EditorStroke list)`.
- **MF3 — write contract (load-bearing for the per-row choice):** `saveHost` MUST **diff by stroke id** against the rows already on disk for that host — **UPSERT only changed/new rows, DELETE only removed rows**. It must NOT delete-all-rows-for-host then re-insert (that is exactly what the live notes path does at database_service.dart:281306, and at 2,000 strokes it is **slower** than a single blob rewrite — re-inserting 2,000 rows per save). The per-stroke-row schema (§3) is justified **only** under this diff contract: erasing 1 of 2,000 strokes ⇒ 1 DELETE, 0 re-inserts; adding 1 stroke ⇒ 1 INSERT. The `SaveScheduler` hands `saveHost` the synchronously-captured stroke snapshot; the repository keeps a last-persisted id-set per host to compute the diff. If diffing proves fiddly under churn, the fallback is a per-page blob (NOT delete-all+reinsert) — but the diff path is the default and is what makes per-row worthwhile.
- **Durability invariant (Critic):** the UPSERT(s) + DELETE(s) for one `saveHost` MUST run inside **one `sqflite` transaction**, and the in-memory **per-host last-persisted id-set is updated ONLY after that transaction commits** (in the `then`/post-await success path) — never optimistically before the write. A crash or interruption mid-diff must leave memory and disk consistent: either the whole diff applied (and the id-set advances) or none of it did (and the id-set is unchanged, so the next save re-derives the same diff and retries). This mirrors the existing `updateNote` interruption warning (database_service.dart:281306: "an interruption mid-way would permanently lose strokes, so the whole sequence must run inside one transaction").
- `persistence/save_scheduler.dart` **[ADD]** — debounced ~800ms; **serialize JSON synchronously before any await**; flush on page-leave/dispose.
- `ui/editor_screen.dart`, `ui/editor_toolbar.dart` **[ADD]** — port `pen_editor_screen` UI (floating Material You palette + page pill already good).
**Public interfaces (key):**
```dart
abstract class CoordinateSpaceHost {
String get hostId; int get pageIndex;
Offset toContent(Offset deviceLocal, Size deviceSize);
Offset toDevice(Offset content, Size deviceSize);
void applyContentToCanvas(Canvas c, Size deviceSize);
}
class EditorController extends ChangeNotifier { // ui-facing, Riverpod-provided
void beginStroke(CoordinateSpaceHost h, EditorPoint p);
void extendStroke(EditorPoint p);
void commitStroke(); // getStroke once → append → revision++ → undo push → scheduleSave
void eraseAt(CoordinateSpaceHost h, Offset content, double radius);
void undo(); void redo();
void setTool(EditorTool t); void setColor(Color c); void setLayout(PageLayout l);
}
```
**Acceptance:** draw/erase/undo/redo on a page; close+reopen → strokes persisted (DB); `StaticInkPainter.shouldRepaint==false` while drawing a new stroke on a 2,000-stroke page (revision constant); no per-frame point allocation for committed strokes.
### F2 — Page layout modes (P1)
- `layout/page_layout.dart` **[ADD]** — `enum PageLayout { continuousSingle, continuousDouble, pagedSingle, pagedDouble }`.
- `layout/page_viewport.dart` **[ADD]** — windowed lazy hosting: only pages in `[firstVisible - cacheExtent, lastVisible + cacheExtent]` mount an `AnnotationLayer` + `PageTile`; others are disposed (Picture evicted). Continuous = scrollable column/two-column; paged = `PageView`. Recenter on layout switch.
- `pdf/pdf_document_source.dart` **[ADD]** — `PdfDocument` wrapper exposing `pageSize(i)`, `renderTile(...)`, page count; owns dispose. Replaces ad-hoc `PdfDocument.openFile` in `pen_editor_screen`.
- `pdf/page_tile.dart` **[ADD] — multi-resolution tile (R11 mitigation).** Under one shared `InteractiveViewer`, a `PdfPageView` renders a bitmap sized to its **layout constraints × devicePixelRatio** and the matrix scales that 1× bitmap, so pinch-zoom GPU-**upscales** → blurry text/rules at high zoom (unlike pdfrx's own viewer, which re-renders crisp tiles per zoom level). `page_tile` watches the transform scale and on **zoom-settle** (debounced) re-instantiates its render at a DPI matching the current scale — via either a re-laid-out `PdfPageView` at the new pixel size or `PdfPage.render(width/height at target DPI)` into `page_tile_cache`. **Cap retained DPI** (e.g. ≤ 3× base) to bound memory; downscale path stays matrix-only (sharp enough). This is **new own-canvas work**, not relocation.
- `pdf/page_tile_cache.dart` **[ADD]** — `LRU<TileKey{hostId, dpiBucket}, ui.Image>` of rendered **page bitmaps** (NOT ink). Key includes the **DPI bucket** so a page re-rendered at higher DPI replaces (not duplicates) its lower-DPI tile. **Owns the native-handle dispose lifecycle** of each `PdfPage.render()` result (`PdfImage` → backing `ui.Image`), deferred to post-frame so the raster thread is done with an evicted tile before disposal. **Distinct from `render/ink_picture_cache.dart`** (§2/F1) which holds resolution-independent ink `ui.Picture`s with no DPI bucket.
- `ui/editor_toolbar.dart` **[MODIFY]** — layout switcher control.
**Two separate caches + memory budgets (Critic — do not conflate):**
- **`ink_picture_cache` (vector ink):** `LRU<revision, ui.Picture>`, ~812 mounted-page Pictures, dispose deferred to post-frame. Budget ~ a few MB (vector op-lists are cheap). No DPI bucket.
- **`page_tile_cache` (raster page bitmaps):** `LRU<TileKey, ui.Image>`. **This is the heavy one.** A single A4 page rendered at 3× DPI is ~**1030 MB** (≈ 1785×2526 px × 4 bytes ≈ 18 MB at 3× of a 595×842 pt page @ ~2 dppt). So **812 tiles at 3× would be ~150350 MB — the old "≤64 MB / 812 pages" figure (R10) was unit-confused** (it conflated ink Pictures with page bitmaps). **Resolution:** size the *tile* window to the device memory budget, not a fixed page count — e.g. keep **full-DPI tiles only for the visible + ±1 pages (≈ 4 in double-page), and downgrade off-window pages to a 1× thumbnail tier** (matrix-upscaled, accepted as blurry only while scrolling). The ink cache keeps its wider ~812 window (cheap). R10's ≤64 MB now applies to **ink + downgraded-tier tiles**; the small high-DPI tile set is budgeted separately (~64128 MB depending on device), tuned in P0.5/P1.
**Data structures:** `PageWindow { int first, last }`; `Map<int, HostState> mountedHosts`; `TileKey { hostId, dpiBucket }`. **Acceptance:** all four modes render; switching recenters; double-page shows two pages side-by-side; ink tracks scroll/zoom; **page text/rules stay crisp at 4× zoom (no GPU-upscale blur) — DPI refreshes on zoom-settle** (R11); combined cache memory stays within the device budget under a full scroll; perf gate (§7) holds.
### F3 — Book-like reading (P1)
- `layout/reader_controller.dart` **[ADD]** — `ReaderMode { read, annotate }`; in `read` the arbiter never draws (pen ignored); page-flip animation for `pagedSingle/Double` via `PageView` physics; spread layout from `pagedDouble`.
- `ui/editor_screen.dart` **[MODIFY]** — mode toggle in toolbar; quick-jump via slider/grid (no keyboard).
**Acceptance:** read mode blocks ink; annotate mode draws; page-flip animates; two-page spread aligns facing pages; quick-jump scrolls/animates to target.
### F4 — Thumbnail-grid navigation (P1)
- `ui/thumbnail_grid.dart` **[ADD]** — Drawboard-style grid; tiles render via `pdf/pdf_document_source.renderTile` (replaces `lib/services/thumbnail_service.dart` syncfusion path). Tap → jump; **drag-reorder** for notebook page order (F6). Slider remains the linear scrubber (already in `pen_editor_screen._buildPagePill`).
- `lib/services/thumbnail_service.dart` **[MODIFY/RETIRE]** — migrate to pdfrx render; drop `syncfusion_pdfviewer_platform_interface` dep once ported.
**Acceptance:** grid shows all pages; tap jumps; drag reorders (F6); no text-field page input.
### F5 — Configurable pen (P2)
- `input/pen_config.dart` **[ADD]** — `PenConfig { ButtonAction sideButton; ButtonAction eraserEnd; PressureCurveType curve; double palmSensitivity; bool fingerDrawing; }`; `enum ButtonAction { eraser, undo, toggleTool, pan, lasso, none }`. `InputArbiter` consults it (replaces hardcoded `_isEraserSignal`).
- `lib/providers/settings_provider.dart` **[MODIFY]** — persist `PenConfig` (extend existing `SharedPreferences` notifier; it already stores pressure curve + stabilization).
- `lib/screens/settings_screen.dart` **[MODIFY]** / **[ADD]** `ui/pen_settings_page.dart` — mapping UI (SpeedyNote-style dialog).
- Reuse `lib/models/pressure_curve.dart` (already has linear/soft/hard/custom + `apply`).
**Acceptance:** remapping side-button to undo makes the barrel button undo; pressure curve changes stroke taper; palm sensitivity changes touch-cooldown; finger-drawing toggle works; all persist across restart.
### F6 — One-notebook-per-PDF (P2)
- `pdf/pdf_document_source.dart` **[MODIFY]** — a **page-map**: logical notebook pages → either a source-PDF page index or a synthetic blank page. Insert-blank adds a synthetic page **without rasterizing or editing the source bytes** (keep source vector + searchable). Ink/text bind to the **logical** page id (stable UUID per logical page), not raw PDF index, so inserts don't reshuffle annotations.
- `lib/services/pdf_service.dart` **[MODIFY]** — keep headless syncfusion export/mutate; export walks the page-map. **Fix the hairline bug**: `_renderStrokes` must build a `PdfPath` from `buildStrokeOutline` points and **fill** it (currently strokes line-segments → hairline). Shape/line/arrow keep stroke semantics.
- DB **[MODIFY]** — `notebook_pages(id, document_id, ordinal, source_page_index INTEGER NULL, kind)`; ink keyed by `notebook_page_id` (§4). Portable bundle = zip {source.pdf, badnote.json(strokes/text/links/page-map)} + relink-on-open (match by content hash, fall back to picker).
**Acceptance:** insert blank page between PDF pages → source PDF untouched (still vector/searchable in another viewer); reorder pages keeps ink attached; export `.pdf` shows filled ink matching screen (golden); bundle round-trips on another machine.
### F7 — Infinite board + 双链 (P3)
- `board/board_host_pane.dart` **[ADD]** — `BoardHost` (absolute px) in a constrained-false `InteractiveViewer`; reuses `AnnotationLayer` + engine. Migrates `lib/screens/split_view_screen.dart` + `scratchpads` table.
- `board/link_graph.dart` **[ADD]** — `Link { srcRef, dstRef }` where a ref is `(kind: notebookPage|board|note, id)`; `backlinksOf(ref)`. Wiki-style `[[...]]` parsing in text boxes; sticky-note = a small board region or text box that can link to a page/note. Backlink panel queries `links` table.
- DB **[ADD]** — `links(id, src_kind, src_id, dst_kind, dst_id, created_at)` + indexes both directions; `boards(id, document_id NULL, strokes_json, ...)` (generalize `scratchpads`).
**Acceptance:** create a sticky linking page 3 → note X; open note X shows a backlink to page 3; board draws at perf target; link graph survives restart; deleting a target leaves a dangling-link indicator (no crash).
### F8 — Library-wide full-text search (P3)
- `search/search_indexer.dart` **[ADD]** — index three sources into a unified library index: **(a)** PDF embedded text (pdfrx `PdfPage.loadText`/`charRects` text API — source-pinned in P0.5 per SF4), **(b)** typed text boxes (`content`), **(c)** handwriting OCR (existing `services/ocr` text-line ONNX over rasterized strokes via `StrokeRasterizer`). Reuse existing `document_fts` FTS5 + a new `library_fts` spanning notebooks/boards/notes with `(ref_kind, ref_id, page, snippet)`.
- `search/ocr_ingest.dart` **[ADD]** — batch handwriting OCR per logical page (best-effort; set expectations: text-line only, no math). Background isolate; debounced after ink idle. **The OCR backend itself no-ops when the model is unavailable (onnx_recognition_backend.dart header: "verify on-device") — search MUST never block on, nor be gated by, OCR results.**
- `lib/screens/search_screen.dart` **[MODIFY]** — unified results with snippets + **jump-to-location** (open editor at the page + scroll, or board at the region).
**Acceptance — split into two exit tiers (SF2):**
- **COMMITTED (P3 exit blocker):** PDF embedded text + typed-text-box search returns results with snippets; tapping a result opens and scrolls to the page/region. This tier alone satisfies the P3 search exit (it has no unverified-recall dependency).
- **ADDITIVE (best-effort, NOT a P3 exit blocker):** handwriting OCR contributes hits when the text-line model is available and confident; a word handwritten on page 5 may be returned. **Expectation set:** handwriting recall is limited by the text-line model and may return nothing on cursive/handwriting; math/formula is explicitly P5. P3 ships even if handwriting recall is poor, with UI copy stating the limit.
### F9 — Server sync + AI refinement (P4)
- `sync/sync_client.dart` **[ADD]** — wire the **already-built** FastAPI endpoints (`/api/auth`, `/api/sync/push|pull`, `/api/notes`, `/api/documents`, `/api/ocr`). Last-writer-wins by `updated_at` (server already implements this). Sync notes + (later) notebooks/boards.
- `server/badnote_server/` **[MODIFY]** — extend `sync_router`/`models.py` to cover notebooks/boards/links (today only notes). Add an **AI-refine** endpoint that runs llm_wiki/VLM/LLM over a note/page → returns organized markdown (server-side, heavy deps gated like the OCR worker).
- `lib/providers/` **[ADD]** sync state provider; settings page server URL/token.
**Acceptance:** push from device A, pull on device B reproduces notes; AI-refine returns organized text for a selected note; offline still fully functional (server optional, per README).
### F10 — Ink CAS / formula (P5)
- `search/ocr_ingest.dart` **[MODIFY]** — add a **formula/math recognition** backend (new model; the current PP-OCRv4 is text-line only — this is the known hard sub-problem). Behind a feature flag.
- `engine/` **[ADD]** `cas/` — recognized formula → CAS (compute/solve) behind a toggle; renders result near the ink. Reuses the `CoordinateSpaceHost` seam for overlay placement.
**Acceptance (stretch):** a handwritten `2+3=` toggled → shows `5`; recognized formulas become searchable. Gated, optional, lowest priority.
### F11 — Modern UX polish (woven P0P3)
- Material You already wired (`main.dart` `DynamicColorBuilder` + harmonized schemes + Inter). Continue: subtoolbars in `editor_toolbar`, drag-reorder thumbnails (F4/F6), **hover pre-warm** (warm the next page tile + Picture on stylus hover to cut pen-down latency — reuse the live `_onPointerHover` seam), animated mode/layout transitions.
---
## 3. Data model & persistence (clean schema — data may reset)
Bump DB to a fresh version with `_onCreate` only (keep `_onUpgrade` harmless). Today: notes/strokes, documents/annotations(JSON-per-page), bookmarks, ocr_results, document_fts(FTS5), scratchpads (DB v5, `lib/services/database_service.dart`).
**Target tables (additions/changes in bold):**
- `documents` **[KEEP]** (drop editor reliance on `rotation`; page-map owns rotation).
- **`notebook_pages(id PK, document_id FK, ordinal INTEGER, source_page_index INTEGER NULL, kind TEXT, created_at)`** **[ADD]** — F6 logical pages.
- **`ink(id PK, host_kind TEXT, host_id TEXT, stroke_json TEXT, ordinal INTEGER, updated_at)`** **[ADD]** — strokes addressed by host (`host_kind ∈ {page, board}`, `host_id` = `notebook_page_id` or `board_id`). **Per-stroke rows** (not per-page blob) so a 2,000-stroke page doesn't rewrite on every save — **but ONLY valid under the MF3 diff-write contract (§2/F1 `editor_repository.saveHost`): UPSERT changed rows + DELETE removed rows, never delete-all+re-insert.** Without diffing, per-row is *worse* than a blob; the contract is what makes this schema correct. Index `(host_kind, host_id)`.
- **`text_boxes(id PK, host_kind, host_id, content TEXT, rect_json, font_size, color, updated_at)`** **[ADD]** — F2-area text + F8 indexing.
- **`boards(id PK, document_id FK NULL, title, created_at, updated_at)`** **[ADD]** — generalize `scratchpads`; board strokes live in `ink` with `host_kind='board'`.
- **`links(id PK, src_kind, src_id, dst_kind, dst_id, created_at)`** **[ADD]** — 双链; indexes on `(src_kind,src_id)` and `(dst_kind,dst_id)`.
- `bookmarks` **[KEEP]**.
- `ocr_results` **[KEEP/EXTEND]** — per logical page handwriting OCR text.
- **`library_fts` (FTS5)** **[ADD]** — `(ref_kind, ref_id, page, content)` unified search over PDF text + typed text + OCR. `document_fts` **[KEEP]** for PDF-page text.
- `notes`/`strokes`/`notes_fts` **[KEEP]** (existing ink-note path; eventually folded into boards, but not deleted in P0).
- **`sync_state(entity_kind, entity_id, last_pushed_at, last_pulled_at, dirty)`** **[ADD, P4]**.
**Coordinate semantics on disk:** page ink = normalized `[0,1]` unrotated; board ink = absolute logical px. (Matches live conventions.)
**Stroke-model convergence (important):** there are currently **two** stroke models — `PenStroke/PenPoint` (live canvas, in-memory only) and `InkStroke/InkPoint` (freezed/JSON, DB+OCR+export). P0 introduces **one** canonical `EditorStroke/EditorPoint` (freezed/JSON, normalized) and adapters to/from `InkStroke` for OCR/export reuse during transition; old screens keep `InkStroke` until retired (§6).
---
## 4. Refactor strategy (evolve, don't rewrite)
**Principle: every phase ships; the engine generalizes under load.**
1. **P0 = relocation + persistence.** Move the 4 live canvas files into `engine/` + `render/` + `input/` with minimal logic change; add `StrokeStore.revision` + `ui.Picture` cache + DB persistence via `editor_repository`. The live single-page editor keeps working throughout. **Keep** `PenStroke` as a thin alias of `EditorStroke` until callers migrate.
2. **P0.5 = vertical slice (continuous-single only) — the new perf/crispness gate (see §5/§9).** Stand up `layout/page_viewport` + `pdf/page_tile` rendering continuous-SINGLE only, with the **rewritten** perf bench (targeting `ui/editor_screen`, not the spike) and the zoom-DPI refresh proven crisp at 4× on the Surface. This proves the own-canvas multi-page + multi-resolution model on the real device before any double/paged/spread work.
3. **P1 = full multi-page windowing.** Wrap the (now-relocated) `AnnotationLayer` in `page_viewport`; `_strokesByPage` Map → repository-backed mounted hosts. Single-page path stays as `pagedSingle`. Continuous-double / paged / spread land here, AFTER P0.5 passes.
4. **Old-code retirement (gated on parity, then delete in one step):**
- **[DELETE after P1 parity]** `lib/screens/pdf_annotator_screen.dart`, `lib/widgets/pdf_annotation_layer.dart`, `lib/widgets/ink_canvas.dart` (extract any unique draw/erase logic to `engine/` first), `lib/editor/pdf/spike_*.dart` (throwaway M1 spike — includes the spike-based perf bench, replaced in P0.5), `lib/editor/pdf/pen_capture_region.dart` + `PenCaptureBinding` in `main.dart` (own-canvas `Listener` model won — remove the unused arena-bypass binding once P0 device-confirms).
- **[DELETE in P3, NOT P1 (SF3)]** `lib/screens/split_view_screen.dart` — its replacement is the **P3** infinite board (F7); deleting it in P1 would leave a 2-phase functionality gap (no scratchpad between P1 and P3). It stays live until the board lands.
- **[KEEP]** `thumbnail_service` until F4 ports it; `stroke_rasterizer` (OCR), `ctc_decoder`, `onnx_recognition_backend`, OCR assets; `pdf_service` (export/mutate, with the fill fix); `pptx_service`/`ppt_annotator_screen` (PPT is separate; not in scope but not deleted).
5. **Navigation swap:** `home_screen` `_openDocument`/`_importPdf` currently push `PdfAnnotatorScreen`; `openM1Spike` pushes `PenEditorScreen`. Repoint both to `ui/editor_screen.dart` once P1 parity passes (checklist below).
6. **Parity checklist before any delete:** page rotate/delete/insert-blank/insert-image (`pdf_service`), bookmark add/toggle/jump, undo/redo across pages, save-on-leave, zoom in/out/fit, export-matches-screen (golden). Mirrors phase-1 §10 M3 checklist C1C11.
---
## 5. Phased delivery (each = shippable milestone with exit criteria)
> Sequence de-risks: pen core + persistence first (P0), then the 60fps multi-mode layout (P1), then config/notebook (P2), then board/双链/search (P3), then sync/AI (P4), then CAS (P5).
**P0 — Pen core hardened + persisted (own-canvas).** *Exit:* draw/erase/undo/redo on a single PDF page persist to DB and reload; `EditorStroke` canonical model (superset of `InkPoint`, SF1) + revision-gated `ui.Picture` cache; `editor_repository.saveHost` honors the MF3 diff-write contract (test: erase 1 of 2,000 ⇒ 1 DELETE, 0 re-inserts); pure `InputArbiter` + `stroke_eraser` unit-tested; **on-device Surface Pen confirms** pressure + palm rejection + pinch-zoom in the live shared-transform model (the one device gate). Export hairline bug fixed (fill). No regression to existing screens.
**P0.5 — Vertical slice gate (own-canvas multi-page + multi-resolution), continuous-SINGLE only. [NEW — SYNTHESIS]** Stand up `layout/page_viewport` + `pdf/page_tile` rendering **continuous-single only** on the real document path (`ui/editor_screen`), and:
- (a) **REWRITE the perf bench** (§7/§8): a new `integration_test/editor_scroll_bench.dart` drives `ui/editor_screen` + `layout/page_viewport`; **DELETE the spike-based `integration_test/perf_scroll_bench.dart`** (it imports `spike_editor_pane.dart` / `pageOverlaysBuilder` = the invalidated Option B; it CANNOT validate own-canvas).
- (b) **Prove crisp-on-zoom (R11):** page-tile DPI refresh on zoom-settle renders crisp text/rules at 4× on the Surface (no GPU-upscale blur).
- (c) **Source-pin the pdfrx render/text APIs (SF4):** confirm `PdfPage.loadText`/`charRects`/`PdfPage.render()`/`PdfPageView` signatures present in pdfrx 2.4.4; smoke-test each, so F4/F8 don't hit drift late.
- *Exit (hard gate before any double/paged/spread):* **continuous-single median build+raster ≤ 16.6ms, p95 ≤ 22ms** on the 300-page asset via the REWRITTEN bench, **AND** a manual crisp-on-zoom-at-4× PASS on the Surface, **AND** the four pdfrx APIs source-pinned + smoke-tested. Continuous-double / paged / spread do NOT begin until P0.5 is GREEN.
**P1 — Full layout modes + book-like reading + thumbnails (60fps). Precondition: P0.5 GREEN.** *Exit:* the remaining `PageLayout`s (continuous-double, pagedSingle, pagedDouble); reader vs annotate; page-flip + two-page spread; thumbnail-grid jump + drag-reorder; windowed lazy hosting; **perf gates met** (§7) incl. continuous-double on a 300-page PDF via the rewritten bench; old PDF screens deleted after parity checklist (split_view retirement deferred to P3 per SF3); nav repointed.
**P2 — Configurable pen + one-notebook-per-PDF.** *Exit:* Pen settings page (button/eraser mapping, curve, palm sensitivity, finger toggle) persists + drives arbiter; insert-blank/reorder pages keep source vector + ink attached to logical pages; portable bundle round-trips; export walks page-map.
**P3 — Infinite board + 双链 + library search.** *Exit:* board reuses engine at perf; sticky-notes + `[[links]]` produce backlinks; **library search COMMITTED tier** (PDF embedded text + typed-text-box) returns snippets + jump-to-location (this tier is the exit blocker, SF2); **handwriting-OCR tier is ADDITIVE/best-effort and does NOT block P3 exit**; `split_view_screen` retired into board (the SF3 deletion point).
**P4 — Server sync + AI refinement.** *Exit:* push/pull notes+notebooks across two devices (LWW); AI-refine endpoint returns organized markdown; app fully functional offline.
**P5 — Formula OCR + ink CAS.** *Exit (stretch, gated):* formula recognition backend; searchable formulas; CAS solve toggle for simple expressions.
Every milestone ends with verifier/critic pass + perf-results doc updated with commit hash.
---
## 6. Refactor of input transport (record the pivot)
The live `PenCanvas` proves the **own-canvas** model: a single `Listener` over an `InteractiveViewer` whose child is `Stack(PdfPageView bitmap, StaticInk, LiveInk)`. Because the pen, page bitmap, and ink share **one** transform and the `Listener` arbitrates by `_activePointers.length` + `kind`, there is **no gesture-arena fight** — pdfrx never sees gestures (it only renders). This **removes** the phase-1 need for `PenCaptureRegion`/`PenCaptureBinding` (RenderProxyBox arena bypass), which exist for the abandoned `pageOverlaysBuilder` approach. **Retirement gate:** delete them once P0 confirms on the physical Surface Pen that the `Listener` model handles stylus draw + single-finger pan + pinch-zoom + palm rejection (the live code is built for exactly this; confirm, then remove the dead binding).
---
## 7. Risks & mitigations
| # | Risk | L | I | Mitigation / trigger |
| --- | --- | --- | --- | --- |
| R1 | **60fps continuous + double-page on a 300-page PDF.** Two columns × windowed tiles × ink Pictures may blow frame budget. | Med | High | Windowed lazy hosting (only visible+cacheExtent mounted), bounded LRU + post-frame dispose for BOTH the `ink_picture_cache` and the `page_tile_cache` (R10), per-page `RepaintBoundary`, revision-gated static Picture. **Gate continuous-SINGLE in P0.5, continuous-double in P1** (§5/§9). **MF1 — the existing `integration_test/perf_scroll_bench.dart` is UNUSABLE here: it imports `spike_editor_pane.dart` and benchmarks `pageOverlaysBuilder` (Option B, the INVALIDATED architecture) — it cannot validate own-canvas. P0.5 REWRITES the bench against `ui/editor_screen` + `layout/page_viewport` and DELETES the spike-based one.** Uses `large_300p.pdf`. |
| R2 | **Pen feel only verifiable on the user's Surface** (CI has no pen). | High | Med | P0 device gate (pressure/palm/pinch) + manual checklist in perf-results doc; synthesized-stylus widget tests as interim signal only. |
| R3 | **Handwriting/formula OCR feasibility.** Current model is text-line PP-OCRv4; math is unsolved. | High | Med | F8 ships text-line best-effort with **explicit expectation-setting**; formula isolated to P5 behind a flag; never block search on OCR quality. |
| R4 | **双链 graph scale** (thousands of links/sticky-notes). | Low | Med | Indexed `links` table (both directions), lazy backlink queries, no in-memory full graph. |
| R5 | **Server/AI scope creep.** | Med | Med | Server stays optional (README); P4 wires existing endpoints + one AI-refine route; AI heavy deps gated like OCR worker. |
| R6 | **Windows pen edge cases** (no advertised pressure range; barrel-button eraser; inverted stylus). | Med | Med | Live `_normalizedPressure` already handles degenerate ranges; F5 makes button/eraser mappable; test matrix in arbiter unit tests. |
| R7 | **Export fidelity** (hairline bug today). | High (today) | Med | `stroke_geometry.buildStrokeOutline` shared screen+export; `pdf_service._renderStrokes` **fills** a `PdfPath`; golden test (P0). |
| R8 | **Stroke-model convergence churn** (two models today). | Med | Low | One canonical `EditorStroke` + adapters; old `InkStroke` retained only where old screens/OCR/export still use it, deleted with them. |
| R9 | **pdfrx page-render / text API drift** (thumbnails, search text, double-page sizing). | Med | Low | **Source-pin `PdfDocument`/`PdfPageView`/`PdfPage.loadText`/`charRects`/`render()` signatures + smoke-test in P0.5 (pulled forward, SF4)** so F4/F8 don't hit drift late; keep syncfusion export until verified. |
| R10 | **Cache memory budget vs many mounted pages (TWO caches — Critic).** | Med | Med | **`ink_picture_cache`** (vector `ui.Picture`, no DPI bucket): ~812 mounted pages, a few MB. **`page_tile_cache`** (raster `ui.Image`, DPI-bucketed): the heavy store — a 3×-DPI A4 tile is ~1030 MB, so 812 high-DPI tiles would be ~150350 MB. **The old "≤64MB / 812 pages" figure was unit-confused (conflated ink with page bitmaps).** Fix (§2/F2): keep full-DPI tiles only for visible ±1 pages, downgrade off-window pages to a 1× thumbnail tier; ≤64MB applies to ink + downgraded tiles, the small high-DPI tile set budgeted separately (~64128MB, device-tuned). Sample BOTH caches during the scroll bench. |
| R11 | **Blurry page at high zoom under the shared transform (MF2).** One `InteractiveViewer` matrix-scales a `PdfPageView` bitmap that was rendered at **layout-constraint × devicePixelRatio** (1×); pinch-zoom GPU-**upscales** it → blurry text/rules at 4×, whereas pdfrx's own viewer re-renders crisp tiles per zoom level. We **own multi-resolution tiling**, not just layout/windowing. | Med | High | `pdf/page_tile.dart` drives render DPI from the transform scale: on zoom-settle re-instantiate the tile (re-laid-out `PdfPageView` at the new pixel size, or `PdfPage.render()` at target DPI) into the **`pdf/page_tile_cache.dart`** store (`LRU<TileKey{hostId, dpiBucket}, ui.Image>` — NOT the ink cache); cache key includes a DPI bucket; **cap retained DPI** (~3× base) to bound memory; downscale stays matrix-only. **`PdfPage.render()` returns an async `PdfImage` owning a native handle → `page_tile_cache` owns its post-frame dispose** (see Open Questions; confirm in the SF4 P0.5 source-pin). **Proven crisp at 4× on the Surface in the P0.5 gate** (§5). Budgeted P0.5/P1, not relocation. |
### Pre-mortem (DELIBERATE — 3 scenarios)
1. **"Continuous double-page janks / pages are blurry at 4× on the user's big scanned PDF."** Cause: built layout modes before profiling two-column windowing, and matrix-scaled a 1× bitmap (R11). *Prevention:* the **P0.5 vertical-slice gate** profiles continuous-SINGLE first with a **rewritten** bench against `ui/editor_screen` (the spike-based `perf_scroll_bench.dart` is deleted — it benchmarks the invalidated Option B) AND proves crisp-on-zoom DPI refresh at 4× on the Surface; continuous-double/paged/spread merge only after P0.5 is GREEN, and the P1 double-page gate reuses the rewritten bench.
2. **"Search returns nothing for handwriting."** Cause: over-promised OCR. *Prevention:* F8 ships PDF-text + typed-text search first (reliable), handwriting OCR as additive best-effort with UI copy stating limits; formula explicitly P5.
3. **"Insert-blank-page silently rasterized / detached annotations."** Cause: editing source bytes or keying ink to raw PDF index. *Prevention:* logical page-map + ink keyed to `notebook_page_id`; golden test that the source PDF bytes are unchanged after insert and remains selectable-text in an external viewer.
---
## 8. Testing strategy
> **sqlite workaround:** DB-touching tests run via `tool/test.sh` (system sqlite + `LD_LIBRARY_PATH`); pure-logic tests avoid the DB. Reusable assets: `tool/gen_bench_pdf.dart`, `tool/gen_dense_strokes.dart`, `test/assets/large_300p.pdf`, `test/assets/dense_strokes.json`, `integration_test/coordinate_assertion_test.dart`. **NOT reusable: `integration_test/perf_scroll_bench.dart` — it imports `spike_editor_pane.dart` (`pageOverlaysBuilder`, invalidated Option B) and is DELETED + replaced by `integration_test/editor_scroll_bench.dart` (targets `ui/editor_screen`) in P0.5 (MF1).**
- **Unit (no DB/widgets):**
- Coordinate transforms: `NormalizedPageHost`/`BoardHost` round-trip `toContent(toDevice(x))≈x`.
- Stroke geometry: `buildStrokeOutline` non-empty for ≥1 point; live ⊆ committed bounds (no "pop").
- Eraser: `eraseHits` whole-stroke removal (extraction of live `pen_canvas._eraseAt` — regression port). **`splitStroke` segment-erase is NEW behavior, not a port** (the live canvas only does whole-stroke erase; `undo_manager` records but never computes replacements): mid-erase ⇒ 2 segments, endpoint ⇒ 1 segment, full-erase ⇒ empty, <2-pt result dropped — these exercise greenfield code. (`undo_manager_test.dart` informs the replacement-bookkeeping discipline only.)
- Revision gating: `StrokeStore.add` bumps revision; `StaticInkPainter.shouldRepaint` iff revision changed.
- `InputArbiter` SM: table-driven over the device×mode matrix incl. palm rejection (touch dropped while stylus active) and `pointerCount>=2`→cancel.
- Undo: global commit-time order, host-tagged reversal.
- `SaveScheduler`: mutating "current host" after schedule but before write completes does not change persisted snapshot.
- **`editor_repository.saveHost` diff-write (MF3, fake DB counting statements):** erase 1 of 2,000 strokes ⇒ exactly **1 DELETE, 0 INSERT**; add 1 stroke ⇒ **1 INSERT, 0 DELETE**; no-op save ⇒ 0 statements. (Guards against the live notes-path delete-all+re-insert anti-pattern at database_service.dart:281306.)
- `EditorStroke``InkStroke` adapter round-trip (SF1): `tilt`/`timestamp`/`pointerDeviceKind` survive both directions (no lossy OCR/export conversion).
- `link_graph.backlinksOf` (P3); `search_indexer` snippet/jump-ref mapping for the COMMITTED tier — PDF-text + typed-text (P3).
- **Widget:** stylus→stroke committed, touch→not consumed (synthesized pointers); layout-mode switch recenters; reader-mode blocks ink; thumbnail tap jumps; text-box place/move/edit persists (fake DB).
- **Perf:** the **rewritten** `integration_test/editor_scroll_bench.dart` (targets `ui/editor_screen` + `layout/page_viewport`; the spike-based `perf_scroll_bench.dart` is deleted, MF1) on `large_300p.pdf`**continuous-single in P0.5 (gate), continuous-double in P1**, profile mode, N≥120 frames warm, median build+raster ≤16.6ms / p95 ≤22ms; `StaticInkPainter` no-rebuild assertion; Picture-memory ≤64MB sample. Archived in `docs/plans/full-refactor-perf-results.md` with commit hash. Not a hard CI gate (no GPU) but required for milestone sign-off.
- **Manual on-device pen checklist (Surface Pen, P0 + each milestone):** pressure varies width; barrel/inverted = erase; palm rest doesn't mark; single-finger scroll; two-finger pinch; **page text/rules stay crisp at 4× zoom — no GPU-upscale blur, DPI refreshes on zoom-settle (R11, P0.5 gate)**; hover pre-warm reduces first-stroke latency; page-flip feels book-like. Recorded with device + commit.
- **Regression:** keep `ctc_decoder_test.dart`, `undo_manager_test.dart` green; update `widget_test.dart` to boot `editor_screen`.
- **Export golden (P0):** annotate a known page → export → image-compare filled ink matches screen (R7).
- **Server (P4):** existing `server/tests/` (`test_sync.py` etc.) green; add notebook/board sync tests.
---
## 9. Milestones / sequencing — immediate next chunk (concrete)
**Next chunk = P0 (engine relocation + persistence), executable now; followed by the P0.5 vertical-slice gate (steps 1013) before any double/paged/spread layout work:**
1. **[ADD]** `lib/editor/engine/stroke_model.dart``EditorStroke`/`EditorPoint` (freezed + JSON), normalized; `fromPenStroke`/`toInkStroke` adapters. Run `build_runner`.
2. **[ADD]** `lib/editor/engine/stroke_geometry.dart` — lift `buildStrokePath``buildStrokeOutline` (verbatim recipe from `ink_painters.dart`).
3. **[ADD]** `lib/editor/engine/stroke_store.dart` + `lib/editor/render/{static_ink_painter,live_ink_painter,ink_picture_cache,annotation_layer}.dart` — relocate live painters; add the resolution-independent ink `ui.Picture` cache keyed by `revision` (NO DPI bucket; the DPI-bucketed `page_tile_cache` is a separate P0.5 file, step 10).
4. **[ADD]** `lib/editor/input/input_arbiter.dart` + `lib/editor/engine/stroke_eraser.dart` — extract from `pen_canvas.dart` (pure, unit-tested).
5. **[ADD]** `lib/editor/persistence/{editor_repository,save_scheduler}.dart` + DB additions (`ink`, `notebook_pages` minimal) in `database_service.dart` (fresh version). `saveHost` implements the **MF3 diff-write contract** (UPSERT changed + DELETE removed, by stroke id; NO delete-all+re-insert) with a per-host last-persisted id-set.
6. **[MODIFY]** `lib/editor/canvas/pen_editor_screen.dart` (or new `ui/editor_screen.dart`) to load/commit/save through the repository instead of the in-memory `_strokesByPage` Map.
7. **[MODIFY]** `lib/services/pdf_service.dart` `_renderStrokes` → fill `buildStrokeOutline` path (R7) + export golden test.
8. **Tests:** arbiter SM, eraser, geometry, revision-gating, save-scheduler snapshot, **diff-write statement-count (MF3: erase 1/2000 ⇒ 1 DELETE 0 INSERT)**, `EditorStroke↔InkStroke` lossless round-trip (SF1), export golden. Run via `tool/test.sh`.
9. **Device gate:** build Windows package (CI), confirm pen/palm/pinch on Surface Pen; record in perf-results doc. → unblocks P0.5.
**Then P0.5 (vertical-slice gate — must pass before P1's double/paged/spread):**
10. **[ADD]** `lib/editor/layout/page_viewport.dart` (continuous-single only) + `lib/editor/pdf/{pdf_document_source,page_tile,page_tile_cache}.dart` with zoom-settle DPI refresh (R11) + the **DPI-bucketed `page_tile_cache`** (`LRU<TileKey, ui.Image>`, page bitmaps; owns native-handle dispose). (The resolution-independent `render/ink_picture_cache.dart` for vector ink lands in P0, step 3 — it is a separate cache, no DPI bucket.)
11. **[ADD]** `integration_test/editor_scroll_bench.dart` targeting `ui/editor_screen`; **[DELETE]** `integration_test/perf_scroll_bench.dart` + `lib/editor/pdf/spike_*.dart` (the spike pane the old bench imports). Run the rewritten bench on a **scanned-image** 300-page asset (see Open Questions — `large_300p.pdf` is synthetic/vector and may not honestly stress raster re-render at 3× DPI) → continuous-single median ≤16.6ms / p95 ≤22ms; sample both caches' memory.
12. **[VERIFY]** source-pin pdfrx `PdfPage.loadText`/`charRects`/`render()`/`PdfPageView` in 2.4.4 (SF4) + smoke test; **confirm `PdfPage.render()`'s `PdfImage`/native-handle ownership + dispose semantics** so `page_tile_cache` can manage post-frame disposal; record signatures in perf-results doc.
13. **Device gate:** crisp-on-zoom at 4× PASS on the Surface (R11). → unblocks P1.
Each subsequent milestone (P1…P5) follows §5 exit criteria; verifier/critic + perf-results update per milestone.
---
## 10. RALPLAN-DR
### Principles (35)
1. **Single source of truth = host content coordinates.** Screen mapping is a paint-time `canvas` transform; never store/duplicate transformed geometry. (Already the live convention — generalize it.)
2. **One host-agnostic ink engine.** PDF page, infinite board, and (P5) CAS overlay are `CoordinateSpaceHost`s behind one renderer — never fork the stroke pipeline.
3. **Own the gesture pipeline; pdfrx only renders.** A single `Listener` over a shared `InteractiveViewer` arbitrates draw/pan/zoom/palm by pointer kind + count — no gesture-arena fights (proven live).
4. **De-risk performance AND crispness before features.** A **P0.5 vertical-slice gate** (continuous-single, rewritten bench, crisp-at-4× on the Surface) precedes all double/paged/spread work; continuous-double is gated again in P1.
5. **Ship every phase; generalize under load.** Relocation-first refactor keeps the editor working at all times; old screens deleted only after parity.
### Decision Drivers (top 3)
1. **D1 — Pen feel + palm rejection + pinch on Windows Surface Pen** is the make-or-break primary-device requirement (own-canvas model already targets it).
2. **D2 — 60fps across all layout modes** on big PDFs with thousands of strokes.
3. **D3 — One engine reused across PDF page, infinite board, and CAS**, forward-compatible with search/双链/sync.
### Viable options (≥2) with bounded pros/cons
**Option A — Own-canvas engine (single `Listener` + `InteractiveViewer`, pdfrx as renderer). CHOSEN (already live).**
- Pros: D1 solved structurally (one transform, no arena fight — already working in `pen_canvas.dart`); D3 trivial (hosts share the transform); minimal new deps; matches the proven Saber model the user cited.
- Cons: we own page layout/windowing/tiling **AND multi-resolution re-rasterization** (more code than a stock viewer); double-page perf is on us (R1) and crisp-on-zoom is on us (R11 — pdfrx's own viewer re-renders crisp tiles per zoom for free; we must re-derive tile DPI from the transform); thumbnails/text-extraction still need pdfrx page APIs (R9).
**Option B — pdfrx `pageOverlaysBuilder`-hosted ink + RenderProxyBox arena-bypass (the phase-1 plan).**
- Pros: pdfrx gives continuous scroll/tiling/text-extraction for free; ink-follows-page is structural via page overlays.
- Cons: requires a custom `PenCaptureBinding` arena bypass that fights pdfrx's greedy scale recognizer (the live code already **abandoned** this — `PenCaptureRegion` is unused); pdfrx owns the transform so double-page/board reuse is awkward; D1 proven harder than Option A in practice. **Invalidated** — see below.
**Option C — Flutter shell + Rust hot-path (rnote-style) for ink/render.**
- Pros: maximal ink perf headroom.
- Cons: rejected in memory (`badnote-flutter-344-windows-pen`) — Windows pen is weaker in the Rust/GTK stack; huge FFI surface; contradicts "stay Flutter." **Invalidated.**
### Invalidation rationale
- **B invalidated (as the INPUT model):** the live codebase already moved off it; `pageOverlaysBuilder`+arena-bypass made stylus/touch arbitration fight pdfrx's recognizer, whereas Option A's single-`Listener`-over-shared-transform sidesteps the arena entirely and is already drawing with pressure/pinch/palm. **Correction (MF2): B had TWO edges, not one** — (1) free continuous scroll/windowing, AND (2) free **per-zoom crisp re-rasterization** (pdfrx re-renders tiles at each zoom level). Edge (1) is recoverable in A via windowed hosting (needed for double-page anyway); edge (2) is **NOT free in A** — Option A must own multi-resolution tiling (R11), which is the genuinely new cost of this choice. We accept that cost (gated in P0.5) because A's structural D1/D3 wins outweigh it. B retained only as a fallback page-**render** strategy, not the input model.
- **C invalidated:** documented Windows-pen regression in the Rust/GTK path + "stay Flutter" hard constraint; the 3.44 WM_POINTER fix already unblocked Flutter pen, removing C's motivation.
- Net: **A chosen; B retained as a partial fallback (page rendering only); C rejected.**
### ADR
- **Status:** Architect APPROVE-WITH-MUST-FIX applied (2026-06-21): MF1 (rewrite perf bench off the invalidated spike), MF2 (R11 zoom re-rasterization + multi-resolution tiling), MF3 (diff-write contract); P0.5 vertical-slice synthesis gate; SF1SF5. Pending Critic. Records the live pivot from the superseded phase-1 input architecture.
- **Decision:** Build the full BadNote vision on an **own-canvas, host-agnostic ink engine** (single `Listener` + shared `InteractiveViewer`; pdfrx as page renderer/text source). Generalize the live `lib/editor/canvas/` into `engine/render/input/layout/...`; persist a single canonical `EditorStroke`; phase features P0→P5 with a **P0.5 vertical-slice gate**.
- **Drivers:** D1 Surface-Pen feel, D2 60fps multi-mode, D3 one reusable engine.
- **Alternatives considered:** B (pdfrx-overlay + arena bypass — invalidated as input model, kept as render fallback; had two free edges — scroll AND per-zoom crispness — the latter is the new cost we take on), C (Flutter+Rust — rejected).
- **Why chosen:** A is already proven live for pen/palm/pinch and gives D1+D3 structurally; the remaining risks (D2 multi-mode perf AND R11 crisp-on-zoom) are gated up front in **P0.5** then re-gated for double-page in P1.
- **Consequences:** we own layout/windowing/tiling **AND multi-resolution page-tile re-rasterization** (R11 — not just layout, MF2); `PenCaptureRegion`/`PenCaptureBinding` retired (dead under own-canvas); the spike-based perf bench is deleted and rewritten against `ui/editor_screen` (MF1); per-stroke-row persistence is valid ONLY under the diff-write contract (MF3); `EditorPoint` is a non-lossy superset of `InkPoint` (SF1); export must fill (not stroke) ink; handwriting OCR is additive/non-blocking and formula is P5 (SF2); `split_view_screen` deletion deferred to P3 (SF3); server stays optional.
- **Follow-ups:** P0 device gate confirms pen model; **P0.5 gate** proves continuous-single perf + crisp-at-4× + source-pinned pdfrx APIs (SF4) before any double/paged/spread; resolve open questions below.
---
## Open Questions (persist to `.omc/plans/open-questions.md`)
- [ ] Continuous-double-page on a real 300-page scanned PDF — does windowed two-column hosting hold 60fps, or do we need tile pre-rasterization? — *P0.5 gates continuous-single first; P1 perf gate decides double; affects R1/R10.*
- [ ] **How is page-tile DPI refreshed on zoom under the shared transform (SF5)?** — re-laid-out `PdfPageView` at the new pixel size, or `PdfPage.render()` at target DPI into `page_tile_cache`? What scale-change threshold + debounce triggers a refresh, and what is the retained-DPI cap? — *Load-bearing for the 60fps + crisp-at-4× goal; resolved in P0.5 (R11).*
- [ ] **`PdfPage.render()` native-handle lifecycle (Critic 4a):** `render()` returns an async `PdfImage` owning a native handle backing a `ui.Image``page_tile_cache` must own its dispose (post-frame, after the raster thread is done) on eviction. *Confirm exact ownership + dispose API in the SF4 P0.5 source-pin; affects page_tile_cache + R10/R11.*
- [ ] **Honest R11 raster stress asset (Critic 4b):** does the synthetic/vector `large_300p.pdf` actually force RASTER re-render cost at 3× DPI, or does a **scanned-image** asset better exercise the crispness + tile-memory gate? — *Add a scanned 300-page asset (`tool/gen_bench_pdf.dart` image mode or a real scan) for the P0.5/P1 perf+crispness gates; affects §8 / R11 honesty.*
- [ ] Stroke-model convergence: fold `notes`/`InkStroke` ink-note path into `boards`/`EditorStroke`, or keep notes separate long-term? — *Affects §3/§4 churn.*
- [ ] Handwriting OCR: is the existing PP-OCRv4 text-line model usable on cursive/handwriting at all, or do we need a handwriting-specific model even for non-formula text? — *Affects F8 expectation-setting / R3.*
- [ ] Portable bundle relink: match source PDF by content-hash only, or also store original path + size? — *Affects F6.*
- [ ] Server AI-refine: run llm_wiki/VLM server-side only, or allow a local LLM path for offline users? — *Affects F9 scope / R5.*
- [ ] Keep `pptx_service`/`ppt_annotator_screen` in the new engine, or freeze PPT support? — *Out of the 11 features; decide before P1 nav swap.*