feat(pen): pressure-responsive width, configurable thinning, native Windows pen (tilt/buttons)
All checks were successful
CI / Windows build (push) Successful in 11m34s
All checks were successful
CI / Windows build (push) Successful in 11m34s
W1 — Custom pen width + pressure sensitivity (Saber-style):
- Root cause of "压感没用": perfect_freehand 1.0.4 IGNORES real stylus pressure
(hardcodes radius=size/2 when simulatePressure=false) — width never tracked pen
force. Upgraded perfect_freehand ^1.0.0 -> ^2.0.0 (honors real pressure); migrated
all 5 getStroke call sites to the 2.x API (PointVector / StrokeOptions / Offset).
- De-hardcoded `thinning` into `kDefaultPenThinning` (0.85), single source shared by
the on-screen painter and the PDF export path; exposed as PenConfig.pressureSensitivity
with a Pressure Sensitivity slider; live-applies via a config listener.
W3 — Native Windows pen plugin (tilt + barrel/eraser buttons):
- windows/runner/pen_channel.{h,cpp}: observe WM_POINTER at the TOP of MessageHandler
(before HandleTopLevelWindowProc, which Flutter uses to consume pen events), read
GetPointerPenInfo penFlags + tilt, stream over EventChannel('badnote/pen'); non-consuming.
- PenInputService: single latched hardware state (no Win32-pointerId<->event.pointer
correlation); graceful no-op off-Windows.
- pen_canvas maps barrel/inverted/eraser through PenConfig.sideButton/eraserEnd
(eraser/undo/toggleTool/pan) and captures tilt into PenPoint.tilt -> EditorPoint.tilt.
W2 — Zoom flicker: page raster isolated in its own RepaintBoundary (safe interim);
definitive crisp-on-zoom fix gated on the on-device root-cause probe (plan M3).
Plans: ralplan-consensus plan at docs/plans/2026-06-22-badnote-pen-polish.md
(Architect APPROVE-WITH-MUST-FIX M1-M4 + Critic ITERATE->APPROVE).
Tests: 58/58 pass incl. shared-thinning invariant + thinning-affects-outline +
tilt-adapter round-trip. flutter analyze clean; linux debug build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
446
docs/plans/2026-06-21-badnote-full-refactor.md
Normal file
446
docs/plans/2026-06-21-badnote-full-refactor.md
Normal file
@@ -0,0 +1,446 @@
|
||||
# BadNote — Full Refactor + Feature Roadmap (own-canvas engine)
|
||||
|
||||
**Status:** PLAN (ralplan consensus — Architect APPROVE-WITH-MUST-FIX + Critic-ITERATE fixes applied 2026-06-21: MF1–MF3 + P0.5 slice + SF1–SF5 + cache-split / diff-write-durability / erase-is-new-behavior / render-handle-lifecycle; pending final Critic confirm)
|
||||
**Date:** 2026-06-21
|
||||
**Mode:** DELIBERATE (high-risk: engine generalization, 60fps continuous/double-page, OCR/CAS feasibility, server/AI scope)
|
||||
**Owner plan file:** `docs/plans/2026-06-21-badnote-full-refactor.md`
|
||||
**Supersedes (in part):** `docs/plans/2026-06-21-badnote-phase1.md`
|
||||
|
||||
> **Grounding.** Written after reading the live code, not from memory. Verified sources:
|
||||
> live editor `lib/editor/canvas/{pen_canvas,ink_painters,pen_stroke,pen_editor_screen}.dart`,
|
||||
> `lib/editor/pdf/pen_capture_region.dart`, `lib/main.dart` (dynamic_color + `pdfrxFlutterInitialize` + `PenCaptureBinding`),
|
||||
> `lib/services/{database_service,pdf_service,stroke_rasterizer,undo_manager}.dart`,
|
||||
> `lib/services/ocr/onnx_recognition_backend.dart`, `lib/models/{ink_stroke,ink_point,pressure_curve}.dart`,
|
||||
> `lib/providers/settings_provider.dart`, `lib/screens/home_screen.dart`,
|
||||
> `server/badnote_server/{main,models,routers/sync_router}.py`, `server/README.md`,
|
||||
> `pubspec.yaml`, existing `test/` + `integration_test/` + `tool/` assets.
|
||||
>
|
||||
> **CRITICAL DIRECTION CHANGE captured here (vs phase1 plan):** the live `lib/editor/canvas/` has **already abandoned** the phase-1 "pdfrx `pageOverlaysBuilder`-hosted ink + RenderProxyBox arena bypass" architecture. The live `PenCanvas` owns **one `InteractiveViewer` + a single `Listener`** that draws a `PdfPageView` *bitmap* and the ink in the **same** child subtree (Saber clean-room model). pdfrx is used only as a **page renderer / document API** (`PdfDocument.openFile`, `PdfPageView`), never for gestures. `PenCaptureRegion`/`PenCaptureBinding` are installed in `main()` but are **NOT wired into the live canvas** and are slated for retirement (§6, §9). This plan builds the full product on the **own-canvas** model and records that decision in the ADR (§10).
|
||||
|
||||
---
|
||||
|
||||
## 0. Vision & Scope
|
||||
|
||||
**BadNote** is a pen-first, performant note-taking app. Primary device: **Windows Tablet + Surface Pen** (Flutter ≥ 3.44 for WM_POINTER stylus/multitouch); desktop for review/复习/search. Differentiators the user explicitly wants to win on: **(1)** library-wide full-text search over PDF text + typed text + handwriting OCR with jump-to-location; **(2)** modern cohesive **Material You** UI; **(3)** truly **book-like reading** (page-flip, two-page spread, thumbnail grid, reader-vs-annotate modes). Plus a **双链 sticky-note infinite board**, **one-notebook-per-PDF** with insertable blank pages and never-rasterized source, and a later **server sync + AI refinement** and **ink CAS**.
|
||||
|
||||
**Existing user data MAY be reset** — no migration burden; prefer the cleanest schema. **Stack stays Flutter** (own gesture pipeline; pdfrx only renders pages). GFW-aware CI already solved (pdfium pre-fetch + Dev Mode + vendored sqlite3 + onnxruntime pre-fetch).
|
||||
|
||||
### 0.1 The 11 feature areas → phase map (full vision; nothing dropped, everything sequenced)
|
||||
|
||||
| # | Feature | Phase | Risk |
|
||||
| --- | --- | --- | --- |
|
||||
| F1 | Pen-first editor core (harden): pressure, palm rejection, perfect_freehand quality, eraser, tools, undo/redo, **DB persistence of strokes per doc/page** | **P0** | Low–Med |
|
||||
| F2 | Page layout modes: continuous-single/-double, paged-single/-double + switcher; lazy render @60fps | **P1** | **High** |
|
||||
| F3 | Book-like reading: page-flip, two-page spread, reader vs annotate mode, quick jump | **P1** | Med |
|
||||
| F4 | Thumbnail-grid navigation (Drawboard-style), slider, no keyboard input | **P1** | Low |
|
||||
| F5 | Configurable pen: side-button + eraser-end mapping, pressure curve, palm sensitivity, finger-drawing toggle — Pen settings page | **P2** | Med |
|
||||
| F6 | One-notebook-per-PDF: page-level binding, insert blank pages between PDF pages, keep PDF vector, portable bundle + relink | **P2** | Med |
|
||||
| F7 | Infinite 便利贴 board + **双链** bidirectional links / backlinks | **P3** | Med–High |
|
||||
| F8 | Library-wide full-text search (PDF text + typed text + handwriting OCR) with snippets + jump | **P3** | **High (handwriting OCR)** |
|
||||
| F9 | Server sync + AI refinement (llm_wiki, VLM/LLM organize) | **P4** | High (scope) |
|
||||
| F10 | Ink CAS / formula recognition → searchable + solve toggle | **P5** | **Very High** |
|
||||
| F11 | Modern UX polish: Material You, subtoolbars, drag-reorder thumbnails, hover pre-warm | **woven across P0–P3** | Low |
|
||||
|
||||
### 0.2 Non-goals per phase
|
||||
- **P0:** no layout modes, no board, no text boxes, no search UI, no settings page beyond what exists, no server. Single-page editor + persistence + eraser/undo only.
|
||||
- **P1:** no 双链, no OCR-search, no server, no CAS. Layout/reading/thumbnails only.
|
||||
- **P2:** no server, no CAS, no handwriting OCR. Pen-config + notebook/bundle only.
|
||||
- **P3:** no CAS, no server-side AI. Board+双链 and **local** search (incl. best-effort handwriting OCR via existing text-line model; formula deferred to P5).
|
||||
- **P4:** sync + AI organize; CAS still out.
|
||||
- **P5:** formula OCR + CAS.
|
||||
- **All phases:** never rasterize the source PDF; no on-screen keyboard reliance (slider/grid nav).
|
||||
|
||||
---
|
||||
|
||||
## 1. Target Architecture
|
||||
|
||||
### 1.1 Layered module decomposition (target `lib/editor/`)
|
||||
|
||||
```
|
||||
lib/editor/
|
||||
├─ engine/ # host-agnostic ink engine (pure-ish, testable)
|
||||
│ ├─ coordinate_space_host.dart # CoordinateSpaceHost + NormalizedPageHost + BoardHost
|
||||
│ ├─ stroke_model.dart # EditorStroke/EditorPoint (canonical; replaces PenStroke split — §3, §4)
|
||||
│ ├─ stroke_store.dart # StrokeStore { committed, int revision } (O(1) shouldRepaint gate)
|
||||
│ ├─ stroke_geometry.dart # buildStrokeOutline(getStroke) — single source for screen+export (kills hairline bug)
|
||||
│ ├─ stroke_eraser.dart # pure splitStroke / eraseHits (extract from pen_canvas._eraseAt)
|
||||
│ └─ undo_stack.dart # global, commit-time-ordered, host-tagged (generalize undo_manager)
|
||||
├─ input/
|
||||
│ ├─ input_arbiter.dart # pure state machine: pointerCount>=2→pan/zoom, stylus→draw, palm reject
|
||||
│ └─ pen_config.dart # button/eraser-end action mapping, pressure curve, palm sensitivity (F5)
|
||||
├─ render/
|
||||
│ ├─ annotation_layer.dart # Stack(StaticInkPainter, LiveInkPainter, TextBoxLayer) over a host
|
||||
│ ├─ static_ink_painter.dart # reads ink_picture_cache, keyed by revision
|
||||
│ ├─ live_ink_painter.dart # in-progress stroke only
|
||||
│ └─ ink_picture_cache.dart # LRU<revision, ui.Picture> for STATIC INK — resolution-INDEPENDENT, NO DPI bucket (vector ink re-rasterizes crisp at composite)
|
||||
├─ layout/ # F2/F3 — page layout + reading
|
||||
│ ├─ page_layout.dart # enum {continuousSingle, continuousDouble, pagedSingle, pagedDouble}
|
||||
│ ├─ page_viewport.dart # lazy windowed page hosting (visible + cache extent), recenter
|
||||
│ └─ reader_controller.dart # reader vs annotate mode, page-flip, spread, quick-jump
|
||||
├─ pdf/
|
||||
│ ├─ pdf_document_source.dart # wraps PdfDocument (open/dispose/pageSize/render), notebook page-map (F6)
|
||||
│ ├─ page_tile.dart # PdfPageView host widget (bitmap) sized to host rect; drives tile DPI from transform scale (R11)
|
||||
│ └─ page_tile_cache.dart # LRU<TileKey{hostId, dpiBucket}, ui.Image> for rendered PAGE BITMAPS — DPI-bucketed (tiles blur on upscale); owns native-handle dispose (R11)
|
||||
├─ board/ # F7
|
||||
│ ├─ board_host_pane.dart # infinite InteractiveViewer board, reuses engine
|
||||
│ └─ link_graph.dart # 双链 backlink model + queries
|
||||
├─ text/ # OneNote-like editable text boxes (P3+, optional in P2 notebook)
|
||||
│ ├─ text_box_model.dart
|
||||
│ └─ text_box_layer.dart
|
||||
├─ persistence/
|
||||
│ ├─ editor_repository.dart # batched load/save per document; maps DB ↔ engine
|
||||
│ └─ save_scheduler.dart # debounced, synchronous-snapshot-before-await
|
||||
├─ search/ # F8 (P3)
|
||||
│ ├─ search_indexer.dart # PDF text + typed text + OCR → document_fts/library index
|
||||
│ └─ ocr_ingest.dart # bridge to lib/services/ocr (text-line; formula later)
|
||||
├─ sync/ # F9 (P4)
|
||||
│ └─ sync_client.dart # talks to server/ (already has push/pull/auth)
|
||||
└─ ui/
|
||||
├─ editor_screen.dart # top-level editor (replaces pen_editor_screen + pdf_annotator_screen)
|
||||
├─ editor_toolbar.dart # Material You subtoolbars (F11)
|
||||
├─ thumbnail_grid.dart # F4 nav
|
||||
└─ editor_shortcuts.dart # tool numbers, Ctrl+Z/Y/F/S (mouse/desktop)
|
||||
```
|
||||
|
||||
### 1.2 Dependency graph (acyclic)
|
||||
|
||||
```
|
||||
ui/ ─────────────► layout/ ─► pdf/ ─► (pdfrx PdfDocument/PdfPageView)
|
||||
│ │
|
||||
│ ├──────► render/ ─► engine/ (geometry, store, hosts)
|
||||
│ └──────► input/ ─► engine/
|
||||
├──► board/ ─────────────► render/, engine/
|
||||
├──► text/ ─────────────► engine/(hosts)
|
||||
├──► search/ ────────────► persistence/, services/ocr/
|
||||
├──► sync/ ────────────► persistence/, server API
|
||||
└──► persistence/ ───────► services/database_service.dart, engine/(stroke_model)
|
||||
```
|
||||
**engine/** depends on nothing in editor/ except itself (and `perfect_freehand`). **render/input/layout/** depend on engine. **ui/** is the only place wiring Riverpod controllers. This is the seam that lets PDF-page host, infinite board, and (P5) CAS overlay all reuse one renderer.
|
||||
|
||||
### 1.3 Coordinate model — single source of truth
|
||||
**Truth = host content coordinates** (already the live convention; generalize it):
|
||||
- **NormalizedPageHost:** points ∈ `[0,1]` of the page rect (today `PenPoint(nx, ny)`); width as fraction of page width (today `PenStroke.width`). Stored per `(documentId, pageIndex)`.
|
||||
- **BoardHost:** absolute logical px on an unbounded canvas; `InteractiveViewer` supplies pan/zoom.
|
||||
|
||||
`CoordinateSpaceHost` exposes `toContent(deviceLocal, deviceSize)`, `toDevice(content, deviceSize)`, `applyContentToCanvas(canvas, deviceSize)`. The live painters already scale `nx*pageSize.width` at paint time — formalize that into `applyContentToCanvas` (a `canvas.scale`) so no transformed-point copies are ever allocated. Text boxes are a **sanctioned widget-space exception** (Positioned widgets multiply by current zoom; stored truth stays content coords for search).
|
||||
|
||||
### 1.4 How today's `lib/editor/canvas/` evolves into this (no rewrite)
|
||||
|
||||
| Today (live) | Becomes | Action |
|
||||
| --- | --- | --- |
|
||||
| `pen_stroke.dart` `PenStroke`/`PenPoint` (in-memory only) | `engine/stroke_model.dart` `EditorStroke`/`EditorPoint` **persistable** (freezed/JSON, normalized) | **generalize + add JSON**; converge with `InkStroke` (§4) |
|
||||
| `ink_painters.dart` `StaticInkPainter`/`LiveInkPainter` + `buildStrokePath` | `render/static_ink_painter.dart` (+ `render/ink_picture_cache.dart`), `render/live_ink_painter.dart`, `engine/stroke_geometry.dart` | **move + add ink `ui.Picture` cache (resolution-independent, revision-keyed) + revision gate**; `buildStrokePath`→`buildStrokeOutline` shared with export |
|
||||
| `pen_canvas.dart` `_PenCanvasState` gesture logic (`_activePointers`, `_shouldDraw`, palm reject, `_eraseAt`) | `input/input_arbiter.dart` (pure SM) + `render/annotation_layer.dart` (widget) | **extract** SM out of the widget; eraser→`engine/stroke_eraser.dart` |
|
||||
| `pen_editor_screen.dart` single-page + `_strokesByPage` Map + `_transform` + slider | `ui/editor_screen.dart` + `layout/page_viewport.dart` + `persistence/editor_repository.dart` | **generalize** single-page → windowed multi-page; `_strokesByPage` → repository-backed `HostState`s |
|
||||
| `pen_capture_region.dart` + `PenCaptureBinding` (unused by live canvas) | — | **retire** after P0 confirms the `Listener`-in-shared-transform model holds on device (§6, §9) |
|
||||
| `pen_editor_screen` single `PdfPageView` bitmap inside the `InteractiveViewer` (1× layout-sized bitmap, matrix-scaled by pinch) | `pdf/page_tile.dart` **multi-resolution tile** whose render DPI tracks the transform scale | **NEW work (not relocation): own multi-resolution tiling** — pinch GPU-upscales a 1× bitmap → blurry at high zoom (R11). On zoom-settle, re-instantiate the tile (or `PdfPage.render()`) at a DPI matching the current scale into `pdf/page_tile_cache.dart` (DPI-bucketed page-bitmap cache, distinct from the ink cache). Budgeted P1/P0.5. |
|
||||
|
||||
Net: the four live files are **promoted, not thrown away**. P0 is mostly *relocation + persistence + revision-gated Picture cache*, which is low risk and immediately shippable. The one genuinely **new** burden the own-canvas choice imposes (beyond layout/windowing) is **multi-resolution page-tile rendering** — see R11 (§7) and the P0.5 gate (§5/§9): we own crisp-on-zoom re-rasterization that pdfrx's own viewer would have given for free.
|
||||
|
||||
---
|
||||
|
||||
## 2. Per-feature design (file-level)
|
||||
|
||||
### F1 — Pen-first editor core + persistence (P0)
|
||||
**Add/modify:**
|
||||
- `engine/stroke_model.dart` **[ADD]** — `EditorStroke { id, List<EditorPoint> points, EditorTool tool, int color, double width, bool filled, String? textContent, double fontSize }`, `EditorPoint { double x, y; double? pressure; double? tilt; int? timestamp; InputDeviceKind? pointerDeviceKind }`. freezed + `toJson/fromJson`. (Normalized; `width` = fraction of page width, matching live `PenStroke`.) **SF1 — non-lossy superset of `InkStroke`/`InkPoint`:** `InkPoint` carries `tilt`/`timestamp`/`pointerDeviceKind` (database_service.dart:346–362) that the live `PenPoint` drops; `EditorPoint` includes them now (nullable, cheap with freezed) so the `InkStroke`↔`EditorStroke` adapter (for OCR/export reuse) is **lossless in both directions**. The live `PenCanvas` simply leaves the extra fields null at capture; existing `InkStroke` data round-trips intact. (Avoids a silent data-loss footgun for OCR, which keys on `pointerDeviceKind`/`pressure`.)
|
||||
- `engine/stroke_store.dart` **[ADD]** — `class StrokeStore { List<EditorStroke> committed; int revision; add/removeAt/replace bump revision; }`. Fixes the live `pen_editor_screen._commitStroke` "new list identity" hack by making revision explicit.
|
||||
- `engine/stroke_geometry.dart` **[ADD]** — `Path buildStrokeOutline(EditorStroke, Size, {bool isComplete})` lifted verbatim from `ink_painters.buildStrokePath` (the proven `getStroke(thinning: hl?0:0.7, smoothing:.5, streamline:.5, simulatePressure: !hasRealPressure && !hl)` recipe). **Single source** for screen + export.
|
||||
- `engine/stroke_eraser.dart` **[ADD]** — pure `eraseHits(strokes, point, radius)` (extract live `_eraseAt`) + `splitStroke` for partial/segment erase. **Critic note — `splitStroke` is NEW behavior, not an extraction:** the live `pen_canvas._eraseAt` is **whole-stroke** (it removes the entire stroke on the first proximity hit and returns). There is no segment-split logic in the live canvas to port — `undo_manager.removeStroke` only *records* replacements, it does not compute them. So `splitStroke` (point-run splitting → 0/1/2 sub-strokes) is greenfield code; its tests (§8) exercise new functionality, not a regression port. `eraseHits` (whole-stroke) IS an extraction of the live behavior and stays available as the default erase mode.
|
||||
- `engine/undo_stack.dart` **[ADD]** — generalize `lib/services/undo_manager.dart` to host-tagged, commit-time-ordered global stack (entries carry `hostId`+`pageIndex`).
|
||||
- `render/{annotation_layer,static_ink_painter,live_ink_painter,ink_picture_cache}.dart` **[ADD]** — relocate live painters; `StaticInkPainter` reads `ink_picture_cache` (`LRU<revision, ui.Picture>`) keyed by `revision` (`shouldRepaint = old.revision != revision` — O(1)); per-host `RepaintBoundary`. **`ink_picture_cache` is for STATIC VECTOR INK only — resolution-INDEPENDENT, NO DPI bucket** (a `ui.Picture` of vector strokes re-rasterizes crisp at composite time at any zoom). It is a **different cache from the page-bitmap `page_tile_cache`** (§2/F2, R11), which IS DPI-bucketed because raster page tiles blur on upscale. Do not conflate the two.
|
||||
- `input/input_arbiter.dart` **[ADD]** — pure SM extracted from `pen_canvas` (`idle→inking→erasing`, `pointerCount>=2`→cancel+pan, palm reject = touch dropped while stylus active, eraser = `kSecondaryButton || invertedStylus`). Keeps the live pressure-normalization logic (`_normalizedPressure`).
|
||||
- `persistence/editor_repository.dart` **[ADD]** — `loadDocument(documentId) → Map<int,StrokeStore>` (one batched query), `saveHost(documentId, hostId, EditorStroke list)`.
|
||||
- **MF3 — write contract (load-bearing for the per-row choice):** `saveHost` MUST **diff by stroke id** against the rows already on disk for that host — **UPSERT only changed/new rows, DELETE only removed rows**. It must NOT delete-all-rows-for-host then re-insert (that is exactly what the live notes path does at database_service.dart:281–306, and at 2,000 strokes it is **slower** than a single blob rewrite — re-inserting 2,000 rows per save). The per-stroke-row schema (§3) is justified **only** under this diff contract: erasing 1 of 2,000 strokes ⇒ 1 DELETE, 0 re-inserts; adding 1 stroke ⇒ 1 INSERT. The `SaveScheduler` hands `saveHost` the synchronously-captured stroke snapshot; the repository keeps a last-persisted id-set per host to compute the diff. If diffing proves fiddly under churn, the fallback is a per-page blob (NOT delete-all+reinsert) — but the diff path is the default and is what makes per-row worthwhile.
|
||||
- **Durability invariant (Critic):** the UPSERT(s) + DELETE(s) for one `saveHost` MUST run inside **one `sqflite` transaction**, and the in-memory **per-host last-persisted id-set is updated ONLY after that transaction commits** (in the `then`/post-await success path) — never optimistically before the write. A crash or interruption mid-diff must leave memory and disk consistent: either the whole diff applied (and the id-set advances) or none of it did (and the id-set is unchanged, so the next save re-derives the same diff and retries). This mirrors the existing `updateNote` interruption warning (database_service.dart:281–306: "an interruption mid-way would permanently lose strokes, so the whole sequence must run inside one transaction").
|
||||
- `persistence/save_scheduler.dart` **[ADD]** — debounced ~800ms; **serialize JSON synchronously before any await**; flush on page-leave/dispose.
|
||||
- `ui/editor_screen.dart`, `ui/editor_toolbar.dart` **[ADD]** — port `pen_editor_screen` UI (floating Material You palette + page pill already good).
|
||||
|
||||
**Public interfaces (key):**
|
||||
```dart
|
||||
abstract class CoordinateSpaceHost {
|
||||
String get hostId; int get pageIndex;
|
||||
Offset toContent(Offset deviceLocal, Size deviceSize);
|
||||
Offset toDevice(Offset content, Size deviceSize);
|
||||
void applyContentToCanvas(Canvas c, Size deviceSize);
|
||||
}
|
||||
class EditorController extends ChangeNotifier { // ui-facing, Riverpod-provided
|
||||
void beginStroke(CoordinateSpaceHost h, EditorPoint p);
|
||||
void extendStroke(EditorPoint p);
|
||||
void commitStroke(); // getStroke once → append → revision++ → undo push → scheduleSave
|
||||
void eraseAt(CoordinateSpaceHost h, Offset content, double radius);
|
||||
void undo(); void redo();
|
||||
void setTool(EditorTool t); void setColor(Color c); void setLayout(PageLayout l);
|
||||
}
|
||||
```
|
||||
**Acceptance:** draw/erase/undo/redo on a page; close+reopen → strokes persisted (DB); `StaticInkPainter.shouldRepaint==false` while drawing a new stroke on a 2,000-stroke page (revision constant); no per-frame point allocation for committed strokes.
|
||||
|
||||
### F2 — Page layout modes (P1)
|
||||
- `layout/page_layout.dart` **[ADD]** — `enum PageLayout { continuousSingle, continuousDouble, pagedSingle, pagedDouble }`.
|
||||
- `layout/page_viewport.dart` **[ADD]** — windowed lazy hosting: only pages in `[firstVisible - cacheExtent, lastVisible + cacheExtent]` mount an `AnnotationLayer` + `PageTile`; others are disposed (Picture evicted). Continuous = scrollable column/two-column; paged = `PageView`. Recenter on layout switch.
|
||||
- `pdf/pdf_document_source.dart` **[ADD]** — `PdfDocument` wrapper exposing `pageSize(i)`, `renderTile(...)`, page count; owns dispose. Replaces ad-hoc `PdfDocument.openFile` in `pen_editor_screen`.
|
||||
- `pdf/page_tile.dart` **[ADD] — multi-resolution tile (R11 mitigation).** Under one shared `InteractiveViewer`, a `PdfPageView` renders a bitmap sized to its **layout constraints × devicePixelRatio** and the matrix scales that 1× bitmap, so pinch-zoom GPU-**upscales** → blurry text/rules at high zoom (unlike pdfrx's own viewer, which re-renders crisp tiles per zoom level). `page_tile` watches the transform scale and on **zoom-settle** (debounced) re-instantiates its render at a DPI matching the current scale — via either a re-laid-out `PdfPageView` at the new pixel size or `PdfPage.render(width/height at target DPI)` into `page_tile_cache`. **Cap retained DPI** (e.g. ≤ 3× base) to bound memory; downscale path stays matrix-only (sharp enough). This is **new own-canvas work**, not relocation.
|
||||
- `pdf/page_tile_cache.dart` **[ADD]** — `LRU<TileKey{hostId, dpiBucket}, ui.Image>` of rendered **page bitmaps** (NOT ink). Key includes the **DPI bucket** so a page re-rendered at higher DPI replaces (not duplicates) its lower-DPI tile. **Owns the native-handle dispose lifecycle** of each `PdfPage.render()` result (`PdfImage` → backing `ui.Image`), deferred to post-frame so the raster thread is done with an evicted tile before disposal. **Distinct from `render/ink_picture_cache.dart`** (§2/F1) which holds resolution-independent ink `ui.Picture`s with no DPI bucket.
|
||||
- `ui/editor_toolbar.dart` **[MODIFY]** — layout switcher control.
|
||||
|
||||
**Two separate caches + memory budgets (Critic — do not conflate):**
|
||||
- **`ink_picture_cache` (vector ink):** `LRU<revision, ui.Picture>`, ~8–12 mounted-page Pictures, dispose deferred to post-frame. Budget ~ a few MB (vector op-lists are cheap). No DPI bucket.
|
||||
- **`page_tile_cache` (raster page bitmaps):** `LRU<TileKey, ui.Image>`. **This is the heavy one.** A single A4 page rendered at 3× DPI is ~**10–30 MB** (≈ 1785×2526 px × 4 bytes ≈ 18 MB at 3× of a 595×842 pt page @ ~2 dppt). So **8–12 tiles at 3× would be ~150–350 MB — the old "≤64 MB / 8–12 pages" figure (R10) was unit-confused** (it conflated ink Pictures with page bitmaps). **Resolution:** size the *tile* window to the device memory budget, not a fixed page count — e.g. keep **full-DPI tiles only for the visible + ±1 pages (≈ 4 in double-page), and downgrade off-window pages to a 1× thumbnail tier** (matrix-upscaled, accepted as blurry only while scrolling). The ink cache keeps its wider ~8–12 window (cheap). R10's ≤64 MB now applies to **ink + downgraded-tier tiles**; the small high-DPI tile set is budgeted separately (~64–128 MB depending on device), tuned in P0.5/P1.
|
||||
|
||||
**Data structures:** `PageWindow { int first, last }`; `Map<int, HostState> mountedHosts`; `TileKey { hostId, dpiBucket }`. **Acceptance:** all four modes render; switching recenters; double-page shows two pages side-by-side; ink tracks scroll/zoom; **page text/rules stay crisp at 4× zoom (no GPU-upscale blur) — DPI refreshes on zoom-settle** (R11); combined cache memory stays within the device budget under a full scroll; perf gate (§7) holds.
|
||||
|
||||
### F3 — Book-like reading (P1)
|
||||
- `layout/reader_controller.dart` **[ADD]** — `ReaderMode { read, annotate }`; in `read` the arbiter never draws (pen ignored); page-flip animation for `pagedSingle/Double` via `PageView` physics; spread layout from `pagedDouble`.
|
||||
- `ui/editor_screen.dart` **[MODIFY]** — mode toggle in toolbar; quick-jump via slider/grid (no keyboard).
|
||||
|
||||
**Acceptance:** read mode blocks ink; annotate mode draws; page-flip animates; two-page spread aligns facing pages; quick-jump scrolls/animates to target.
|
||||
|
||||
### F4 — Thumbnail-grid navigation (P1)
|
||||
- `ui/thumbnail_grid.dart` **[ADD]** — Drawboard-style grid; tiles render via `pdf/pdf_document_source.renderTile` (replaces `lib/services/thumbnail_service.dart` syncfusion path). Tap → jump; **drag-reorder** for notebook page order (F6). Slider remains the linear scrubber (already in `pen_editor_screen._buildPagePill`).
|
||||
- `lib/services/thumbnail_service.dart` **[MODIFY/RETIRE]** — migrate to pdfrx render; drop `syncfusion_pdfviewer_platform_interface` dep once ported.
|
||||
|
||||
**Acceptance:** grid shows all pages; tap jumps; drag reorders (F6); no text-field page input.
|
||||
|
||||
### F5 — Configurable pen (P2)
|
||||
- `input/pen_config.dart` **[ADD]** — `PenConfig { ButtonAction sideButton; ButtonAction eraserEnd; PressureCurveType curve; double palmSensitivity; bool fingerDrawing; }`; `enum ButtonAction { eraser, undo, toggleTool, pan, lasso, none }`. `InputArbiter` consults it (replaces hardcoded `_isEraserSignal`).
|
||||
- `lib/providers/settings_provider.dart` **[MODIFY]** — persist `PenConfig` (extend existing `SharedPreferences` notifier; it already stores pressure curve + stabilization).
|
||||
- `lib/screens/settings_screen.dart` **[MODIFY]** / **[ADD]** `ui/pen_settings_page.dart` — mapping UI (SpeedyNote-style dialog).
|
||||
- Reuse `lib/models/pressure_curve.dart` (already has linear/soft/hard/custom + `apply`).
|
||||
|
||||
**Acceptance:** remapping side-button to undo makes the barrel button undo; pressure curve changes stroke taper; palm sensitivity changes touch-cooldown; finger-drawing toggle works; all persist across restart.
|
||||
|
||||
### F6 — One-notebook-per-PDF (P2)
|
||||
- `pdf/pdf_document_source.dart` **[MODIFY]** — a **page-map**: logical notebook pages → either a source-PDF page index or a synthetic blank page. Insert-blank adds a synthetic page **without rasterizing or editing the source bytes** (keep source vector + searchable). Ink/text bind to the **logical** page id (stable UUID per logical page), not raw PDF index, so inserts don't reshuffle annotations.
|
||||
- `lib/services/pdf_service.dart` **[MODIFY]** — keep headless syncfusion export/mutate; export walks the page-map. **Fix the hairline bug**: `_renderStrokes` must build a `PdfPath` from `buildStrokeOutline` points and **fill** it (currently strokes line-segments → hairline). Shape/line/arrow keep stroke semantics.
|
||||
- DB **[MODIFY]** — `notebook_pages(id, document_id, ordinal, source_page_index INTEGER NULL, kind)`; ink keyed by `notebook_page_id` (§4). Portable bundle = zip {source.pdf, badnote.json(strokes/text/links/page-map)} + relink-on-open (match by content hash, fall back to picker).
|
||||
|
||||
**Acceptance:** insert blank page between PDF pages → source PDF untouched (still vector/searchable in another viewer); reorder pages keeps ink attached; export `.pdf` shows filled ink matching screen (golden); bundle round-trips on another machine.
|
||||
|
||||
### F7 — Infinite board + 双链 (P3)
|
||||
- `board/board_host_pane.dart` **[ADD]** — `BoardHost` (absolute px) in a constrained-false `InteractiveViewer`; reuses `AnnotationLayer` + engine. Migrates `lib/screens/split_view_screen.dart` + `scratchpads` table.
|
||||
- `board/link_graph.dart` **[ADD]** — `Link { srcRef, dstRef }` where a ref is `(kind: notebookPage|board|note, id)`; `backlinksOf(ref)`. Wiki-style `[[...]]` parsing in text boxes; sticky-note = a small board region or text box that can link to a page/note. Backlink panel queries `links` table.
|
||||
- DB **[ADD]** — `links(id, src_kind, src_id, dst_kind, dst_id, created_at)` + indexes both directions; `boards(id, document_id NULL, strokes_json, ...)` (generalize `scratchpads`).
|
||||
|
||||
**Acceptance:** create a sticky linking page 3 → note X; open note X shows a backlink to page 3; board draws at perf target; link graph survives restart; deleting a target leaves a dangling-link indicator (no crash).
|
||||
|
||||
### F8 — Library-wide full-text search (P3)
|
||||
- `search/search_indexer.dart` **[ADD]** — index three sources into a unified library index: **(a)** PDF embedded text (pdfrx `PdfPage.loadText`/`charRects` text API — source-pinned in P0.5 per SF4), **(b)** typed text boxes (`content`), **(c)** handwriting OCR (existing `services/ocr` text-line ONNX over rasterized strokes via `StrokeRasterizer`). Reuse existing `document_fts` FTS5 + a new `library_fts` spanning notebooks/boards/notes with `(ref_kind, ref_id, page, snippet)`.
|
||||
- `search/ocr_ingest.dart` **[ADD]** — batch handwriting OCR per logical page (best-effort; set expectations: text-line only, no math). Background isolate; debounced after ink idle. **The OCR backend itself no-ops when the model is unavailable (onnx_recognition_backend.dart header: "verify on-device") — search MUST never block on, nor be gated by, OCR results.**
|
||||
- `lib/screens/search_screen.dart` **[MODIFY]** — unified results with snippets + **jump-to-location** (open editor at the page + scroll, or board at the region).
|
||||
|
||||
**Acceptance — split into two exit tiers (SF2):**
|
||||
- **COMMITTED (P3 exit blocker):** PDF embedded text + typed-text-box search returns results with snippets; tapping a result opens and scrolls to the page/region. This tier alone satisfies the P3 search exit (it has no unverified-recall dependency).
|
||||
- **ADDITIVE (best-effort, NOT a P3 exit blocker):** handwriting OCR contributes hits when the text-line model is available and confident; a word handwritten on page 5 may be returned. **Expectation set:** handwriting recall is limited by the text-line model and may return nothing on cursive/handwriting; math/formula is explicitly P5. P3 ships even if handwriting recall is poor, with UI copy stating the limit.
|
||||
|
||||
### F9 — Server sync + AI refinement (P4)
|
||||
- `sync/sync_client.dart` **[ADD]** — wire the **already-built** FastAPI endpoints (`/api/auth`, `/api/sync/push|pull`, `/api/notes`, `/api/documents`, `/api/ocr`). Last-writer-wins by `updated_at` (server already implements this). Sync notes + (later) notebooks/boards.
|
||||
- `server/badnote_server/` **[MODIFY]** — extend `sync_router`/`models.py` to cover notebooks/boards/links (today only notes). Add an **AI-refine** endpoint that runs llm_wiki/VLM/LLM over a note/page → returns organized markdown (server-side, heavy deps gated like the OCR worker).
|
||||
- `lib/providers/` **[ADD]** sync state provider; settings page server URL/token.
|
||||
|
||||
**Acceptance:** push from device A, pull on device B reproduces notes; AI-refine returns organized text for a selected note; offline still fully functional (server optional, per README).
|
||||
|
||||
### F10 — Ink CAS / formula (P5)
|
||||
- `search/ocr_ingest.dart` **[MODIFY]** — add a **formula/math recognition** backend (new model; the current PP-OCRv4 is text-line only — this is the known hard sub-problem). Behind a feature flag.
|
||||
- `engine/` **[ADD]** `cas/` — recognized formula → CAS (compute/solve) behind a toggle; renders result near the ink. Reuses the `CoordinateSpaceHost` seam for overlay placement.
|
||||
|
||||
**Acceptance (stretch):** a handwritten `2+3=` toggled → shows `5`; recognized formulas become searchable. Gated, optional, lowest priority.
|
||||
|
||||
### F11 — Modern UX polish (woven P0–P3)
|
||||
- Material You already wired (`main.dart` `DynamicColorBuilder` + harmonized schemes + Inter). Continue: subtoolbars in `editor_toolbar`, drag-reorder thumbnails (F4/F6), **hover pre-warm** (warm the next page tile + Picture on stylus hover to cut pen-down latency — reuse the live `_onPointerHover` seam), animated mode/layout transitions.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data model & persistence (clean schema — data may reset)
|
||||
|
||||
Bump DB to a fresh version with `_onCreate` only (keep `_onUpgrade` harmless). Today: notes/strokes, documents/annotations(JSON-per-page), bookmarks, ocr_results, document_fts(FTS5), scratchpads (DB v5, `lib/services/database_service.dart`).
|
||||
|
||||
**Target tables (additions/changes in bold):**
|
||||
- `documents` **[KEEP]** (drop editor reliance on `rotation`; page-map owns rotation).
|
||||
- **`notebook_pages(id PK, document_id FK, ordinal INTEGER, source_page_index INTEGER NULL, kind TEXT, created_at)`** **[ADD]** — F6 logical pages.
|
||||
- **`ink(id PK, host_kind TEXT, host_id TEXT, stroke_json TEXT, ordinal INTEGER, updated_at)`** **[ADD]** — strokes addressed by host (`host_kind ∈ {page, board}`, `host_id` = `notebook_page_id` or `board_id`). **Per-stroke rows** (not per-page blob) so a 2,000-stroke page doesn't rewrite on every save — **but ONLY valid under the MF3 diff-write contract (§2/F1 `editor_repository.saveHost`): UPSERT changed rows + DELETE removed rows, never delete-all+re-insert.** Without diffing, per-row is *worse* than a blob; the contract is what makes this schema correct. Index `(host_kind, host_id)`.
|
||||
- **`text_boxes(id PK, host_kind, host_id, content TEXT, rect_json, font_size, color, updated_at)`** **[ADD]** — F2-area text + F8 indexing.
|
||||
- **`boards(id PK, document_id FK NULL, title, created_at, updated_at)`** **[ADD]** — generalize `scratchpads`; board strokes live in `ink` with `host_kind='board'`.
|
||||
- **`links(id PK, src_kind, src_id, dst_kind, dst_id, created_at)`** **[ADD]** — 双链; indexes on `(src_kind,src_id)` and `(dst_kind,dst_id)`.
|
||||
- `bookmarks` **[KEEP]**.
|
||||
- `ocr_results` **[KEEP/EXTEND]** — per logical page handwriting OCR text.
|
||||
- **`library_fts` (FTS5)** **[ADD]** — `(ref_kind, ref_id, page, content)` unified search over PDF text + typed text + OCR. `document_fts` **[KEEP]** for PDF-page text.
|
||||
- `notes`/`strokes`/`notes_fts` **[KEEP]** (existing ink-note path; eventually folded into boards, but not deleted in P0).
|
||||
- **`sync_state(entity_kind, entity_id, last_pushed_at, last_pulled_at, dirty)`** **[ADD, P4]**.
|
||||
|
||||
**Coordinate semantics on disk:** page ink = normalized `[0,1]` unrotated; board ink = absolute logical px. (Matches live conventions.)
|
||||
|
||||
**Stroke-model convergence (important):** there are currently **two** stroke models — `PenStroke/PenPoint` (live canvas, in-memory only) and `InkStroke/InkPoint` (freezed/JSON, DB+OCR+export). P0 introduces **one** canonical `EditorStroke/EditorPoint` (freezed/JSON, normalized) and adapters to/from `InkStroke` for OCR/export reuse during transition; old screens keep `InkStroke` until retired (§6).
|
||||
|
||||
---
|
||||
|
||||
## 4. Refactor strategy (evolve, don't rewrite)
|
||||
|
||||
**Principle: every phase ships; the engine generalizes under load.**
|
||||
|
||||
1. **P0 = relocation + persistence.** Move the 4 live canvas files into `engine/` + `render/` + `input/` with minimal logic change; add `StrokeStore.revision` + `ui.Picture` cache + DB persistence via `editor_repository`. The live single-page editor keeps working throughout. **Keep** `PenStroke` as a thin alias of `EditorStroke` until callers migrate.
|
||||
2. **P0.5 = vertical slice (continuous-single only) — the new perf/crispness gate (see §5/§9).** Stand up `layout/page_viewport` + `pdf/page_tile` rendering continuous-SINGLE only, with the **rewritten** perf bench (targeting `ui/editor_screen`, not the spike) and the zoom-DPI refresh proven crisp at 4× on the Surface. This proves the own-canvas multi-page + multi-resolution model on the real device before any double/paged/spread work.
|
||||
3. **P1 = full multi-page windowing.** Wrap the (now-relocated) `AnnotationLayer` in `page_viewport`; `_strokesByPage` Map → repository-backed mounted hosts. Single-page path stays as `pagedSingle`. Continuous-double / paged / spread land here, AFTER P0.5 passes.
|
||||
4. **Old-code retirement (gated on parity, then delete in one step):**
|
||||
- **[DELETE after P1 parity]** `lib/screens/pdf_annotator_screen.dart`, `lib/widgets/pdf_annotation_layer.dart`, `lib/widgets/ink_canvas.dart` (extract any unique draw/erase logic to `engine/` first), `lib/editor/pdf/spike_*.dart` (throwaway M1 spike — includes the spike-based perf bench, replaced in P0.5), `lib/editor/pdf/pen_capture_region.dart` + `PenCaptureBinding` in `main.dart` (own-canvas `Listener` model won — remove the unused arena-bypass binding once P0 device-confirms).
|
||||
- **[DELETE in P3, NOT P1 (SF3)]** `lib/screens/split_view_screen.dart` — its replacement is the **P3** infinite board (F7); deleting it in P1 would leave a 2-phase functionality gap (no scratchpad between P1 and P3). It stays live until the board lands.
|
||||
- **[KEEP]** `thumbnail_service` until F4 ports it; `stroke_rasterizer` (OCR), `ctc_decoder`, `onnx_recognition_backend`, OCR assets; `pdf_service` (export/mutate, with the fill fix); `pptx_service`/`ppt_annotator_screen` (PPT is separate; not in scope but not deleted).
|
||||
5. **Navigation swap:** `home_screen` `_openDocument`/`_importPdf` currently push `PdfAnnotatorScreen`; `openM1Spike` pushes `PenEditorScreen`. Repoint both to `ui/editor_screen.dart` once P1 parity passes (checklist below).
|
||||
6. **Parity checklist before any delete:** page rotate/delete/insert-blank/insert-image (`pdf_service`), bookmark add/toggle/jump, undo/redo across pages, save-on-leave, zoom in/out/fit, export-matches-screen (golden). Mirrors phase-1 §10 M3 checklist C1–C11.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phased delivery (each = shippable milestone with exit criteria)
|
||||
|
||||
> Sequence de-risks: pen core + persistence first (P0), then the 60fps multi-mode layout (P1), then config/notebook (P2), then board/双链/search (P3), then sync/AI (P4), then CAS (P5).
|
||||
|
||||
**P0 — Pen core hardened + persisted (own-canvas).** *Exit:* draw/erase/undo/redo on a single PDF page persist to DB and reload; `EditorStroke` canonical model (superset of `InkPoint`, SF1) + revision-gated `ui.Picture` cache; `editor_repository.saveHost` honors the MF3 diff-write contract (test: erase 1 of 2,000 ⇒ 1 DELETE, 0 re-inserts); pure `InputArbiter` + `stroke_eraser` unit-tested; **on-device Surface Pen confirms** pressure + palm rejection + pinch-zoom in the live shared-transform model (the one device gate). Export hairline bug fixed (fill). No regression to existing screens.
|
||||
|
||||
**P0.5 — Vertical slice gate (own-canvas multi-page + multi-resolution), continuous-SINGLE only. [NEW — SYNTHESIS]** Stand up `layout/page_viewport` + `pdf/page_tile` rendering **continuous-single only** on the real document path (`ui/editor_screen`), and:
|
||||
- (a) **REWRITE the perf bench** (§7/§8): a new `integration_test/editor_scroll_bench.dart` drives `ui/editor_screen` + `layout/page_viewport`; **DELETE the spike-based `integration_test/perf_scroll_bench.dart`** (it imports `spike_editor_pane.dart` / `pageOverlaysBuilder` = the invalidated Option B; it CANNOT validate own-canvas).
|
||||
- (b) **Prove crisp-on-zoom (R11):** page-tile DPI refresh on zoom-settle renders crisp text/rules at 4× on the Surface (no GPU-upscale blur).
|
||||
- (c) **Source-pin the pdfrx render/text APIs (SF4):** confirm `PdfPage.loadText`/`charRects`/`PdfPage.render()`/`PdfPageView` signatures present in pdfrx 2.4.4; smoke-test each, so F4/F8 don't hit drift late.
|
||||
- *Exit (hard gate before any double/paged/spread):* **continuous-single median build+raster ≤ 16.6ms, p95 ≤ 22ms** on the 300-page asset via the REWRITTEN bench, **AND** a manual crisp-on-zoom-at-4× PASS on the Surface, **AND** the four pdfrx APIs source-pinned + smoke-tested. Continuous-double / paged / spread do NOT begin until P0.5 is GREEN.
|
||||
|
||||
**P1 — Full layout modes + book-like reading + thumbnails (60fps). Precondition: P0.5 GREEN.** *Exit:* the remaining `PageLayout`s (continuous-double, pagedSingle, pagedDouble); reader vs annotate; page-flip + two-page spread; thumbnail-grid jump + drag-reorder; windowed lazy hosting; **perf gates met** (§7) incl. continuous-double on a 300-page PDF via the rewritten bench; old PDF screens deleted after parity checklist (split_view retirement deferred to P3 per SF3); nav repointed.
|
||||
|
||||
**P2 — Configurable pen + one-notebook-per-PDF.** *Exit:* Pen settings page (button/eraser mapping, curve, palm sensitivity, finger toggle) persists + drives arbiter; insert-blank/reorder pages keep source vector + ink attached to logical pages; portable bundle round-trips; export walks page-map.
|
||||
|
||||
**P3 — Infinite board + 双链 + library search.** *Exit:* board reuses engine at perf; sticky-notes + `[[links]]` produce backlinks; **library search COMMITTED tier** (PDF embedded text + typed-text-box) returns snippets + jump-to-location (this tier is the exit blocker, SF2); **handwriting-OCR tier is ADDITIVE/best-effort and does NOT block P3 exit**; `split_view_screen` retired into board (the SF3 deletion point).
|
||||
|
||||
**P4 — Server sync + AI refinement.** *Exit:* push/pull notes+notebooks across two devices (LWW); AI-refine endpoint returns organized markdown; app fully functional offline.
|
||||
|
||||
**P5 — Formula OCR + ink CAS.** *Exit (stretch, gated):* formula recognition backend; searchable formulas; CAS solve toggle for simple expressions.
|
||||
|
||||
Every milestone ends with verifier/critic pass + perf-results doc updated with commit hash.
|
||||
|
||||
---
|
||||
|
||||
## 6. Refactor of input transport (record the pivot)
|
||||
The live `PenCanvas` proves the **own-canvas** model: a single `Listener` over an `InteractiveViewer` whose child is `Stack(PdfPageView bitmap, StaticInk, LiveInk)`. Because the pen, page bitmap, and ink share **one** transform and the `Listener` arbitrates by `_activePointers.length` + `kind`, there is **no gesture-arena fight** — pdfrx never sees gestures (it only renders). This **removes** the phase-1 need for `PenCaptureRegion`/`PenCaptureBinding` (RenderProxyBox arena bypass), which exist for the abandoned `pageOverlaysBuilder` approach. **Retirement gate:** delete them once P0 confirms on the physical Surface Pen that the `Listener` model handles stylus draw + single-finger pan + pinch-zoom + palm rejection (the live code is built for exactly this; confirm, then remove the dead binding).
|
||||
|
||||
---
|
||||
|
||||
## 7. Risks & mitigations
|
||||
|
||||
| # | Risk | L | I | Mitigation / trigger |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| R1 | **60fps continuous + double-page on a 300-page PDF.** Two columns × windowed tiles × ink Pictures may blow frame budget. | Med | High | Windowed lazy hosting (only visible+cacheExtent mounted), bounded LRU + post-frame dispose for BOTH the `ink_picture_cache` and the `page_tile_cache` (R10), per-page `RepaintBoundary`, revision-gated static Picture. **Gate continuous-SINGLE in P0.5, continuous-double in P1** (§5/§9). **MF1 — the existing `integration_test/perf_scroll_bench.dart` is UNUSABLE here: it imports `spike_editor_pane.dart` and benchmarks `pageOverlaysBuilder` (Option B, the INVALIDATED architecture) — it cannot validate own-canvas. P0.5 REWRITES the bench against `ui/editor_screen` + `layout/page_viewport` and DELETES the spike-based one.** Uses `large_300p.pdf`. |
|
||||
| R2 | **Pen feel only verifiable on the user's Surface** (CI has no pen). | High | Med | P0 device gate (pressure/palm/pinch) + manual checklist in perf-results doc; synthesized-stylus widget tests as interim signal only. |
|
||||
| R3 | **Handwriting/formula OCR feasibility.** Current model is text-line PP-OCRv4; math is unsolved. | High | Med | F8 ships text-line best-effort with **explicit expectation-setting**; formula isolated to P5 behind a flag; never block search on OCR quality. |
|
||||
| R4 | **双链 graph scale** (thousands of links/sticky-notes). | Low | Med | Indexed `links` table (both directions), lazy backlink queries, no in-memory full graph. |
|
||||
| R5 | **Server/AI scope creep.** | Med | Med | Server stays optional (README); P4 wires existing endpoints + one AI-refine route; AI heavy deps gated like OCR worker. |
|
||||
| R6 | **Windows pen edge cases** (no advertised pressure range; barrel-button eraser; inverted stylus). | Med | Med | Live `_normalizedPressure` already handles degenerate ranges; F5 makes button/eraser mappable; test matrix in arbiter unit tests. |
|
||||
| R7 | **Export fidelity** (hairline bug today). | High (today) | Med | `stroke_geometry.buildStrokeOutline` shared screen+export; `pdf_service._renderStrokes` **fills** a `PdfPath`; golden test (P0). |
|
||||
| R8 | **Stroke-model convergence churn** (two models today). | Med | Low | One canonical `EditorStroke` + adapters; old `InkStroke` retained only where old screens/OCR/export still use it, deleted with them. |
|
||||
| R9 | **pdfrx page-render / text API drift** (thumbnails, search text, double-page sizing). | Med | Low | **Source-pin `PdfDocument`/`PdfPageView`/`PdfPage.loadText`/`charRects`/`render()` signatures + smoke-test in P0.5 (pulled forward, SF4)** so F4/F8 don't hit drift late; keep syncfusion export until verified. |
|
||||
| R10 | **Cache memory budget vs many mounted pages (TWO caches — Critic).** | Med | Med | **`ink_picture_cache`** (vector `ui.Picture`, no DPI bucket): ~8–12 mounted pages, a few MB. **`page_tile_cache`** (raster `ui.Image`, DPI-bucketed): the heavy store — a 3×-DPI A4 tile is ~10–30 MB, so 8–12 high-DPI tiles would be ~150–350 MB. **The old "≤64MB / 8–12 pages" figure was unit-confused (conflated ink with page bitmaps).** Fix (§2/F2): keep full-DPI tiles only for visible ±1 pages, downgrade off-window pages to a 1× thumbnail tier; ≤64MB applies to ink + downgraded tiles, the small high-DPI tile set budgeted separately (~64–128MB, device-tuned). Sample BOTH caches during the scroll bench. |
|
||||
| R11 | **Blurry page at high zoom under the shared transform (MF2).** One `InteractiveViewer` matrix-scales a `PdfPageView` bitmap that was rendered at **layout-constraint × devicePixelRatio** (1×); pinch-zoom GPU-**upscales** it → blurry text/rules at 4×, whereas pdfrx's own viewer re-renders crisp tiles per zoom level. We **own multi-resolution tiling**, not just layout/windowing. | Med | High | `pdf/page_tile.dart` drives render DPI from the transform scale: on zoom-settle re-instantiate the tile (re-laid-out `PdfPageView` at the new pixel size, or `PdfPage.render()` at target DPI) into the **`pdf/page_tile_cache.dart`** store (`LRU<TileKey{hostId, dpiBucket}, ui.Image>` — NOT the ink cache); cache key includes a DPI bucket; **cap retained DPI** (~3× base) to bound memory; downscale stays matrix-only. **`PdfPage.render()` returns an async `PdfImage` owning a native handle → `page_tile_cache` owns its post-frame dispose** (see Open Questions; confirm in the SF4 P0.5 source-pin). **Proven crisp at 4× on the Surface in the P0.5 gate** (§5). Budgeted P0.5/P1, not relocation. |
|
||||
|
||||
### Pre-mortem (DELIBERATE — 3 scenarios)
|
||||
1. **"Continuous double-page janks / pages are blurry at 4× on the user's big scanned PDF."** Cause: built layout modes before profiling two-column windowing, and matrix-scaled a 1× bitmap (R11). *Prevention:* the **P0.5 vertical-slice gate** profiles continuous-SINGLE first with a **rewritten** bench against `ui/editor_screen` (the spike-based `perf_scroll_bench.dart` is deleted — it benchmarks the invalidated Option B) AND proves crisp-on-zoom DPI refresh at 4× on the Surface; continuous-double/paged/spread merge only after P0.5 is GREEN, and the P1 double-page gate reuses the rewritten bench.
|
||||
2. **"Search returns nothing for handwriting."** Cause: over-promised OCR. *Prevention:* F8 ships PDF-text + typed-text search first (reliable), handwriting OCR as additive best-effort with UI copy stating limits; formula explicitly P5.
|
||||
3. **"Insert-blank-page silently rasterized / detached annotations."** Cause: editing source bytes or keying ink to raw PDF index. *Prevention:* logical page-map + ink keyed to `notebook_page_id`; golden test that the source PDF bytes are unchanged after insert and remains selectable-text in an external viewer.
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing strategy
|
||||
|
||||
> **sqlite workaround:** DB-touching tests run via `tool/test.sh` (system sqlite + `LD_LIBRARY_PATH`); pure-logic tests avoid the DB. Reusable assets: `tool/gen_bench_pdf.dart`, `tool/gen_dense_strokes.dart`, `test/assets/large_300p.pdf`, `test/assets/dense_strokes.json`, `integration_test/coordinate_assertion_test.dart`. **NOT reusable: `integration_test/perf_scroll_bench.dart` — it imports `spike_editor_pane.dart` (`pageOverlaysBuilder`, invalidated Option B) and is DELETED + replaced by `integration_test/editor_scroll_bench.dart` (targets `ui/editor_screen`) in P0.5 (MF1).**
|
||||
|
||||
- **Unit (no DB/widgets):**
|
||||
- Coordinate transforms: `NormalizedPageHost`/`BoardHost` round-trip `toContent(toDevice(x))≈x`.
|
||||
- Stroke geometry: `buildStrokeOutline` non-empty for ≥1 point; live ⊆ committed bounds (no "pop").
|
||||
- Eraser: `eraseHits` whole-stroke removal (extraction of live `pen_canvas._eraseAt` — regression port). **`splitStroke` segment-erase is NEW behavior, not a port** (the live canvas only does whole-stroke erase; `undo_manager` records but never computes replacements): mid-erase ⇒ 2 segments, endpoint ⇒ 1 segment, full-erase ⇒ empty, <2-pt result dropped — these exercise greenfield code. (`undo_manager_test.dart` informs the replacement-bookkeeping discipline only.)
|
||||
- Revision gating: `StrokeStore.add` bumps revision; `StaticInkPainter.shouldRepaint` iff revision changed.
|
||||
- `InputArbiter` SM: table-driven over the device×mode matrix incl. palm rejection (touch dropped while stylus active) and `pointerCount>=2`→cancel.
|
||||
- Undo: global commit-time order, host-tagged reversal.
|
||||
- `SaveScheduler`: mutating "current host" after schedule but before write completes does not change persisted snapshot.
|
||||
- **`editor_repository.saveHost` diff-write (MF3, fake DB counting statements):** erase 1 of 2,000 strokes ⇒ exactly **1 DELETE, 0 INSERT**; add 1 stroke ⇒ **1 INSERT, 0 DELETE**; no-op save ⇒ 0 statements. (Guards against the live notes-path delete-all+re-insert anti-pattern at database_service.dart:281–306.)
|
||||
- `EditorStroke`↔`InkStroke` adapter round-trip (SF1): `tilt`/`timestamp`/`pointerDeviceKind` survive both directions (no lossy OCR/export conversion).
|
||||
- `link_graph.backlinksOf` (P3); `search_indexer` snippet/jump-ref mapping for the COMMITTED tier — PDF-text + typed-text (P3).
|
||||
- **Widget:** stylus→stroke committed, touch→not consumed (synthesized pointers); layout-mode switch recenters; reader-mode blocks ink; thumbnail tap jumps; text-box place/move/edit persists (fake DB).
|
||||
- **Perf:** the **rewritten** `integration_test/editor_scroll_bench.dart` (targets `ui/editor_screen` + `layout/page_viewport`; the spike-based `perf_scroll_bench.dart` is deleted, MF1) on `large_300p.pdf` — **continuous-single in P0.5 (gate), continuous-double in P1**, profile mode, N≥120 frames warm, median build+raster ≤16.6ms / p95 ≤22ms; `StaticInkPainter` no-rebuild assertion; Picture-memory ≤64MB sample. Archived in `docs/plans/full-refactor-perf-results.md` with commit hash. Not a hard CI gate (no GPU) but required for milestone sign-off.
|
||||
- **Manual on-device pen checklist (Surface Pen, P0 + each milestone):** pressure varies width; barrel/inverted = erase; palm rest doesn't mark; single-finger scroll; two-finger pinch; **page text/rules stay crisp at 4× zoom — no GPU-upscale blur, DPI refreshes on zoom-settle (R11, P0.5 gate)**; hover pre-warm reduces first-stroke latency; page-flip feels book-like. Recorded with device + commit.
|
||||
- **Regression:** keep `ctc_decoder_test.dart`, `undo_manager_test.dart` green; update `widget_test.dart` to boot `editor_screen`.
|
||||
- **Export golden (P0):** annotate a known page → export → image-compare filled ink matches screen (R7).
|
||||
- **Server (P4):** existing `server/tests/` (`test_sync.py` etc.) green; add notebook/board sync tests.
|
||||
|
||||
---
|
||||
|
||||
## 9. Milestones / sequencing — immediate next chunk (concrete)
|
||||
|
||||
**Next chunk = P0 (engine relocation + persistence), executable now; followed by the P0.5 vertical-slice gate (steps 10–13) before any double/paged/spread layout work:**
|
||||
1. **[ADD]** `lib/editor/engine/stroke_model.dart` — `EditorStroke`/`EditorPoint` (freezed + JSON), normalized; `fromPenStroke`/`toInkStroke` adapters. Run `build_runner`.
|
||||
2. **[ADD]** `lib/editor/engine/stroke_geometry.dart` — lift `buildStrokePath` → `buildStrokeOutline` (verbatim recipe from `ink_painters.dart`).
|
||||
3. **[ADD]** `lib/editor/engine/stroke_store.dart` + `lib/editor/render/{static_ink_painter,live_ink_painter,ink_picture_cache,annotation_layer}.dart` — relocate live painters; add the resolution-independent ink `ui.Picture` cache keyed by `revision` (NO DPI bucket; the DPI-bucketed `page_tile_cache` is a separate P0.5 file, step 10).
|
||||
4. **[ADD]** `lib/editor/input/input_arbiter.dart` + `lib/editor/engine/stroke_eraser.dart` — extract from `pen_canvas.dart` (pure, unit-tested).
|
||||
5. **[ADD]** `lib/editor/persistence/{editor_repository,save_scheduler}.dart` + DB additions (`ink`, `notebook_pages` minimal) in `database_service.dart` (fresh version). `saveHost` implements the **MF3 diff-write contract** (UPSERT changed + DELETE removed, by stroke id; NO delete-all+re-insert) with a per-host last-persisted id-set.
|
||||
6. **[MODIFY]** `lib/editor/canvas/pen_editor_screen.dart` (or new `ui/editor_screen.dart`) to load/commit/save through the repository instead of the in-memory `_strokesByPage` Map.
|
||||
7. **[MODIFY]** `lib/services/pdf_service.dart` `_renderStrokes` → fill `buildStrokeOutline` path (R7) + export golden test.
|
||||
8. **Tests:** arbiter SM, eraser, geometry, revision-gating, save-scheduler snapshot, **diff-write statement-count (MF3: erase 1/2000 ⇒ 1 DELETE 0 INSERT)**, `EditorStroke↔InkStroke` lossless round-trip (SF1), export golden. Run via `tool/test.sh`.
|
||||
9. **Device gate:** build Windows package (CI), confirm pen/palm/pinch on Surface Pen; record in perf-results doc. → unblocks P0.5.
|
||||
|
||||
**Then P0.5 (vertical-slice gate — must pass before P1's double/paged/spread):**
|
||||
10. **[ADD]** `lib/editor/layout/page_viewport.dart` (continuous-single only) + `lib/editor/pdf/{pdf_document_source,page_tile,page_tile_cache}.dart` with zoom-settle DPI refresh (R11) + the **DPI-bucketed `page_tile_cache`** (`LRU<TileKey, ui.Image>`, page bitmaps; owns native-handle dispose). (The resolution-independent `render/ink_picture_cache.dart` for vector ink lands in P0, step 3 — it is a separate cache, no DPI bucket.)
|
||||
11. **[ADD]** `integration_test/editor_scroll_bench.dart` targeting `ui/editor_screen`; **[DELETE]** `integration_test/perf_scroll_bench.dart` + `lib/editor/pdf/spike_*.dart` (the spike pane the old bench imports). Run the rewritten bench on a **scanned-image** 300-page asset (see Open Questions — `large_300p.pdf` is synthetic/vector and may not honestly stress raster re-render at 3× DPI) → continuous-single median ≤16.6ms / p95 ≤22ms; sample both caches' memory.
|
||||
12. **[VERIFY]** source-pin pdfrx `PdfPage.loadText`/`charRects`/`render()`/`PdfPageView` in 2.4.4 (SF4) + smoke test; **confirm `PdfPage.render()`'s `PdfImage`/native-handle ownership + dispose semantics** so `page_tile_cache` can manage post-frame disposal; record signatures in perf-results doc.
|
||||
13. **Device gate:** crisp-on-zoom at 4× PASS on the Surface (R11). → unblocks P1.
|
||||
|
||||
Each subsequent milestone (P1…P5) follows §5 exit criteria; verifier/critic + perf-results update per milestone.
|
||||
|
||||
---
|
||||
|
||||
## 10. RALPLAN-DR
|
||||
|
||||
### Principles (3–5)
|
||||
1. **Single source of truth = host content coordinates.** Screen mapping is a paint-time `canvas` transform; never store/duplicate transformed geometry. (Already the live convention — generalize it.)
|
||||
2. **One host-agnostic ink engine.** PDF page, infinite board, and (P5) CAS overlay are `CoordinateSpaceHost`s behind one renderer — never fork the stroke pipeline.
|
||||
3. **Own the gesture pipeline; pdfrx only renders.** A single `Listener` over a shared `InteractiveViewer` arbitrates draw/pan/zoom/palm by pointer kind + count — no gesture-arena fights (proven live).
|
||||
4. **De-risk performance AND crispness before features.** A **P0.5 vertical-slice gate** (continuous-single, rewritten bench, crisp-at-4× on the Surface) precedes all double/paged/spread work; continuous-double is gated again in P1.
|
||||
5. **Ship every phase; generalize under load.** Relocation-first refactor keeps the editor working at all times; old screens deleted only after parity.
|
||||
|
||||
### Decision Drivers (top 3)
|
||||
1. **D1 — Pen feel + palm rejection + pinch on Windows Surface Pen** is the make-or-break primary-device requirement (own-canvas model already targets it).
|
||||
2. **D2 — 60fps across all layout modes** on big PDFs with thousands of strokes.
|
||||
3. **D3 — One engine reused across PDF page, infinite board, and CAS**, forward-compatible with search/双链/sync.
|
||||
|
||||
### Viable options (≥2) with bounded pros/cons
|
||||
|
||||
**Option A — Own-canvas engine (single `Listener` + `InteractiveViewer`, pdfrx as renderer). CHOSEN (already live).**
|
||||
- Pros: D1 solved structurally (one transform, no arena fight — already working in `pen_canvas.dart`); D3 trivial (hosts share the transform); minimal new deps; matches the proven Saber model the user cited.
|
||||
- Cons: we own page layout/windowing/tiling **AND multi-resolution re-rasterization** (more code than a stock viewer); double-page perf is on us (R1) and crisp-on-zoom is on us (R11 — pdfrx's own viewer re-renders crisp tiles per zoom for free; we must re-derive tile DPI from the transform); thumbnails/text-extraction still need pdfrx page APIs (R9).
|
||||
|
||||
**Option B — pdfrx `pageOverlaysBuilder`-hosted ink + RenderProxyBox arena-bypass (the phase-1 plan).**
|
||||
- Pros: pdfrx gives continuous scroll/tiling/text-extraction for free; ink-follows-page is structural via page overlays.
|
||||
- Cons: requires a custom `PenCaptureBinding` arena bypass that fights pdfrx's greedy scale recognizer (the live code already **abandoned** this — `PenCaptureRegion` is unused); pdfrx owns the transform so double-page/board reuse is awkward; D1 proven harder than Option A in practice. **Invalidated** — see below.
|
||||
|
||||
**Option C — Flutter shell + Rust hot-path (rnote-style) for ink/render.**
|
||||
- Pros: maximal ink perf headroom.
|
||||
- Cons: rejected in memory (`badnote-flutter-344-windows-pen`) — Windows pen is weaker in the Rust/GTK stack; huge FFI surface; contradicts "stay Flutter." **Invalidated.**
|
||||
|
||||
### Invalidation rationale
|
||||
- **B invalidated (as the INPUT model):** the live codebase already moved off it; `pageOverlaysBuilder`+arena-bypass made stylus/touch arbitration fight pdfrx's recognizer, whereas Option A's single-`Listener`-over-shared-transform sidesteps the arena entirely and is already drawing with pressure/pinch/palm. **Correction (MF2): B had TWO edges, not one** — (1) free continuous scroll/windowing, AND (2) free **per-zoom crisp re-rasterization** (pdfrx re-renders tiles at each zoom level). Edge (1) is recoverable in A via windowed hosting (needed for double-page anyway); edge (2) is **NOT free in A** — Option A must own multi-resolution tiling (R11), which is the genuinely new cost of this choice. We accept that cost (gated in P0.5) because A's structural D1/D3 wins outweigh it. B retained only as a fallback page-**render** strategy, not the input model.
|
||||
- **C invalidated:** documented Windows-pen regression in the Rust/GTK path + "stay Flutter" hard constraint; the 3.44 WM_POINTER fix already unblocked Flutter pen, removing C's motivation.
|
||||
- Net: **A chosen; B retained as a partial fallback (page rendering only); C rejected.**
|
||||
|
||||
### ADR
|
||||
- **Status:** Architect APPROVE-WITH-MUST-FIX applied (2026-06-21): MF1 (rewrite perf bench off the invalidated spike), MF2 (R11 zoom re-rasterization + multi-resolution tiling), MF3 (diff-write contract); P0.5 vertical-slice synthesis gate; SF1–SF5. Pending Critic. Records the live pivot from the superseded phase-1 input architecture.
|
||||
- **Decision:** Build the full BadNote vision on an **own-canvas, host-agnostic ink engine** (single `Listener` + shared `InteractiveViewer`; pdfrx as page renderer/text source). Generalize the live `lib/editor/canvas/` into `engine/render/input/layout/...`; persist a single canonical `EditorStroke`; phase features P0→P5 with a **P0.5 vertical-slice gate**.
|
||||
- **Drivers:** D1 Surface-Pen feel, D2 60fps multi-mode, D3 one reusable engine.
|
||||
- **Alternatives considered:** B (pdfrx-overlay + arena bypass — invalidated as input model, kept as render fallback; had two free edges — scroll AND per-zoom crispness — the latter is the new cost we take on), C (Flutter+Rust — rejected).
|
||||
- **Why chosen:** A is already proven live for pen/palm/pinch and gives D1+D3 structurally; the remaining risks (D2 multi-mode perf AND R11 crisp-on-zoom) are gated up front in **P0.5** then re-gated for double-page in P1.
|
||||
- **Consequences:** we own layout/windowing/tiling **AND multi-resolution page-tile re-rasterization** (R11 — not just layout, MF2); `PenCaptureRegion`/`PenCaptureBinding` retired (dead under own-canvas); the spike-based perf bench is deleted and rewritten against `ui/editor_screen` (MF1); per-stroke-row persistence is valid ONLY under the diff-write contract (MF3); `EditorPoint` is a non-lossy superset of `InkPoint` (SF1); export must fill (not stroke) ink; handwriting OCR is additive/non-blocking and formula is P5 (SF2); `split_view_screen` deletion deferred to P3 (SF3); server stays optional.
|
||||
- **Follow-ups:** P0 device gate confirms pen model; **P0.5 gate** proves continuous-single perf + crisp-at-4× + source-pinned pdfrx APIs (SF4) before any double/paged/spread; resolve open questions below.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions (persist to `.omc/plans/open-questions.md`)
|
||||
- [ ] Continuous-double-page on a real 300-page scanned PDF — does windowed two-column hosting hold 60fps, or do we need tile pre-rasterization? — *P0.5 gates continuous-single first; P1 perf gate decides double; affects R1/R10.*
|
||||
- [ ] **How is page-tile DPI refreshed on zoom under the shared transform (SF5)?** — re-laid-out `PdfPageView` at the new pixel size, or `PdfPage.render()` at target DPI into `page_tile_cache`? What scale-change threshold + debounce triggers a refresh, and what is the retained-DPI cap? — *Load-bearing for the 60fps + crisp-at-4× goal; resolved in P0.5 (R11).*
|
||||
- [ ] **`PdfPage.render()` native-handle lifecycle (Critic 4a):** `render()` returns an async `PdfImage` owning a native handle backing a `ui.Image` — `page_tile_cache` must own its dispose (post-frame, after the raster thread is done) on eviction. *Confirm exact ownership + dispose API in the SF4 P0.5 source-pin; affects page_tile_cache + R10/R11.*
|
||||
- [ ] **Honest R11 raster stress asset (Critic 4b):** does the synthetic/vector `large_300p.pdf` actually force RASTER re-render cost at 3× DPI, or does a **scanned-image** asset better exercise the crispness + tile-memory gate? — *Add a scanned 300-page asset (`tool/gen_bench_pdf.dart` image mode or a real scan) for the P0.5/P1 perf+crispness gates; affects §8 / R11 honesty.*
|
||||
- [ ] Stroke-model convergence: fold `notes`/`InkStroke` ink-note path into `boards`/`EditorStroke`, or keep notes separate long-term? — *Affects §3/§4 churn.*
|
||||
- [ ] Handwriting OCR: is the existing PP-OCRv4 text-line model usable on cursive/handwriting at all, or do we need a handwriting-specific model even for non-formula text? — *Affects F8 expectation-setting / R3.*
|
||||
- [ ] Portable bundle relink: match source PDF by content-hash only, or also store original path + size? — *Affects F6.*
|
||||
- [ ] Server AI-refine: run llm_wiki/VLM server-side only, or allow a local LLM path for offline users? — *Affects F9 scope / R5.*
|
||||
- [ ] Keep `pptx_service`/`ppt_annotator_screen` in the new engine, or freeze PPT support? — *Out of the 11 features; decide before P1 nav swap.*
|
||||
Reference in New Issue
Block a user