OCR: embedded, cross-platform ONNX backend with pluggable fallback
Some checks failed
CI / Flutter (analyze, test, Windows build) (push) Failing after 30s
CI / Server tests (optional) (push) Failing after 29s

Make on-device OCR a pluggable local service so it runs locally on every
platform (not just Windows), aimed at GoodNotes/Notability-class handwriting on
low-power hardware (e.g. Zen2 APU, CPU/iGPU).

- New OcrBackend abstraction (lib/services/ocr/): selector prefers an embedded
  ONNX recognition backend, falling back to the OS-native backend (Windows
  WinRT), and to a clean no-op when neither is available.
- OnnxRecognitionBackend: flutter_onnxruntime session from a bundled asset,
  dart:ui preprocessing (resize to 48px, CHW float32, normalized), pure-Dart CTC
  greedy decode. Fully guarded — absent model/dict is a no-op; never throws.
- ocr_engine.dart kept as a thin facade (recognizeImage) delegating to the
  selector, so ocr_service.dart is unchanged.
- CtcDecoder unit-tested (6 tests). flutter analyze clean; all tests pass.
- Model is not committed; tool/fetch_ocr_model.sh + assets/models/ocr/README.md
  document fetching PP-OCRv4 rec + dict on the dev machine.
- CI: forward HTTPS_PROXY to the Windows build so CMake can fetch the ONNX
  Runtime native lib behind the GFW; README documents the system-install
  alternative. PP-OCR geometry/blank assumptions documented for on-device tuning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-21 03:51:54 +08:00
parent 25ba717c97
commit 99b98b96b0
19 changed files with 684 additions and 18 deletions

View File

@@ -9,7 +9,9 @@ All notes, documents, search, and OCR run on your device. No server is required
- Ink notes with Surface Pen (pressure, stabilizer, undo/redo)
- PDF and PPT import with page-level annotation
- Full-text search over note titles, typed text, and OCR results
- **Local OCR** — handwriting recognition via Windows built-in OCR (Windows desktop)
- **Local OCR** — pluggable, fully on-device. An embedded ONNX recognition
backend (cross-platform, CPU/iGPU) with a graceful fallback to the platform's
built-in OCR (Windows). See [Local OCR](#local-ocr).
## Build (Windows)
@@ -60,7 +62,13 @@ lib/
│ ├── database_service.dart # SQLite + FTS5
│ ├── ocr_service.dart # Local OCR orchestration
│ ├── stroke_rasterizer.dart # Ink → PNG for OCR
── ocr_engine.dart # Platform OCR bridge
── ocr_engine.dart # OCR entry point (delegates to a backend)
│ └── ocr/ # Pluggable OCR backends
│ ├── ocr_backend.dart # Backend interface
│ ├── ocr_backends.dart # Backend selector (ONNX → native)
│ ├── onnx_recognition_backend.dart # Embedded ONNX (cross-platform)
│ ├── native_ocr_backend.dart # OS OCR (Windows WinRT)
│ └── ctc_decoder.dart # Pure-Dart CTC greedy decode
├── providers/ # Riverpod state
└── widgets/ # Ink canvas, toolbars, thumbnails
```
@@ -69,9 +77,43 @@ OCR flow on save:
1. Extract typed text from text-tool strokes
2. Rasterize handwriting strokes to PNG
3. Run Windows OCR on the PNG
3. Recognize via the active local OCR backend (embedded ONNX if a model is
bundled, otherwise the platform's native OCR)
4. Merge recognized text into the local FTS index for search
## Local OCR
OCR runs entirely on-device through a pluggable backend (`lib/services/ocr/`).
`OcrBackends` selects, in order:
1. **`OnnxRecognitionBackend`** — embedded, cross-platform recognition via
`flutter_onnxruntime` (CPU/iGPU; suited to low-power APUs). Active only when
an ONNX model is bundled.
2. **`NativeOcrBackend`** — the OS built-in OCR (Windows WinRT today).
If no backend is available, OCR is a clean no-op — the app still works.
### Enabling the embedded ONNX model
The model is **not committed** (it is large). Fetch it onto your dev machine
before building so it bundles as an asset:
```bash
tool/fetch_ocr_model.sh # downloads PP-OCRv4 rec ONNX + ppocr_keys_v1.txt
# into assets/models/ocr/ (proxy hint inside)
```
See [assets/models/ocr/README.md](assets/models/ocr/README.md). The recognition
geometry / CTC-blank assumptions (PP-OCRv4 mobile rec, 3×48×W, blank=0) are
documented in `onnx_recognition_backend.dart` and should be verified on-device
against your exact exported model.
> **Windows build note:** the `flutter_onnxruntime` plugin downloads the ONNX
> Runtime native library (v1.22.0) from GitHub at build time. Behind a firewall,
> set `HTTPS_PROXY` for the build (CMake honours it), or install ONNX Runtime
> system-wide and build with `-DUSE_SYSTEM_ONNXRUNTIME=ON
> -DONNXRUNTIME_ROOT_DIR=<path>`.
## Optional server
The `server/` directory contains an experimental FastAPI backend (sync + EasyOCR). It is **not required** for the desktop app and is kept separately for future multi-device sync experiments. See [server/README.md](server/README.md).