BadNote/docs/plans/2026-06-24-file-based-storage.md

# BadNote — File-Based Storage Re-Architecture (Obsidian-style vault)

**Status:** DESIGN (not yet implemented)
**Date:** 2026-06-24
**Owner plan file:** `docs/plans/2026-06-24-file-based-storage.md`
**Mode:** DELIBERATE (touches persistence of a working pen editor — must not lose ink)

> Grounding note: written after reading the actual current sources —
> `lib/services/database_service.dart`, `lib/editor/persistence/editor_repository.dart`,
> `lib/editor/persistence/save_scheduler.dart`, `lib/editor/canvas/pen_editor_screen.dart`,
> `lib/screens/split_view_screen.dart`, `lib/screens/home_screen.dart`,
> `lib/services/pdf_service.dart`, `lib/services/pptx_service.dart`,
> `lib/providers/document_provider.dart`, `lib/providers/settings_provider.dart`,
> `lib/main.dart`, and the stroke models
> (`lib/editor/engine/stroke_model.dart`, `lib/models/ink_stroke.dart`,
> `lib/models/ink_point.dart`, `lib/editor/canvas/pen_stroke.dart`,
> `lib/models/scratch_link.dart`, `lib/models/bookmark.dart`,
> `lib/models/document.dart`). `file_picker` `getDirectoryPath` (Windows desktop
> support) confirmed via Context7 against the installed `file_picker: ^8.0.0`.

---

## 0. The user's intent (verbatim)

1. **"Import file" is a TOP-LEVEL home-screen action**, sibling of "Create
   notebook", accepting **multiple file types** (docx, pptx, pdf, …).
2. **Annotations travel WITH the file** ("跟着文件走") — stored as JSON
   **sidecar files** next to the source file, **NOT in SQLite**. Sync follows the
   file.
3. After importing a PDF, the file is placed inside a **notebook folder**, then
   synced to a **server**. User believes this is **more robust than SQLite**.
4. On first launch, the user **picks a notebook ROOT folder** (an Obsidian-style
   vault).
5. **Robust auto-save.**

---

## 1. What is stored in SQLite TODAY (inventory)

Read from `lib/services/database_service.dart` (schema version **8**). This is the
data that must move to files or be re-homed.

| Table | Written/read by | Holds | Coordinate format | Disposition |
|---|---|---|---|---|
| `documents` | `DocumentListNotifier` (`document_provider.dart`), `DatabaseService.insertDocument/getAllDocuments/deleteDocument` | id (uuid), filename, doc_type (`pdf`/`pptx`), **absolute** file_path, page_count, rotation, timestamps | — | → derived from vault scan; becomes a cache/index only |
| `ink` | `EditorRepository` (`editor_repository.dart`), `SaveScheduler` | committed PDF-editor strokes; one row/stroke; `host_id = "doc:<documentId>:page:<pageIndex>"`, `stroke_json` = `EditorStroke.toJson()` | normalized [0,1], width = fraction of page width | → **sidecar `strokes[pageIndex]`** |
| `scratch_links` | `PenEditorScreen._placeScratchLink`, `DatabaseService.saveScratchLink/loadScratchLinks` | anchor id, document_id, page_index, nx, ny | normalized [0,1] | → **sidecar `scratchLinks[]`** |
| `scratchpads` | `SplitViewScreen` (keyed by **anchor id**), `DatabaseService.saveScratchpad/loadScratchpad` | one row per anchor id; `strokes_json` = list of `InkStroke.toJson()` | **absolute world pixels** (infinite canvas) | → **sidecar `scratchLinks[].scratchpad.strokes[]`** |
| `bookmarks` | `DatabaseService.insertBookmark/getBookmarks` | id, document_id, page_number, label, color | page index | → **sidecar `bookmarks[]`** |
| `annotations` | `DatabaseService.saveAnnotations/getAnnotations` | legacy per-page `annotation_json` blob | per-page | **DEAD in new editor** — migrate if present, else ignore |
| `notes` + `strokes` | `note_provider.dart` via `insertNote/updateNote`; `Note` model | free ink notebooks (not file-backed); strokes are `InkStroke` rows | normalized | → **standalone notebook sidecars** (see §A.4) |
| `board_cards` | `DatabaseService.saveBoardCards/loadBoard` (F7 sticky board) | board cards geometry+text | absolute | → **per-notebook `board.json`** (or keep in SQLite short-term, §B) |
| `document_fts`, `notes_fts`, `ocr_results`, `notebook_pages` | FTS + OCR | search index, OCR text | — | **stays in SQLite as a rebuildable cache** (never the source of truth) |

**Key existing facts the new format must preserve (cite):**

- `EditorStroke.toJson()` (freezed/json_serializable, `stroke_model.dart`) emits:
  `{ "id", "points":[{ "x","y","pressure","tilt","timestamp","pointerDeviceKind" }], "tool":"pen"|"highlighter"|"eraser", "color":<int ARGB>, "width":<double>, "filled":<bool>, "textContent":<String?>, "fontSize":<double> }`.
  **`brush` is deliberately NOT serialized** (`@JsonKey(includeFromJson:false,includeToJson:false)` in `stroke_model.dart`), so loaded strokes default to `BrushKind.fountainPen`. The sidecar format inherits this limitation (see §A.5 "brush TODO").
- `InkStroke.toJson()` (`ink_stroke.dart`) emits the same point shape plus
  `createdAt` and `strokeWidth` (note: `strokeWidth`, not `width`). This is the
  format already persisted for scratchpads and free notes.
- `ScratchLink.toJson()` (`scratch_link.dart`, hand-written): `{id, documentId, pageIndex, nx, ny}`.
- The editor derives its **documentId from the file path** via a djb2 hash
  (`_documentIdFromPath` in `pen_editor_screen.dart`). Once strokes live in a
  sidecar next to the file, **the path-hash document id becomes irrelevant** —
  the sidecar IS the identity. This removes a class of bugs (moving a file
  orphaned its SQLite rows).

---

## 2. Decision summary (what we are building)

- A **vault** = a user-picked root folder, path stored in `SharedPreferences`.
- Each imported document becomes a **notebook folder** inside the vault:
  `<vault>/<notebook-name>/` containing the **source file** + one **sidecar**
  `<file>.badnote.json` holding all annotations for that file.
- **The sidecar is the source of truth.** SQLite is demoted to a
  **rebuildable index/cache** (FTS + thumbnails + recent list). Recommendation:
  **keep SQLite, but only as cache** (§B explains why dropping it entirely is
  more work than it's worth right now).
- **Auto-save** writes the sidecar atomically (temp + rename), debounced, with
  flush on pause/close.
- **Sync** = file-level sync of the whole vault folder to a server; sync unit =
  notebook folder; conflict policy = last-write-wins per file + `.conflict` copy.

---

## A. On-disk layout + JSON schema

### A.1 Vault layout

```
<vault root>/                         ← user-picked, persisted in SharedPreferences
├─ .badnote/                          ← vault-level app metadata (hidden)
│  ├─ vault.json                      ← { "schemaVersion": 1, "vaultId": "<uuid>", "createdAt": ... }
│  └─ index.sqlite                    ← OPTIONAL local cache/index (FTS, thumbnails). Rebuildable. NOT synced.
├─ Calculus Lecture 3/               ← a notebook folder (one per imported doc)
│  ├─ Calculus Lecture 3.pdf          ← the source file (pdf/docx/pptx/…)
│  ├─ Calculus Lecture 3.pdf.badnote.json   ← the sidecar (annotations travel with the file)
│  └─ .badnote-assets/                ← optional: rendered page PNGs for pptx/docx, thumbnails
│     ├─ slide_1.png …
│     └─ thumb_0.png …
├─ My freehand notes/                ← a NON-file-backed notebook (free ink, no source doc)
│  └─ notebook.badnote.json           ← strokes-only notebook (replaces SQLite notes/strokes)
└─ …
```

Rules:
- **Notebook folder name** = sanitized source filename (basename without
  extension), de-duplicated with a numeric suffix on collision.
- **Sidecar name** = `<source filename incl. ext>.badnote.json`. Keeping the full
  source extension in the sidecar name means a folder with both `slides.pdf` and
  `slides.pptx` never collides.
- **Sidecar lives in the same folder as its file** → moving/copying/syncing the
  folder moves the annotations with it ("跟着文件走"). This is the whole point.
- `.badnote/index.sqlite` is **per-vault** and **excluded from sync** (it is a
  cache; each device rebuilds its own). It replaces today's
  `getApplicationDocumentsDirectory()/badnote.db`.

### A.2 Sidecar JSON schema (file-backed document)

`<file>.badnote.json` (UTF-8, pretty-printed for diff-friendliness / git sync):

```json
{
  "badnoteSidecarVersion": 1,
  "sourceFile": "Calculus Lecture 3.pdf",
  "docType": "pdf",
  "pageCount": 42,
  "rotation": 0,
  "createdAt": "2026-06-24T10:00:00.000Z",
  "updatedAt": "2026-06-24T10:32:11.500Z",

  "strokes": {
    "0": [ <EditorStroke.toJson()>, <EditorStroke.toJson()>, … ],
    "3": [ <EditorStroke.toJson()>, … ]
  },

  "highlights": {
    "0": [ { "l": 0.12, "t": 0.20, "r": 0.88, "b": 0.235, "color": 1714657595 } ]
  },

  "bookmarks": [
    { "id": "<uuid>", "pageNumber": 5, "label": "Proof", "color": 4283215696, "createdAt": "…" }
  ],

  "scratchLinks": [
    {
      "id": "<uuid>",
      "pageIndex": 7,
      "nx": 0.83, "ny": 0.41,
      "createdAt": "…",
      "scratchpad": {
        "canvasWidth": 4000,
        "canvasHeight": 4000,
        "strokes": [ <InkStroke.toJson()>, … ]   // absolute world pixels (unchanged format)
      }
    }
  ]
}
```

Schema notes, tied to existing code:

- **`strokes`** is a map keyed by **string page index** → list of
  `EditorStroke.toJson()` objects. This is byte-for-byte the JSON already written
  to the `ink` table's `stroke_json` column by `EditorRepository.saveHost`
  (`jsonEncode(stroke.toJson())`). Loading just calls `EditorStroke.fromJson`.
  - Drop the `host_id = "doc:<id>:page:<i>"` convention entirely; the sidecar key
    IS the page index. `EditorRepository.pageHostId` /
    `_pageIndexFromHostId` (in `pen_editor_screen.dart`) become obsolete for the
    file path.
- **`highlights`** persists what is TODAY in-memory only — see
  `_highlightsByPage` and `TODO(persist-highlights)` in `pen_editor_screen.dart`.
  Each rect stored normalized [0,1] (`l/t/r/b`) exactly as `_highlightSelection`
  computes it. This closes that TODO as a side benefit.
- **`bookmarks`** mirrors `Bookmark.toJson()` (`bookmark.dart`); `documentId`
  field is dropped (the sidecar already scopes it).
- **`scratchLinks[]`** merges today's TWO tables: `scratch_links` (anchor
  geometry) + `scratchpads` (the anchor's private ink, currently keyed by anchor
  id). Each anchor now **embeds** its scratchpad. `scratchpad.strokes` keep the
  `InkStroke.toJson()` format in **absolute world pixels** — unchanged from
  `SplitViewScreen._saveImmediate`, so the infinite-canvas logic
  (`_checkCanvasExpansion`, `penStrokesFromInk`) needs no change. We persist
  `canvasWidth/Height` so the world size restores (today it always resets to
  4000×4000).

### A.3 Why normalized vs absolute coords are preserved verbatim

- PDF-editor strokes (`EditorStroke`) are normalized to the page rect; width is a
  fraction of page width (`stroke_model.dart` header comment). Sidecar stores them
  unchanged → no re-projection, no rounding drift, export
  (`PdfService.exportAnnotatedPdf`) keeps working untouched.
- Scratchpad strokes (`InkStroke`) are absolute world pixels (infinite canvas).
  Sidecar stores them unchanged.
- This is a **format-preserving** migration: same `toJson`/`fromJson`, different
  container (file vs row). That is what makes it low-risk.

### A.4 Standalone (non-file) notebooks — the "Create notebook" path

Free-ink notes today live in `notes` + `strokes` (`Note` model, `note_provider.dart`).
In the vault they become a notebook folder with **no source file**, holding a
`notebook.badnote.json`:

```json
{
  "badnoteSidecarVersion": 1,
  "docType": "notebook",
  "title": "My freehand notes",
  "tags": ["math"],
  "createdAt": "…", "updatedAt": "…",
  "pages": [
    { "ordinal": 0, "strokes": [ <InkStroke.toJson()>, … ] }
  ]
}
```

(`InkStroke.toJson()` is the exact format `Note.strokes` already serialize to.)

### A.5 Brush persistence caveat (honest limitation)

`EditorStroke.brush` and `PenStroke.brush` are **not serialized** today
(`@JsonKey(includeFromJson:false,includeToJson:false)`, see `stroke_model.dart`
and `TODO(brush-persist)` in `pen_editor_screen.dart`). The sidecar inherits this:
a reloaded pen stroke renders as `fountainPen`; a highlighter is recovered from
`tool == highlighter`. **Recommendation:** add an optional `"brush"` field to the
sidecar `EditorStroke` JSON in a later increment by flipping the `@JsonKey` — the
sidecar schema is forward-compatible (unknown fields ignored on read), so this is
non-breaking. Not required for this storage migration.

---

## B. Migration: SQLite → sidecars (no data loss)

**Recommendation: KEEP SQLite, demote it to a rebuildable cache. Do NOT drop it.**

Reasons:
- FTS5 (`document_fts`, `notes_fts`) and OCR (`ocr_results`) are non-trivial and
  query-shaped; re-implementing search over flat JSON files is a separate project.
  Keep them in `.badnote/index.sqlite`, rebuilt by scanning sidecars.
- The home screen's "recent documents" list (`getAllDocuments`) wants fast sorted
  access; a cache table is the pragmatic backing for it (the source of truth is
  still the vault scan).
- Dropping SQLite forces rewriting `note_provider`, `document_provider`,
  `search_provider`, OCR, and the board in one shot — high blast radius. Demoting
  is incremental and reversible.

### B.1 One-time migration on first launch after upgrade

Guarded by a `SharedPreferences` flag `vaultMigrationDone` (and only runs once a
vault root exists — see §C). Algorithm:

1. Open the legacy DB at `getApplicationDocumentsDirectory()/badnote.db` (the path
   `DatabaseService._initialize` uses today). If absent → nothing to migrate.
2. For each row in `documents`:
   a. Resolve the legacy `file_path` (absolute). If the file still exists, **copy**
      it into a new notebook folder `<vault>/<sanitized filename>/`.
   b. Build the sidecar:
      - `strokes`: query `ink WHERE host_id LIKE 'doc:<documentId>:page:%'`
        (the `EditorRepository.loadDocument` query), group by page index parsed
        from host_id, write each `stroke_json` straight through (it's already
        `EditorStroke` JSON — no re-encode).
      - `bookmarks`: `getBookmarks(documentId)`.
      - `scratchLinks`: `loadScratchLinks(documentId)`; for each, `loadScratchpad(anchorId)`
        → embed as `scratchpad.strokes` (re-encode via `InkStroke.toJson`).
      - `annotations` (legacy per-page blob): if any rows exist, attempt to decode
        and fold into `strokes`; if format is unrecognized, copy the raw blob into
        a `legacyAnnotations` field so nothing is silently dropped.
   c. Write the sidecar **atomically** (§F).
3. For each `notes` row → write a standalone notebook sidecar (§A.4) under a
   notebook folder.
4. For `board_cards`: SHORT TERM leave them in SQLite (board is self-contained and
   not part of the "files" intent). LATER, write a `board.json` per notebook.
5. **Do not delete the legacy DB.** Rename it to `badnote.db.premigration` as a
   safety net. Set `vaultMigrationDone = true`.
6. Rebuild `.badnote/index.sqlite` (FTS + recent list) by scanning the new vault.

### B.2 Crash safety of migration

- Process sidecars one notebook at a time; each sidecar write is atomic.
- The migration is **idempotent**: re-running skips notebook folders whose sidecar
  already exists and validates. If it dies halfway, relaunch resumes.
- Because the legacy DB is preserved until the flag flips, a failed migration
  loses nothing.

---

## C. Folder picker on init (the vault prompt)

Use **`file_picker`** — already a dependency (`file_picker: ^8.0.0` in
`pubspec.yaml`, already used by `PdfService.pickPdfFile` /
`PptxService.openPptxFile`). Its **`FilePicker.platform.getDirectoryPath()`** is
**Desktop/Windows supported** (confirmed via Context7). **No new package needed.**
Do NOT add `file_selector` — `file_picker` already covers both file and directory
picking and is wired in.

### C.1 Persistence + flow

- New `VaultService` (singleton, like `DatabaseService`):
  - `Future<String?> getVaultRoot()` — reads `SharedPreferences` key `vaultRoot`.
  - `Future<void> setVaultRoot(String path)` — writes it.
  - `Future<bool> vaultRootValid()` — true iff the stored path exists and is a
    writable directory.
- `main.dart` change: after `SharedPreferences.getInstance()`, check
  `vaultRootValid()`.
  - If **valid** → go to `HomeScreen` as today.
  - If **missing/invalid** → show a `VaultSetupScreen` (a gate before
    `HomeScreen`) that explains "Pick a folder to store your notebooks (like an
    Obsidian vault)" and calls `getDirectoryPath(dialogTitle: 'Choose your BadNote vault', lockParentWindow: true)`.
    On selection: create `<root>/.badnote/vault.json`, persist the path, then
    enter `HomeScreen`.
- **Re-prompt if missing:** if the stored path later disappears (external drive
  unplugged, folder deleted), `vaultRootValid()` returns false → the gate shows
  again with a "your vault folder is missing — relocate or pick a new one"
  message. Never silently fall back to app-documents (that would scatter data).
- A **"Change vault"** entry in `SettingsScreen` re-runs the picker.

### C.2 Windows specifics

- Pass `lockParentWindow: true` so the native dialog is modal (Context7 note).
- Wrap in try/catch (Context7 shows `getDirectoryPath` can throw on Windows for
  permission/system issues) and surface a retry.
- Validate the chosen folder is writable by writing+deleting a probe file before
  committing it as the vault.

---

## D. Multi-format import (docx / pptx / pdf)

This is the genuinely hard part. Be honest about it.

### D.1 The home-screen entry (requirement #1)

Replace today's two separate `IconButton`s (`_importPdf`, `_importPptx` in
`home_screen.dart`) and the empty-state buttons with **one top-level "Import
file" action**, a sibling of "Create notebook" (the FAB / `_createAndOpenNote`).
A single `FilePicker.pickFiles(type: FileType.custom, allowedExtensions: ['pdf','docx','pptx','ppt'])`
call; route on extension. Both actions sit at the same visual level (e.g. two
primary buttons in the empty state, and two entries in the app bar / a small
"+ New" menu with "Create notebook" and "Import file").

### D.2 Per-format strategy

| Format | Annotate how | Mechanism | Windows-viable? |
|---|---|---|---|
| **PDF** | Directly, as today | `pdfrx` `PdfViewer` + normalized ink overlay (`PenEditorScreen`, unchanged) | YES — already shipping |
| **PPTX/PPT** | **Convert to images, annotate as slides** | `PptxService.convertToImages` (LibreOffice headless → PNG, fallback placeholders) + `PenSlideScreen` | PARTIAL — needs LibreOffice |
| **DOCX** | **Convert to PDF, then annotate as PDF** (recommended) | LibreOffice headless `--convert-to pdf`, then the PDF path flows into `PenEditorScreen` | PARTIAL — needs LibreOffice |

### D.3 The realistic recommendation

- **PDF:** unchanged. Place the picked file into a notebook folder, open
  `PenEditorScreen` on the **vault copy** (not the original picked path).
- **DOCX → PDF (convert-on-import):** the cleanest path is to **convert DOCX to a
  PDF once, at import time**, store the **PDF** as the notebook's annotatable
  artifact (keep the original `.docx` alongside it for fidelity/round-trip). Then
  everything downstream is the existing, working PDF pipeline. This is far simpler
  than rendering Word layout natively in Flutter (there is no good pure-Dart DOCX
  renderer).
- **PPTX:** keep the existing slide-image path (`PptxService` + `PenSlideScreen`),
  but **cache the rendered PNGs into the notebook's `.badnote-assets/`** instead of
  a temp dir (today `convertToImages` writes to `getTemporaryDirectory()`, so
  slides re-render every open — see `home_screen.dart` `_openDocument`). Caching
  also makes the notebook self-contained for sync.

### D.4 The hard truth about conversion (call it out)

- Both DOCX→PDF and PPTX→PNG currently depend on **LibreOffice headless** being on
  `PATH` (`PptxService._convertViaLibreOffice` runs `which libreoffice` then
  `libreoffice --headless --convert-to …`). On a **Windows tablet, LibreOffice is
  usually NOT installed**, and `which` is a POSIX tool that won't resolve `soffice.exe`.
  **This path will silently fail today on Windows** and fall back to placeholder
  slides.
- **Proposed fallbacks, in order:**
  1. **Detect LibreOffice/`soffice.exe`** at the standard Windows install paths
     (`C:\Program Files\LibreOffice\program\soffice.exe`) in addition to `PATH`;
     invoke `soffice` (not `libreoffice`) on Windows. Fix the `which` assumption.
  2. If absent, **prompt the user** ("Install LibreOffice to import Word/PowerPoint,
     or convert to PDF first"), and offer a **"locate soffice.exe" picker** that we
     persist in SharedPreferences.
  3. **Bundle/ship nothing heavy.** Do not attempt to embed a converter. For a
     single-user tablet app, requiring LibreOffice (or pre-export to PDF) is an
     acceptable, honest constraint.
- **Minimal first cut:** ship **PDF import end-to-end on the new vault**, plus the
  one-file "Import file" entry that *accepts* docx/pptx, but for docx/pptx route
  through the existing (LibreOffice-dependent) converters with the Windows
  `soffice.exe` fix. Treat full docx/pptx fidelity as a known limitation, not a
  blocker for the storage re-architecture.

---

## E. Server sync (file-level, single user)

Keep it pragmatic. The data is now plain files in one folder, which is exactly what
makes simple sync viable.

### E.1 Sync unit & approach

- **Sync unit = the notebook folder** (source file + sidecar + assets). A notebook
  is self-contained, so syncing the folder syncs the annotations with it.
- **Exclude** `.badnote/index.sqlite` and `.badnote-assets/` from sync if desired
  (assets are re-derivable; the index is per-device). Source file + sidecar are the
  must-sync pair.
- **Recommended concrete approach (single-user): WebDAV** to a self-hosted/Nextcloud
  endpoint, OR a **simple REST blob sync** if the user controls the server.
  - WebDAV is the lowest-effort robust option (PUT/GET/PROPFIND, mtime-based),
    works against Nextcloud/ownCloud/rclone-serve, and there are Dart HTTP clients.
  - If the user already uses Nextcloud/Dropbox/OneDrive **and** the vault folder
    lives inside that synced folder, BadNote needs **zero sync code** — the OS sync
    client handles it. This is the cheapest path and worth recommending as option 0.
  - **git** is possible (text JSON diffs nicely) but binary PDFs bloat history and
    conflict UX is poor for a tablet — not recommended as the default.

### E.2 Conflict policy

- **Last-write-wins per file**, using a manifest of `{ relativePath, sha256, mtime }`
  per notebook (store in the sidecar's `updatedAt` + a small per-vault
  `.badnote/sync-manifest.json`, NOT synced).
- On pull, if remote and local both changed a file since last sync (both differ
  from the last-synced hash): **keep local, write the remote copy as
  `<file>.badnote.json.conflict-<timestamp>`** next to it, and surface a
  non-blocking notice. No silent overwrite, no merge attempt. For a single user on
  ≤2 devices this is rare and acceptable.
- The **source PDF/docx is effectively immutable** after import (annotations live in
  the sidecar), so the only file that realistically conflicts is the sidecar JSON —
  which is small and human-readable, making `.conflict` copies easy to reconcile.

### E.3 "Annotations follow the file" with sidecars

Because the sidecar sits in the same folder as the source file and shares its
basename, any sync that moves the folder moves both together. There is **no
database to keep in lockstep** — that is the robustness the user asked for. The
SQLite index is rebuilt locally from the synced files, never synced.

---

## F. Robust auto-save

### F.1 Atomic sidecar write

Single helper, e.g. `SidecarStore.writeAtomic(File target, String json)`:

1. Write to `target.path + '.tmp'` with `flush: true`
   (`File.writeAsString(..., flush: true)` — same pattern `PdfService` already uses
   with `writeAsBytes(flush: true)`).
2. `await tmp.rename(target.path)` — rename is atomic on the same filesystem on
   Windows/NTFS and POSIX, so a reader never sees a half-written sidecar.
3. Keep a one-deep backup: before rename, if `target` exists copy it to
   `target.path + '.bak'` (cheap insurance against a corrupt write taking out the
   last good copy). On load, if the main file fails to parse, fall back to `.bak`.

### F.2 Debounce + scheduler (reuse existing machinery)

- The editors already debounce: `SaveScheduler` (800 ms, `save_scheduler.dart`) for
  the PDF editor, and `SplitViewScreen`'s own 3 s `Timer`. **Reuse this exact
  shape**, but the scheduler's sink becomes the sidecar writer instead of
  `EditorRepository.saveHost`.
  - Concretely: introduce a `SidecarRepository` with the same method surface the
    `SaveScheduler` expects, so `_schedulePageSave` /
    `PenEditorScreen._initPersistence` change only their wiring, not their control
    flow. The scheduler still captures a synchronous snapshot before the async gap
    (it already does — `save_scheduler.dart` comment).
  - The unit of debounce stays "the whole document sidecar" (write the full JSON;
    sidecars are small — strokes are sparse normalized points). One write per
    debounce window, atomic.
- **Snapshot discipline:** capture the in-memory `_strokesByPage` /
  `_highlightsByPage` / `_scratchLinks` into a plain JSON map synchronously in
  `schedule(...)`, exactly as `SaveScheduler.schedule` captures strokes today, so a
  later edit can't corrupt an in-flight write.

### F.3 Flush on pause/close (robustness)

- `PenEditorScreen.dispose` already calls `scheduler.flush()` then `dispose()`
  (`pen_editor_screen.dart`); `SplitViewScreen.dispose` already calls
  `_saveImmediate()`. Keep both, pointing at the sidecar writer.
- **Add an app-lifecycle flush** (today missing): register a
  `WidgetsBindingObserver` (in `BadNoteApp` or each editor) and on
  `AppLifecycleState.inactive/paused/detached` call `flush()`. On a Windows tablet,
  app suspend/close is the main data-loss window; this closes it.
- Optionally, also flush on a short idle timer so a hard power-off loses ≤1 debounce
  window.

---

## G. Phased migration plan

Each phase is independently shippable and testable, ordered to minimize risk to the
working editors. "Files changed" lists the primary touch points.

### Phase 0 — Vault root + setup gate (no data move yet)
- Add `VaultService` (SharedPreferences-backed) + `VaultSetupScreen`.
- `main.dart`: gate `HomeScreen` behind `vaultRootValid()`; `SettingsScreen`:
  "Change vault".
- **No editor or DB change.** Editors still read/write SQLite. Vault path is merely
  recorded.
- **Ship/test:** first-run prompt appears, path persists, re-prompts when folder
  missing, Windows `getDirectoryPath` works (manual + a `VaultService` unit test).
- Files: `lib/services/vault_service.dart` (new), `lib/screens/vault_setup_screen.dart`
  (new), `lib/main.dart`, `lib/screens/settings_screen.dart`.

### Phase 1 — Sidecar format + atomic store + read/write library (no UI swap)
- Define `BadnoteSidecar` model (toJson/fromJson) per §A, reusing
  `EditorStroke`/`InkStroke`/`ScratchLink`/`Bookmark` JSON.
- `SidecarStore.writeAtomic` (§F.1) + `.bak` fallback loader.
- Pure unit tests: round-trip a sidecar with strokes/highlights/links/scratchpads;
  atomic-write crash simulation; `.bak` recovery.
- **No runtime behavior change yet** (library only).
- Files: `lib/storage/badnote_sidecar.dart` (new), `lib/storage/sidecar_store.dart`
  (new), tests.

### Phase 2 — PDF editor reads/writes sidecar (the core swap)
- Introduce `SidecarRepository` implementing the `SaveScheduler` sink; rewire
  `PenEditorScreen._initPersistence`, `_loadPersistedStrokes`,
  `_schedulePageSave`, `_loadScratchLinks`, `_placeScratchLink`,
  `_confirmDeleteScratchLink` to the sidecar instead of `EditorRepository` /
  `DatabaseService.*ScratchLink`. Persist highlights (closes `TODO(persist-highlights)`).
- `SplitViewScreen` reads/writes its scratchpad from the sidecar's
  `scratchLinks[].scratchpad`.
- The document's identity becomes its **vault path**, not the djb2 path-hash;
  `_documentIdFromPath` retired for this path.
- **Ship/test:** import a PDF (into vault), draw, place scratch links, reopen →
  everything restored from sidecar; SQLite `ink`/`scratch_links`/`scratchpads` no
  longer written for new docs. Add a widget/integration test.
- Files: `lib/editor/persistence/sidecar_repository.dart` (new),
  `lib/editor/canvas/pen_editor_screen.dart`, `lib/screens/split_view_screen.dart`.

### Phase 3 — Vault-backed import + top-level "Import file"
- `VaultService.createNotebook(sourceFilePath)` → makes the folder, copies the file
  in, returns the vault paths.
- Home screen: collapse `_importPdf`/`_importPptx` into one **"Import file"** action
  (sibling of "Create notebook"); multi-extension picker; route by extension.
- Documents list now comes from a **vault scan** (folders with sidecars), not
  `documents` table; `document_provider` reads the vault (cache table optional).
- Windows `soffice.exe` detection fix in `PptxService`; PPTX assets cached into
  `.badnote-assets/`; DOCX→PDF convert-on-import (best-effort, with the LibreOffice
  caveat surfaced to the user).
- **Ship/test:** one Import button accepts pdf/docx/pptx; imported files land in
  vault folders with sidecars; reopening reads from the vault.
- Files: `lib/screens/home_screen.dart`, `lib/providers/document_provider.dart`,
  `lib/services/vault_service.dart`, `lib/services/pptx_service.dart`,
  `lib/services/pdf_service.dart` (open vault copy).

### Phase 4 — Standalone notebooks + free notes on sidecars
- "Create notebook" writes a `notebook.badnote.json` (§A.4) instead of `notes`/`strokes`.
- `note_provider` reads/writes the vault; `home_screen` note tiles come from the scan.
- Files: `lib/providers/note_provider.dart`, `lib/editor/canvas/pen_note_screen.dart`,
  `lib/screens/home_screen.dart`.

### Phase 5 — One-time SQLite→sidecar migration (§B)
- Migrator runs on first launch with a valid vault and `vaultMigrationDone == false`.
- Demote SQLite to `.badnote/index.sqlite` cache; preserve legacy DB as
  `.premigration`.
- **Ship/test:** install over an old DB → all docs/strokes/scratchpads/bookmarks/notes
  appear in the vault; idempotent re-run; legacy DB preserved. Golden-file tests
  with a seeded legacy DB.
- Files: `lib/storage/sqlite_to_sidecar_migrator.dart` (new),
  `lib/services/database_service.dart` (relocate DB path, expose raw read helpers),
  `lib/main.dart` (invoke migrator).

### Phase 6 — Lifecycle-flush hardening + index rebuild
- `WidgetsBindingObserver` app-pause flush (§F.3) across editors.
- `.badnote/index.sqlite` (FTS + recent list + OCR) rebuilt by scanning sidecars;
  `search_provider` queries the cache.
- Files: `lib/main.dart` (or a shared observer), the editors, `lib/providers/search_provider.dart`.

### Phase 7 — Server sync (optional, last)
- `SyncService`: WebDAV (or "vault lives in an OS-synced folder → no-op") +
  per-notebook manifest + last-write-wins `.conflict` policy (§E).
- Settings UI to configure endpoint/credentials; manual "Sync now" + periodic.
- Files: `lib/services/sync_service.dart` (new), `lib/screens/settings_screen.dart`.

### Risk-minimization rationale
- Phases 0–1 add code without changing runtime behavior (lowest risk first).
- Phase 2 is the one delicate swap; it is isolated to the persistence wiring of two
  screens and is backed by Phase-1 round-trip tests — the on-screen stroke models
  and painters are untouched.
- The destructive step (Phase 5 migration) ships **after** the new format is proven
  by Phases 2–4, and never deletes the legacy DB.

---

## Appendix: exact files inspected

`lib/services/database_service.dart` (schema v8; `documents`, `ink`, `scratch_links`,
`scratchpads`, `bookmarks`, `annotations`, `notes`, `strokes`, `board_cards`, FTS),
`lib/editor/persistence/editor_repository.dart` (`saveHost`, `loadDocument`,
`pageHostId`, host_id scheme), `lib/editor/persistence/save_scheduler.dart`
(800 ms debounce, synchronous snapshot, `flush`), `lib/editor/engine/stroke_model.dart`
(`EditorStroke.toJson`, brush not serialized), `lib/models/ink_stroke.dart`,
`lib/models/ink_point.dart`, `lib/editor/canvas/pen_stroke.dart`,
`lib/models/scratch_link.dart`, `lib/models/bookmark.dart`, `lib/models/document.dart`,
`lib/editor/canvas/pen_editor_screen.dart` (`_documentIdFromPath`,
`_initPersistence`, `_schedulePageSave`, `_highlightsByPage` +
`TODO(persist-highlights)`, scratch-link flow), `lib/screens/split_view_screen.dart`
(scratchpad keyed by anchor id, absolute world pixels, 3 s autosave),
`lib/screens/home_screen.dart` (`_importPdf`/`_importPptx`, empty-state buttons),
`lib/services/pdf_service.dart` (`pickPdfFile`, `exportAnnotatedPdf`, `writeAsBytes(flush:true)`),
`lib/services/pptx_service.dart` (LibreOffice headless, `which libreoffice`,
temp-dir output), `lib/providers/document_provider.dart`,
`lib/providers/settings_provider.dart` (SharedPreferences pattern), `lib/main.dart`
(`DatabaseService.getInstance`, `SharedPreferences.getInstance`, `HomeScreen` home).