BadNote/docs/plans/2026-06-24-file-based-storage.md

BadNote — File-Based Storage Re-Architecture (Obsidian-style vault)

Status: DESIGN (not yet implemented) Date: 2026-06-24 Owner plan file: docs/plans/2026-06-24-file-based-storage.md Mode: DELIBERATE (touches persistence of a working pen editor — must not lose ink)

Grounding note: written after reading the actual current sources — lib/services/database_service.dart, lib/editor/persistence/editor_repository.dart, lib/editor/persistence/save_scheduler.dart, lib/editor/canvas/pen_editor_screen.dart, lib/screens/split_view_screen.dart, lib/screens/home_screen.dart, lib/services/pdf_service.dart, lib/services/pptx_service.dart, lib/providers/document_provider.dart, lib/providers/settings_provider.dart, lib/main.dart, and the stroke models (lib/editor/engine/stroke_model.dart, lib/models/ink_stroke.dart, lib/models/ink_point.dart, lib/editor/canvas/pen_stroke.dart, lib/models/scratch_link.dart, lib/models/bookmark.dart, lib/models/document.dart). file_picker getDirectoryPath (Windows desktop support) confirmed via Context7 against the installed file_picker: ^8.0.0.

---

0. The user's intent (verbatim)

1. "Import file" is a TOP-LEVEL home-screen action, sibling of "Create notebook", accepting multiple file types (docx, pptx, pdf, …).
2. Annotations travel WITH the file ("跟着文件走") — stored as JSON sidecar files next to the source file, NOT in SQLite. Sync follows the file.
3. After importing a PDF, the file is placed inside a notebook folder, then synced to a server. User believes this is more robust than SQLite.
4. On first launch, the user picks a notebook ROOT folder (an Obsidian-style vault).
5. Robust auto-save.

---

1. What is stored in SQLite TODAY (inventory)

Read from lib/services/database_service.dart (schema version 8). This is the data that must move to files or be re-homed.

Table	Written/read by	Holds	Coordinate format	Disposition
documents	DocumentListNotifier (document_provider.dart), DatabaseService.insertDocument/getAllDocuments/deleteDocument	id (uuid), filename, doc_type (pdf/pptx), absolute file_path, page_count, rotation, timestamps	—	→ derived from vault scan; becomes a cache/index only
ink	EditorRepository (editor_repository.dart), SaveScheduler	committed PDF-editor strokes; one row/stroke; host_id = "doc:<documentId>:page:<pageIndex>", stroke_json = EditorStroke.toJson()	normalized [0,1], width = fraction of page width	→ sidecar strokes[pageIndex]
scratch_links	PenEditorScreen._placeScratchLink, DatabaseService.saveScratchLink/loadScratchLinks	anchor id, document_id, page_index, nx, ny	normalized [0,1]	→ sidecar scratchLinks[]
scratchpads	SplitViewScreen (keyed by anchor id), DatabaseService.saveScratchpad/loadScratchpad	one row per anchor id; strokes_json = list of InkStroke.toJson()	absolute world pixels (infinite canvas)	→ sidecar scratchLinks[].scratchpad.strokes[]
bookmarks	DatabaseService.insertBookmark/getBookmarks	id, document_id, page_number, label, color	page index	→ sidecar bookmarks[]
annotations	DatabaseService.saveAnnotations/getAnnotations	legacy per-page annotation_json blob	per-page	DEAD in new editor — migrate if present, else ignore
notes + strokes	note_provider.dart via insertNote/updateNote; Note model	free ink notebooks (not file-backed); strokes are InkStroke rows	normalized	→ standalone notebook sidecars (see §A.4)
board_cards	DatabaseService.saveBoardCards/loadBoard (F7 sticky board)	board cards geometry+text	absolute	→ per-notebook board.json (or keep in SQLite short-term, §B)
document_fts, notes_fts, ocr_results, notebook_pages	FTS + OCR	search index, OCR text	—	stays in SQLite as a rebuildable cache (never the source of truth)

Key existing facts the new format must preserve (cite):

• EditorStroke.toJson() (freezed/json_serializable, stroke_model.dart) emits: { "id", "points":[{ "x","y","pressure","tilt","timestamp","pointerDeviceKind" }], "tool":"pen"|"highlighter"|"eraser", "color":<int ARGB>, "width":<double>, "filled":<bool>, "textContent":<String?>, "fontSize":<double> }. brush is deliberately NOT serialized (@JsonKey(includeFromJson:false,includeToJson:false) in stroke_model.dart), so loaded strokes default to BrushKind.fountainPen. The sidecar format inherits this limitation (see §A.5 "brush TODO").
• InkStroke.toJson() (ink_stroke.dart) emits the same point shape plus createdAt and strokeWidth (note: strokeWidth, not width). This is the format already persisted for scratchpads and free notes.
• ScratchLink.toJson() (scratch_link.dart, hand-written): {id, documentId, pageIndex, nx, ny}.
• The editor derives its documentId from the file path via a djb2 hash (_documentIdFromPath in pen_editor_screen.dart). Once strokes live in a sidecar next to the file, the path-hash document id becomes irrelevant — the sidecar IS the identity. This removes a class of bugs (moving a file orphaned its SQLite rows).

---

2. Decision summary (what we are building)

• A vault = a user-picked root folder, path stored in SharedPreferences.
• Each imported document becomes a notebook folder inside the vault: <vault>/<notebook-name>/ containing the source file + one sidecar <file>.badnote.json holding all annotations for that file.
• The sidecar is the source of truth. SQLite is demoted to a rebuildable index/cache (FTS + thumbnails + recent list). Recommendation: keep SQLite, but only as cache (§B explains why dropping it entirely is more work than it's worth right now).
• Auto-save writes the sidecar atomically (temp + rename), debounced, with flush on pause/close.
• Sync = file-level sync of the whole vault folder to a server; sync unit = notebook folder; conflict policy = last-write-wins per file + .conflict copy.

---

A. On-disk layout + JSON schema

A.1 Vault layout

<vault root>/                         ← user-picked, persisted in SharedPreferences
├─ .badnote/                          ← vault-level app metadata (hidden)
│  ├─ vault.json                      ← { "schemaVersion": 1, "vaultId": "<uuid>", "createdAt": ... }
│  └─ index.sqlite                    ← OPTIONAL local cache/index (FTS, thumbnails). Rebuildable. NOT synced.
├─ Calculus Lecture 3/               ← a notebook folder (one per imported doc)
│  ├─ Calculus Lecture 3.pdf          ← the source file (pdf/docx/pptx/…)
│  ├─ Calculus Lecture 3.pdf.badnote.json   ← the sidecar (annotations travel with the file)
│  └─ .badnote-assets/                ← optional: rendered page PNGs for pptx/docx, thumbnails
│     ├─ slide_1.png …
│     └─ thumb_0.png …
├─ My freehand notes/                ← a NON-file-backed notebook (free ink, no source doc)
│  └─ notebook.badnote.json           ← strokes-only notebook (replaces SQLite notes/strokes)
└─ …

Rules:

• Notebook folder name = sanitized source filename (basename without extension), de-duplicated with a numeric suffix on collision.
• Sidecar name = <source filename incl. ext>.badnote.json. Keeping the full source extension in the sidecar name means a folder with both slides.pdf and slides.pptx never collides.
• Sidecar lives in the same folder as its file → moving/copying/syncing the folder moves the annotations with it ("跟着文件走"). This is the whole point.
• .badnote/index.sqlite is per-vault and excluded from sync (it is a cache; each device rebuilds its own). It replaces today's getApplicationDocumentsDirectory()/badnote.db.

A.2 Sidecar JSON schema (file-backed document)

<file>.badnote.json (UTF-8, pretty-printed for diff-friendliness / git sync):

{
  "badnoteSidecarVersion": 1,
  "sourceFile": "Calculus Lecture 3.pdf",
  "docType": "pdf",
  "pageCount": 42,
  "rotation": 0,
  "createdAt": "2026-06-24T10:00:00.000Z",
  "updatedAt": "2026-06-24T10:32:11.500Z",

  "strokes": {
    "0": [ <EditorStroke.toJson()>, <EditorStroke.toJson()>, … ],
    "3": [ <EditorStroke.toJson()>, … ]
  },

  "highlights": {
    "0": [ { "l": 0.12, "t": 0.20, "r": 0.88, "b": 0.235, "color": 1714657595 } ]
  },

  "bookmarks": [
    { "id": "<uuid>", "pageNumber": 5, "label": "Proof", "color": 4283215696, "createdAt": "…" }
  ],

  "scratchLinks": [
    {
      "id": "<uuid>",
      "pageIndex": 7,
      "nx": 0.83, "ny": 0.41,
      "createdAt": "…",
      "scratchpad": {
        "canvasWidth": 4000,
        "canvasHeight": 4000,
        "strokes": [ <InkStroke.toJson()>, … ]   // absolute world pixels (unchanged format)
      }
    }
  ]
}

Schema notes, tied to existing code:

• strokes is a map keyed by string page index → list of EditorStroke.toJson() objects. This is byte-for-byte the JSON already written to the ink table's stroke_json column by EditorRepository.saveHost (jsonEncode(stroke.toJson())). Loading just calls EditorStroke.fromJson.
  • Drop the host_id = "doc:<id>:page:<i>" convention entirely; the sidecar key IS the page index. EditorRepository.pageHostId / _pageIndexFromHostId (in pen_editor_screen.dart) become obsolete for the file path.
• highlights persists what is TODAY in-memory only — see _highlightsByPage and TODO(persist-highlights) in pen_editor_screen.dart. Each rect stored normalized [0,1] (l/t/r/b) exactly as _highlightSelection computes it. This closes that TODO as a side benefit.
• bookmarks mirrors Bookmark.toJson() (bookmark.dart); documentId field is dropped (the sidecar already scopes it).
• scratchLinks[] merges today's TWO tables: scratch_links (anchor geometry) + scratchpads (the anchor's private ink, currently keyed by anchor id). Each anchor now embeds its scratchpad. scratchpad.strokes keep the InkStroke.toJson() format in absolute world pixels — unchanged from SplitViewScreen._saveImmediate, so the infinite-canvas logic (_checkCanvasExpansion, penStrokesFromInk) needs no change. We persist canvasWidth/Height so the world size restores (today it always resets to 4000×4000).

A.3 Why normalized vs absolute coords are preserved verbatim

• PDF-editor strokes (EditorStroke) are normalized to the page rect; width is a fraction of page width (stroke_model.dart header comment). Sidecar stores them unchanged → no re-projection, no rounding drift, export (PdfService.exportAnnotatedPdf) keeps working untouched.
• Scratchpad strokes (InkStroke) are absolute world pixels (infinite canvas). Sidecar stores them unchanged.
• This is a format-preserving migration: same toJson/fromJson, different container (file vs row). That is what makes it low-risk.

A.4 Standalone (non-file) notebooks — the "Create notebook" path

Free-ink notes today live in notes + strokes (Note model, note_provider.dart). In the vault they become a notebook folder with no source file, holding a notebook.badnote.json:

{
  "badnoteSidecarVersion": 1,
  "docType": "notebook",
  "title": "My freehand notes",
  "tags": ["math"],
  "createdAt": "…", "updatedAt": "…",
  "pages": [
    { "ordinal": 0, "strokes": [ <InkStroke.toJson()>, … ] }
  ]
}

(InkStroke.toJson() is the exact format Note.strokes already serialize to.)

A.5 Brush persistence caveat (honest limitation)

EditorStroke.brush and PenStroke.brush are not serialized today (@JsonKey(includeFromJson:false,includeToJson:false), see stroke_model.dart and TODO(brush-persist) in pen_editor_screen.dart). The sidecar inherits this: a reloaded pen stroke renders as fountainPen; a highlighter is recovered from tool == highlighter. Recommendation: add an optional "brush" field to the sidecar EditorStroke JSON in a later increment by flipping the @JsonKey — the sidecar schema is forward-compatible (unknown fields ignored on read), so this is non-breaking. Not required for this storage migration.

---

B. Migration: SQLite → sidecars (no data loss)

Recommendation: KEEP SQLite, demote it to a rebuildable cache. Do NOT drop it.

Reasons:

• FTS5 (document_fts, notes_fts) and OCR (ocr_results) are non-trivial and query-shaped; re-implementing search over flat JSON files is a separate project. Keep them in .badnote/index.sqlite, rebuilt by scanning sidecars.
• The home screen's "recent documents" list (getAllDocuments) wants fast sorted access; a cache table is the pragmatic backing for it (the source of truth is still the vault scan).
• Dropping SQLite forces rewriting note_provider, document_provider, search_provider, OCR, and the board in one shot — high blast radius. Demoting is incremental and reversible.

B.1 One-time migration on first launch after upgrade

Guarded by a SharedPreferences flag vaultMigrationDone (and only runs once a vault root exists — see §C). Algorithm:

1. Open the legacy DB at getApplicationDocumentsDirectory()/badnote.db (the path DatabaseService._initialize uses today). If absent → nothing to migrate.
2. For each row in documents: a. Resolve the legacy file_path (absolute). If the file still exists, copy it into a new notebook folder <vault>/<sanitized filename>/. b. Build the sidecar:
   • strokes: query ink WHERE host_id LIKE 'doc:<documentId>:page:%' (the EditorRepository.loadDocument query), group by page index parsed from host_id, write each stroke_json straight through (it's already EditorStroke JSON — no re-encode).
   • bookmarks: getBookmarks(documentId).
   • scratchLinks: loadScratchLinks(documentId); for each, loadScratchpad(anchorId) → embed as scratchpad.strokes (re-encode via InkStroke.toJson).
   • annotations (legacy per-page blob): if any rows exist, attempt to decode and fold into strokes; if format is unrecognized, copy the raw blob into a legacyAnnotations field so nothing is silently dropped. c. Write the sidecar atomically (§F).
3. For each notes row → write a standalone notebook sidecar (§A.4) under a notebook folder.
4. For board_cards: SHORT TERM leave them in SQLite (board is self-contained and not part of the "files" intent). LATER, write a board.json per notebook.
5. Do not delete the legacy DB. Rename it to badnote.db.premigration as a safety net. Set vaultMigrationDone = true.
6. Rebuild .badnote/index.sqlite (FTS + recent list) by scanning the new vault.

B.2 Crash safety of migration

• Process sidecars one notebook at a time; each sidecar write is atomic.
• The migration is idempotent: re-running skips notebook folders whose sidecar already exists and validates. If it dies halfway, relaunch resumes.
• Because the legacy DB is preserved until the flag flips, a failed migration loses nothing.

---

C. Folder picker on init (the vault prompt)

Use file_picker — already a dependency (file_picker: ^8.0.0 in pubspec.yaml, already used by PdfService.pickPdfFile / PptxService.openPptxFile). Its FilePicker.platform.getDirectoryPath() is Desktop/Windows supported (confirmed via Context7). No new package needed. Do NOT add file_selector — file_picker already covers both file and directory picking and is wired in.

C.1 Persistence + flow

• New VaultService (singleton, like DatabaseService):
  • Future<String?> getVaultRoot() — reads SharedPreferences key vaultRoot.
  • Future<void> setVaultRoot(String path) — writes it.
  • Future<bool> vaultRootValid() — true iff the stored path exists and is a writable directory.
• main.dart change: after SharedPreferences.getInstance(), check vaultRootValid().
  • If valid → go to HomeScreen as today.
  • If missing/invalid → show a VaultSetupScreen (a gate before HomeScreen) that explains "Pick a folder to store your notebooks (like an Obsidian vault)" and calls getDirectoryPath(dialogTitle: 'Choose your BadNote vault', lockParentWindow: true). On selection: create <root>/.badnote/vault.json, persist the path, then enter HomeScreen.
• Re-prompt if missing: if the stored path later disappears (external drive unplugged, folder deleted), vaultRootValid() returns false → the gate shows again with a "your vault folder is missing — relocate or pick a new one" message. Never silently fall back to app-documents (that would scatter data).
• A "Change vault" entry in SettingsScreen re-runs the picker.

C.2 Windows specifics

• Pass lockParentWindow: true so the native dialog is modal (Context7 note).
• Wrap in try/catch (Context7 shows getDirectoryPath can throw on Windows for permission/system issues) and surface a retry.
• Validate the chosen folder is writable by writing+deleting a probe file before committing it as the vault.

---

D. Multi-format import (docx / pptx / pdf)

This is the genuinely hard part. Be honest about it.

D.1 The home-screen entry (requirement #1)

Replace today's two separate IconButtons (_importPdf, _importPptx in home_screen.dart) and the empty-state buttons with one top-level "Import file" action, a sibling of "Create notebook" (the FAB / _createAndOpenNote). A single FilePicker.pickFiles(type: FileType.custom, allowedExtensions: ['pdf','docx','pptx','ppt']) call; route on extension. Both actions sit at the same visual level (e.g. two primary buttons in the empty state, and two entries in the app bar / a small "+ New" menu with "Create notebook" and "Import file").

D.2 Per-format strategy

Format	Annotate how	Mechanism	Windows-viable?
PDF	Directly, as today	pdfrx PdfViewer + normalized ink overlay (PenEditorScreen, unchanged)	YES — already shipping
PPTX/PPT	Convert to images, annotate as slides	PptxService.convertToImages (LibreOffice headless → PNG, fallback placeholders) + PenSlideScreen	PARTIAL — needs LibreOffice
DOCX	Convert to PDF, then annotate as PDF (recommended)	LibreOffice headless --convert-to pdf, then the PDF path flows into PenEditorScreen	PARTIAL — needs LibreOffice

D.3 The realistic recommendation

• PDF: unchanged. Place the picked file into a notebook folder, open PenEditorScreen on the vault copy (not the original picked path).
• DOCX → PDF (convert-on-import): the cleanest path is to convert DOCX to a PDF once, at import time, store the PDF as the notebook's annotatable artifact (keep the original .docx alongside it for fidelity/round-trip). Then everything downstream is the existing, working PDF pipeline. This is far simpler than rendering Word layout natively in Flutter (there is no good pure-Dart DOCX renderer).
• PPTX: keep the existing slide-image path (PptxService + PenSlideScreen), but cache the rendered PNGs into the notebook's .badnote-assets/ instead of a temp dir (today convertToImages writes to getTemporaryDirectory(), so slides re-render every open — see home_screen.dart _openDocument). Caching also makes the notebook self-contained for sync.

D.4 The hard truth about conversion (call it out)

• Both DOCX→PDF and PPTX→PNG currently depend on LibreOffice headless being on PATH (PptxService._convertViaLibreOffice runs which libreoffice then libreoffice --headless --convert-to …). On a Windows tablet, LibreOffice is usually NOT installed, and which is a POSIX tool that won't resolve soffice.exe. This path will silently fail today on Windows and fall back to placeholder slides.
• Proposed fallbacks, in order:
  1. Detect LibreOffice/soffice.exe at the standard Windows install paths (C:\Program Files\LibreOffice\program\soffice.exe) in addition to PATH; invoke soffice (not libreoffice) on Windows. Fix the which assumption.
  2. If absent, prompt the user ("Install LibreOffice to import Word/PowerPoint, or convert to PDF first"), and offer a "locate soffice.exe" picker that we persist in SharedPreferences.
  3. Bundle/ship nothing heavy. Do not attempt to embed a converter. For a single-user tablet app, requiring LibreOffice (or pre-export to PDF) is an acceptable, honest constraint.
• Minimal first cut: ship PDF import end-to-end on the new vault, plus the one-file "Import file" entry that accepts docx/pptx, but for docx/pptx route through the existing (LibreOffice-dependent) converters with the Windows soffice.exe fix. Treat full docx/pptx fidelity as a known limitation, not a blocker for the storage re-architecture.

---

E. Server sync (file-level, single user)

Keep it pragmatic. The data is now plain files in one folder, which is exactly what makes simple sync viable.

E.1 Sync unit & approach

• Sync unit = the notebook folder (source file + sidecar + assets). A notebook is self-contained, so syncing the folder syncs the annotations with it.
• Exclude .badnote/index.sqlite and .badnote-assets/ from sync if desired (assets are re-derivable; the index is per-device). Source file + sidecar are the must-sync pair.
• Recommended concrete approach (single-user): WebDAV to a self-hosted/Nextcloud endpoint, OR a simple REST blob sync if the user controls the server.
  • WebDAV is the lowest-effort robust option (PUT/GET/PROPFIND, mtime-based), works against Nextcloud/ownCloud/rclone-serve, and there are Dart HTTP clients.
  • If the user already uses Nextcloud/Dropbox/OneDrive and the vault folder lives inside that synced folder, BadNote needs zero sync code — the OS sync client handles it. This is the cheapest path and worth recommending as option 0.
  • git is possible (text JSON diffs nicely) but binary PDFs bloat history and conflict UX is poor for a tablet — not recommended as the default.

E.2 Conflict policy

• Last-write-wins per file, using a manifest of { relativePath, sha256, mtime } per notebook (store in the sidecar's updatedAt + a small per-vault .badnote/sync-manifest.json, NOT synced).
• On pull, if remote and local both changed a file since last sync (both differ from the last-synced hash): keep local, write the remote copy as <file>.badnote.json.conflict-<timestamp> next to it, and surface a non-blocking notice. No silent overwrite, no merge attempt. For a single user on ≤2 devices this is rare and acceptable.
• The source PDF/docx is effectively immutable after import (annotations live in the sidecar), so the only file that realistically conflicts is the sidecar JSON — which is small and human-readable, making .conflict copies easy to reconcile.

E.3 "Annotations follow the file" with sidecars

Because the sidecar sits in the same folder as the source file and shares its basename, any sync that moves the folder moves both together. There is no database to keep in lockstep — that is the robustness the user asked for. The SQLite index is rebuilt locally from the synced files, never synced.

---

F. Robust auto-save

F.1 Atomic sidecar write

Single helper, e.g. SidecarStore.writeAtomic(File target, String json):

1. Write to target.path + '.tmp' with flush: true (File.writeAsString(..., flush: true) — same pattern PdfService already uses with writeAsBytes(flush: true)).
2. await tmp.rename(target.path) — rename is atomic on the same filesystem on Windows/NTFS and POSIX, so a reader never sees a half-written sidecar.
3. Keep a one-deep backup: before rename, if target exists copy it to target.path + '.bak' (cheap insurance against a corrupt write taking out the last good copy). On load, if the main file fails to parse, fall back to .bak.

F.2 Debounce + scheduler (reuse existing machinery)

• The editors already debounce: SaveScheduler (800 ms, save_scheduler.dart) for the PDF editor, and SplitViewScreen's own 3 s Timer. Reuse this exact shape, but the scheduler's sink becomes the sidecar writer instead of EditorRepository.saveHost.
  • Concretely: introduce a SidecarRepository with the same method surface the SaveScheduler expects, so _schedulePageSave / PenEditorScreen._initPersistence change only their wiring, not their control flow. The scheduler still captures a synchronous snapshot before the async gap (it already does — save_scheduler.dart comment).
  • The unit of debounce stays "the whole document sidecar" (write the full JSON; sidecars are small — strokes are sparse normalized points). One write per debounce window, atomic.
• Snapshot discipline: capture the in-memory _strokesByPage / _highlightsByPage / _scratchLinks into a plain JSON map synchronously in schedule(...), exactly as SaveScheduler.schedule captures strokes today, so a later edit can't corrupt an in-flight write.

F.3 Flush on pause/close (robustness)

• PenEditorScreen.dispose already calls scheduler.flush() then dispose() (pen_editor_screen.dart); SplitViewScreen.dispose already calls _saveImmediate(). Keep both, pointing at the sidecar writer.
• Add an app-lifecycle flush (today missing): register a WidgetsBindingObserver (in BadNoteApp or each editor) and on AppLifecycleState.inactive/paused/detached call flush(). On a Windows tablet, app suspend/close is the main data-loss window; this closes it.
• Optionally, also flush on a short idle timer so a hard power-off loses ≤1 debounce window.

---

G. Phased migration plan

Each phase is independently shippable and testable, ordered to minimize risk to the working editors. "Files changed" lists the primary touch points.

Phase 0 — Vault root + setup gate (no data move yet)

• Add VaultService (SharedPreferences-backed) + VaultSetupScreen.
• main.dart: gate HomeScreen behind vaultRootValid(); SettingsScreen: "Change vault".
• No editor or DB change. Editors still read/write SQLite. Vault path is merely recorded.
• Ship/test: first-run prompt appears, path persists, re-prompts when folder missing, Windows getDirectoryPath works (manual + a VaultService unit test).
• Files: lib/services/vault_service.dart (new), lib/screens/vault_setup_screen.dart (new), lib/main.dart, lib/screens/settings_screen.dart.

Phase 1 — Sidecar format + atomic store + read/write library (no UI swap)

• Define BadnoteSidecar model (toJson/fromJson) per §A, reusing EditorStroke/InkStroke/ScratchLink/Bookmark JSON.
• SidecarStore.writeAtomic (§F.1) + .bak fallback loader.
• Pure unit tests: round-trip a sidecar with strokes/highlights/links/scratchpads; atomic-write crash simulation; .bak recovery.
• No runtime behavior change yet (library only).
• Files: lib/storage/badnote_sidecar.dart (new), lib/storage/sidecar_store.dart (new), tests.

Phase 2 — PDF editor reads/writes sidecar (the core swap)

• Introduce SidecarRepository implementing the SaveScheduler sink; rewire PenEditorScreen._initPersistence, _loadPersistedStrokes, _schedulePageSave, _loadScratchLinks, _placeScratchLink, _confirmDeleteScratchLink to the sidecar instead of EditorRepository / DatabaseService.*ScratchLink. Persist highlights (closes TODO(persist-highlights)).
• SplitViewScreen reads/writes its scratchpad from the sidecar's scratchLinks[].scratchpad.
• The document's identity becomes its vault path, not the djb2 path-hash; _documentIdFromPath retired for this path.
• Ship/test: import a PDF (into vault), draw, place scratch links, reopen → everything restored from sidecar; SQLite ink/scratch_links/scratchpads no longer written for new docs. Add a widget/integration test.
• Files: lib/editor/persistence/sidecar_repository.dart (new), lib/editor/canvas/pen_editor_screen.dart, lib/screens/split_view_screen.dart.

Phase 3 — Vault-backed import + top-level "Import file"

• VaultService.createNotebook(sourceFilePath) → makes the folder, copies the file in, returns the vault paths.
• Home screen: collapse _importPdf/_importPptx into one "Import file" action (sibling of "Create notebook"); multi-extension picker; route by extension.
• Documents list now comes from a vault scan (folders with sidecars), not documents table; document_provider reads the vault (cache table optional).
• Windows soffice.exe detection fix in PptxService; PPTX assets cached into .badnote-assets/; DOCX→PDF convert-on-import (best-effort, with the LibreOffice caveat surfaced to the user).
• Ship/test: one Import button accepts pdf/docx/pptx; imported files land in vault folders with sidecars; reopening reads from the vault.
• Files: lib/screens/home_screen.dart, lib/providers/document_provider.dart, lib/services/vault_service.dart, lib/services/pptx_service.dart, lib/services/pdf_service.dart (open vault copy).

Phase 4 — Standalone notebooks + free notes on sidecars

• "Create notebook" writes a notebook.badnote.json (§A.4) instead of notes/strokes.
• note_provider reads/writes the vault; home_screen note tiles come from the scan.
• Files: lib/providers/note_provider.dart, lib/editor/canvas/pen_note_screen.dart, lib/screens/home_screen.dart.

Phase 5 — One-time SQLite→sidecar migration (§B)

• Migrator runs on first launch with a valid vault and vaultMigrationDone == false.
• Demote SQLite to .badnote/index.sqlite cache; preserve legacy DB as .premigration.
• Ship/test: install over an old DB → all docs/strokes/scratchpads/bookmarks/notes appear in the vault; idempotent re-run; legacy DB preserved. Golden-file tests with a seeded legacy DB.
• Files: lib/storage/sqlite_to_sidecar_migrator.dart (new), lib/services/database_service.dart (relocate DB path, expose raw read helpers), lib/main.dart (invoke migrator).

Phase 6 — Lifecycle-flush hardening + index rebuild

• WidgetsBindingObserver app-pause flush (§F.3) across editors.
• .badnote/index.sqlite (FTS + recent list + OCR) rebuilt by scanning sidecars; search_provider queries the cache.
• Files: lib/main.dart (or a shared observer), the editors, lib/providers/search_provider.dart.

Phase 7 — Server sync (optional, last)

• SyncService: WebDAV (or "vault lives in an OS-synced folder → no-op") + per-notebook manifest + last-write-wins .conflict policy (§E).
• Settings UI to configure endpoint/credentials; manual "Sync now" + periodic.
• Files: lib/services/sync_service.dart (new), lib/screens/settings_screen.dart.

Risk-minimization rationale

• Phases 0–1 add code without changing runtime behavior (lowest risk first).
• Phase 2 is the one delicate swap; it is isolated to the persistence wiring of two screens and is backed by Phase-1 round-trip tests — the on-screen stroke models and painters are untouched.
• The destructive step (Phase 5 migration) ships after the new format is proven by Phases 2–4, and never deletes the legacy DB.

---

Appendix: exact files inspected

lib/services/database_service.dart (schema v8; documents, ink, scratch_links, scratchpads, bookmarks, annotations, notes, strokes, board_cards, FTS), lib/editor/persistence/editor_repository.dart (saveHost, loadDocument, pageHostId, host_id scheme), lib/editor/persistence/save_scheduler.dart (800 ms debounce, synchronous snapshot, flush), lib/editor/engine/stroke_model.dart (EditorStroke.toJson, brush not serialized), lib/models/ink_stroke.dart, lib/models/ink_point.dart, lib/editor/canvas/pen_stroke.dart, lib/models/scratch_link.dart, lib/models/bookmark.dart, lib/models/document.dart, lib/editor/canvas/pen_editor_screen.dart (_documentIdFromPath, _initPersistence, _schedulePageSave, _highlightsByPage + TODO(persist-highlights), scratch-link flow), lib/screens/split_view_screen.dart (scratchpad keyed by anchor id, absolute world pixels, 3 s autosave), lib/screens/home_screen.dart (_importPdf/_importPptx, empty-state buttons), lib/services/pdf_service.dart (pickPdfFile, exportAnnotatedPdf, writeAsBytes(flush:true)), lib/services/pptx_service.dart (LibreOffice headless, which libreoffice, temp-dir output), lib/providers/document_provider.dart, lib/providers/settings_provider.dart (SharedPreferences pattern), lib/main.dart (DatabaseService.getInstance, SharedPreferences.getInstance, HomeScreen home).