Compare commits

..
2 Commits
Author SHA1 Message Date
marcandClaude Opus 5 8e38404a69 Read a PDF where it is imported, and keep the text with the file
A PDF was searchable on a device once that device had opened it, because
the extracted text lived in that browser's IndexedDB and nowhere else. So
a document imported on the desktop was invisible to a search on the
phone — which is exactly the case that matters for filing invoices, since
those are imported and then never opened again.

The reading still never happens on the server. It has no PDF library and
is not getting one: the bytes arrive encrypted and are stored that way.
The device that imports the file reads it there and then, and the result
is stored beside the file in attachments.text_index, encrypted at rest
like everything else and opaque to the server. Every other device asks
for it after login and can then find a document it has never downloaded.

Not storing it was the earlier decision, on the reasoning that text
derivable from bytes the server already holds should not be kept twice.
True, and useless. A few hundred kilobytes against a file measured in
megabytes is a trade worth making.

A `null` answer is not an error — it means nobody has read this file yet.
Documents extracted before this existed are sent up the next time they
are opened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 19:26:07 +02:00
marcandClaude Opus 5 228552d453 Let a drawing run over as many pages as the note
The ink was stuck on page one while the writing carried on without it,
which stopped making sense the moment a note could run to four pages.
Every page is exported anyway, so every page can be drawn on.

Pages are numbered the way the PDF reader numbers them, and only pages
that carry ink are stored, so the keys are the drawn pages. While drawing
there is always one blank page past the end: pages otherwise come from
how far the writing runs, and to draw on page three you would first have
had to write three pages.

A drawing that lives entirely on page one is still written in the old
single-page shape. Nothing requires that — it means an ordinary note is
stored exactly as it always was, and a device running an older build
still shows it.

The bin clears the page being looked at. Undo and redo work on the
drawing as a whole, because one Undo button that means four different
things depending on where you scrolled is not a button anyone can use.

And a real bug found on the way out: the exporter has written
drawings/<note>.png since the interface was put into English, while the
importer was still looking for zeichnungen/<note>.png. A drawing exported
by this very app came back from its own archive as nothing at all. Both
names are read now, and so are the further pages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 19:25:56 +02:00
17 changed files with 939 additions and 159 deletions

No files matched your search

+27
View File
@@ -8,6 +8,33 @@ not from the output of `docker push`.
---
## Unreleased
**Added**
- **A PDF is searchable on every device, not only on the one that opened it.**
The device that imports the file reads it there and then — the server still has
no PDF library and never opens one — and the extracted text is stored beside
the file, encrypted, so every other device can find the document without ever
downloading it. This is what makes an archive of scanned invoices work: they
are filed and then never opened, so waiting for someone to open one meant
waiting forever.
- **A drawing runs over as many pages as the note has.** The ink used to be stuck
on page one while the writing carried on without it. Pages are numbered the way
the PDF reader numbers them, and there is always one blank page past the end to
start a new one on. The bin clears the page being looked at; undo still works on
the drawing as a whole.
- A drawing that lives entirely on page one is still stored in the old shape, so
an ordinary note is written exactly as it always was and an older client still
shows it.
**Fixed**
- **A drawing exported in a ZIP was lost on re-import.** The exporter has written
`drawings/<note>.png` since the interface was put into English; the importer
was still looking for `zeichnungen/<note>.png` and found nothing. Both names
are read now, and so are the further pages of a drawing.
---
## 2026-08-11 — The writing column
**Images** — web `sha256:ed11f034…`, server unchanged at `sha256:1648498f…`
+2 -2
View File
@@ -31,10 +31,10 @@ Diese Punkte sind Entscheidungen von Marc, keine offenen Fragen:
Beides muss vor einem Push grün sein.
```bash
# Unit (224 Tests, ~4 s)
# Unit (~4 s)
docker run --rm -v "$PWD/frontend:/w" -w /w node:20-alpine npx vitest run
# Browser, zwölf Suiten (~263 Checks) – braucht den Dev-Stack
# Browser, zwölf Suiten – braucht den Dev-Stack
./e2e/run.sh # alle
./e2e/run.sh editor drawing # einzelne
```
+79
View File
@@ -391,6 +391,26 @@ if (!tableColumns("users").includes("preferences")) {
db.exec("ALTER TABLE users ADD COLUMN preferences TEXT");
}
/* The text of an imported PDF, extracted by whichever device imported it.
*
* The server never opens a PDF. It has no PDF library and no way to read one:
* the bytes arrive encrypted and are stored that way, and this column arrives
* the same — already extracted, encrypted at rest like everything else, and
* opaque here.
*
* It exists so a PDF is findable on a device that has never opened it. The text
* used to be extracted on first open and kept in that browser's IndexedDB, which
* meant a document imported on the desktop was invisible to a search on the
* phone until the phone had opened it. Doing the work once, where the file
* already is, and letting the result sync is what makes an archive of scanned
* invoices searchable from anywhere.
*
* On `attachments` rather than in a table of its own: it is a property of the
* file, it is deduplicated with the file, and it is freed with the file. */
if (!tableColumns("attachments").includes("text_index")) {
db.exec("ALTER TABLE attachments ADD COLUMN text_index BLOB");
}
/**
* Makes sure exactly one way in exists.
*
@@ -1284,6 +1304,17 @@ const MAX_STROKES = 20000;
function countStrokes(doc) {
if (!doc || typeof doc !== "object") return null;
if (Array.isArray(doc.strokes)) return doc.strokes.length;
/* A note drawn over several pages: page number → strokes. Flatter than the
* PDF's shape below, because every page of a note is the same size and there
* is nothing per page to record besides the ink. */
if (doc.sheets && typeof doc.sheets === "object") {
let n = 0;
for (const sheet of Object.values(doc.sheets)) {
if (!Array.isArray(sheet)) return null;
n += sheet.length;
}
return n;
}
if (doc.pages && typeof doc.pages === "object") {
let n = 0;
for (const page of Object.values(doc.pages)) {
@@ -1530,6 +1561,54 @@ app.get("/api/attachments/:id", authMiddleware, (req, res) => {
res.setHeader("X-Content-Type-Options", "nosniff");
res.end(bytes);
});
/* ─── The text inside a PDF ───
*
* Extracted by the device that has the file, never here. The server has no PDF
* library, and adding one would mean it could read documents it currently only
* stores — which is the opposite of the point. What arrives is a list of page
* texts; what is stored is that list, encrypted, and it is handed back to any
* other device of the same account so it can search a document it has never
* opened.
*/
/* A cap, so one enormous scan cannot fill the database. The client caps its own
* extraction at the same size; this is the second lock, on the side that cannot
* be edited by whoever is holding the browser. */
const MAX_PDF_TEXT_CHARS = 2_000_000;
app.put("/api/attachments/:id/text", authMiddleware, (req, res) => {
const a = db.prepare("SELECT id FROM attachments WHERE id = ? AND user_id = ?").get(req.params.id, req.userId);
if (!a) return res.status(404).json({ error: "Not found" });
const { pages } = req.body || {};
if (!Array.isArray(pages) || pages.some((p) => typeof p !== "string")) {
return res.status(400).json({ error: "pages must be a list of strings" });
}
const total = pages.reduce((n, p) => n + p.length, 0);
if (total > MAX_PDF_TEXT_CHARS) return res.status(413).json({ error: "Extracted text is too large" });
db.prepare("UPDATE attachments SET text_index = ? WHERE id = ? AND user_id = ?")
.run(Buffer.from(encrypt(JSON.stringify({ pages }))), req.params.id, req.userId);
res.json({ ok: true, pages: pages.length });
});
app.get("/api/attachments/:id/text", authMiddleware, (req, res) => {
const a = db
.prepare("SELECT text_index FROM attachments WHERE id = ? AND user_id = ?")
.get(req.params.id, req.userId);
if (!a) return res.status(404).json({ error: "Not found" });
// No text yet is not an error: it means no device has read this file. The
// client answers that by reading it, if it has the bytes.
if (!a.text_index) return res.json({ pages: null });
try {
res.json(JSON.parse(decrypt(a.text_index.toString())));
} catch (err) {
console.error(` ❌ could not decrypt the text of attachment ${req.params.id}: ${err.message}`);
res.json({ pages: null }); // it can be extracted again; a 500 helps nobody
}
});
/* Two routes used to sit here and were removed on 2026-08-08:
*
* POST /api/notes/pdf accepted a PDF and stored it, but nothing could ever
+85 -10
View File
@@ -23,11 +23,13 @@ import {
const GRID_W = 48,
GRID_H = 64;
/** Rasterises the drawing surface into a GRID_W x GRID_H map of "has ink". */
const inkGrid = (page) =>
/** Rasterises one page's drawing surface into a GRID_W x GRID_H map of "has ink". */
const inkGrid = (page, sheet = 1) =>
page.evaluate(
([gw, gh]) => {
const c = document.querySelector("canvas");
([gw, gh, n]) => {
const c =
document.querySelector(`canvas[data-draw-page="${n}"]`) ||
document.querySelector(`[data-drawn-page="${n}"] canvas`);
if (!c) return null;
const off = document.createElement("canvas");
off.width = gw;
@@ -39,18 +41,23 @@ const inkGrid = (page) =>
for (let i = 0; i < gw * gh; i++) out.push(d[i * 4 + 3] > 40 ? 1 : 0);
return out;
},
[GRID_W, GRID_H]
[GRID_W, GRID_H, sheet]
);
const diff = (a, b) => a.reduce((n, v, i) => n + (v === b[i] ? 0 : 1), 0);
const ink = (g) => g.reduce((n, v) => n + v, 0);
async function draw(page) {
await page.locator("button").filter({ hasText: "Draw" }).first().click();
await page.waitForTimeout(1200);
const box = await page.locator("canvas").first().boundingBox();
async function draw(page, sheet = 1, strokes = null) {
if (sheet === 1) {
await page.locator("button").filter({ hasText: "Draw" }).first().click();
await page.waitForTimeout(1200);
}
const canvas = page.locator(`canvas[data-draw-page="${sheet}"]`);
await canvas.scrollIntoViewIfNeeded();
await page.waitForTimeout(500);
const box = await canvas.boundingBox();
// Three strokes in different parts of the page, so a shifted origin shows up.
const strokes = [
strokes ??= [
[0.2, 0.2, 0.7, 0.35],
[0.25, 0.55, 0.75, 0.5],
[0.5, 0.7, 0.55, 0.85],
@@ -130,6 +137,74 @@ try {
const d = diff(reference, afterReload);
check(d / (GRID_W * GRID_H) < 0.02, "the reloaded drawing matches the original", `${d} cells`);
}
/* ── a drawing runs over as many pages as the note has ──
* The ink used to be stuck on page one while the writing carried on without
* it. Every page is exported anyway, so every page can be drawn on. */
await createNote(p);
await setTitle(p, "Mehrere Seiten");
await p.locator("button").filter({ hasText: "Draw" }).first().click();
await p.waitForTimeout(1200);
check(
(await p.locator('canvas[data-draw-page="2"]').count()) === 1,
"there is a blank page past the end to draw on, before anything is written"
);
await draw(p, 1);
const first = await inkGrid(p, 1);
await draw(p, 2, [
[0.3, 0.25, 0.8, 0.4],
[0.2, 0.6, 0.6, 0.75],
]);
await p.waitForTimeout(2500);
const second = await inkGrid(p, 2);
check(ink(second) > 0, "a stroke lands on page two", `${ink(second)} ink cells`);
check(diff(first, await inkGrid(p, 1)) === 0, "and page one is untouched by it");
check(
(await p.locator("[data-note-position]").innerText()).includes("of 2"),
"drawing on page two makes the note two pages",
await p.locator("[data-note-position]").innerText()
);
check(
(await p.locator('canvas[data-draw-page="3"]').count()) === 1,
"and there is a blank page after it again"
);
await p.screenshot({ path: `${OUT}/drawing-pages.png` });
/* Stored per page, and it survives the trip to the server and back. */
const pageDoc = await p.evaluate(async () => {
const auth = { Authorization: "Bearer " + localStorage.getItem("inknote_token") };
const notes = await (await fetch("/api/notes", { headers: auth })).json();
const mine = notes.find((n) => n.title === "Mehrere Seiten");
return (await (await fetch(`/api/notes/${mine.id}`, { headers: auth })).json()).drawingStrokes;
});
check(!!pageDoc?.sheets, "the server stored it as pages, not as one sheet", JSON.stringify(Object.keys(pageDoc || {})));
check(
(pageDoc?.sheets?.["1"]?.length ?? 0) > 0 && (pageDoc?.sheets?.["2"]?.length ?? 0) > 0,
"with strokes on both",
`page 1: ${pageDoc?.sheets?.["1"]?.length}, page 2: ${pageDoc?.sheets?.["2"]?.length}`
);
await reopenNote(p, "Mehrere Seiten");
await p.waitForTimeout(2000);
const reloadedTwo = await inkGrid(p, 2);
check(reloadedTwo !== null && ink(reloadedTwo) > 0, "page two's ink is still there after a reload");
check(
diff(second, reloadedTwo) / (GRID_W * GRID_H) < 0.02,
"and it is the same ink",
`${diff(second, reloadedTwo)} cells`
);
/* Clearing acts on the page you are looking at, not on the whole drawing. */
await p.locator("button").filter({ hasText: "Draw" }).first().click();
await p.waitForTimeout(1200);
await p.locator('canvas[data-draw-page="1"]').scrollIntoViewIfNeeded();
await p.waitForTimeout(900);
await p.locator("[data-clear-page]").click();
await p.waitForTimeout(2500);
check(ink(await inkGrid(p, 1)) === 0, "the bin empties the page being looked at");
check(ink(await inkGrid(p, 2)) > 0, "and leaves the other pages alone");
} catch (e) {
check(false, "suite ran to the end", e.message);
} finally {
+21 -1
View File
@@ -9,7 +9,7 @@
* A second browser context is a second device: its own storage, its own cache,
* its own session.
*/
import { newPage, signIn, check, createNotebook, launch, report, withFreshUser } from "./lib.mjs";
import { newPage, signIn, check, createNotebook, launch, report, search, withFreshUser } from "./lib.mjs";
import { writeFileSync } from "node:fs";
const PDF = "/tmp/sync-test.pdf";
@@ -71,6 +71,26 @@ try {
await signIn(phone, s.user, s.pass);
await phone.locator("button").filter({ hasText: "Sync" }).first().click();
await phone.waitForTimeout(1500);
/* ── found by what is written inside it, without ever opening it ──
* The text is read on the device that imports the file and stored with it, so
* this device can search a document whose bytes it has not even downloaded.
* It used to be extracted on first open and kept in that one browser, which
* made a PDF filed on the desktop invisible to a search on the phone. */
await phone.waitForTimeout(8000); // the backfill runs after the notes are in
await search(phone, "Handbuch Seite");
check(
(await phone.locator("[data-note-row]").count()) === 1,
"the second device finds the PDF by a phrase inside it, having never opened it",
`${await phone.locator("[data-note-row]").count()} rows`
);
await search(phone, "Wortdasnirgendwostehtxyz");
check(
(await phone.locator("[data-note-row]").count()) === 0,
"and finds nothing for a phrase that is not in it"
);
await search(phone, "");
await phone.locator("[data-note-row]").first().click();
await phone.waitForTimeout(6000);
+12 -3
View File
@@ -33,7 +33,13 @@ import {
import { EditorPane } from "./features/notes/EditorPane.jsx";
import { HistoryModal } from "./features/notes/HistoryModal.jsx";
import { DrawingLayer } from "./features/drawing/DrawingLayer.jsx";
import { CANON_W, CANON_H, PREVIEW_DRAWING_W } from "./features/drawing/strokes.js";
import {
CANON_W,
CANON_H,
PREVIEW_DRAWING_W,
drawnSheets,
sheetStrokes,
} from "./features/drawing/strokes.js";
/** Views that are not a notebook and therefore cannot hold a new note. */
const VIRTUAL_FOLDERS = ["all", "starred", "trash", "dashboard"];
@@ -760,9 +766,12 @@ export default function App() {
data.setNoteContent(historyFor, content);
setRestoreEpoch((e) => e + 1);
}}
renderDrawing={(strokes) => (
/* The first drawn page stands for the drawing in a thumbnail this
small. Asking for page one would show a blank sheet for a note whose
ink starts on page three. */
renderDrawing={(doc) => (
<DrawingLayer
strokes={strokes.strokes || []}
strokes={sheetStrokes(doc, drawnSheets(doc)[0] ?? 1)}
width={PREVIEW_DRAWING_W}
height={Math.round((PREVIEW_DRAWING_W * CANON_H) / CANON_W)}
zIndex={1}
+56 -19
View File
@@ -10,7 +10,15 @@
import * as sync from "../syncEngine";
import { parseNoteContent } from "../editor/content";
import { parsingExtensions } from "../editor/extensions.js";
import { CANON_W, CANON_H, docStrokes, paintStrokeDoc, sizeCanvas } from "../features/drawing/strokes.js";
import {
CANON_W,
CANON_H,
allStrokes,
drawnSheets,
sheetStrokes,
paintStrokeDoc,
sizeCanvas,
} from "../features/drawing/strokes.js";
import type { NoteContent, NoteMeta, Notebook } from "../types";
import { noteToMarkdown, safeFilename } from "./markdown";
import { asPdfStrokeDoc, hasOverlay } from "../features/pdf/pageStrokes";
@@ -31,18 +39,18 @@ type Loaded = { meta: NoteMeta; content: NoteContent };
/* ── pieces ── */
/** Renders a stroke document to PNG bytes, so a drawing survives as a picture. */
async function drawingToPng(doc: unknown): Promise<Uint8Array | null> {
/** Renders one page of a stroke document to PNG bytes. */
async function drawingToPng(doc: unknown, page = 1): Promise<Uint8Array | null> {
// `doc` is already a stroke document as stored — strokeDocOf() builds one from
// a bare array and would wrap this one a second time, hiding the strokes.
if (docStrokes(doc).length === 0) return null;
if (sheetStrokes(doc, page).length === 0) return null;
const canvas = document.createElement("canvas");
const height = Math.round((DRAWING_EXPORT_W * CANON_H) / CANON_W);
const { bw, bh } = sizeCanvas(canvas, DRAWING_EXPORT_W, height);
const ctx = canvas.getContext("2d");
if (!ctx) return null;
paintStrokeDoc(ctx, doc, bw, bh);
paintStrokeDoc(ctx, doc, bw, bh, page);
// Strokes are drawn on nothing, which would export as a transparent PNG and
// leave black ink invisible in a dark image viewer. The paper goes in behind
// afterwards, because paintStrokeDoc begins by clearing the canvas.
@@ -57,6 +65,22 @@ async function drawingToPng(doc: unknown): Promise<Uint8Array | null> {
return new Uint8Array(await blob.arrayBuffer());
}
/**
* Every drawn page of a note, as PNGs with the name each should be filed under.
*
* Page one keeps the name a drawing has always had, so an archive written before
* notes could be drawn on more than one page still reads the same. The rest are
* numbered after it.
*/
async function drawingPages(doc: unknown, base: string): Promise<Array<{ name: string; data: Uint8Array }>> {
const out: Array<{ name: string; data: Uint8Array }> = [];
for (const page of drawnSheets(doc)) {
const data = await drawingToPng(doc, page);
if (data) out.push({ name: page === 1 ? `${base}.png` : `${base}-${page}.png`, data });
}
return out;
}
/** Every attachment id the document refers to. */
function attachmentIds(doc: unknown): string[] {
const ids = new Set<string>();
@@ -133,24 +157,35 @@ export type ExportParts = "text" | "drawing" | "both";
export async function noteHasDrawing(meta: NoteMeta): Promise<boolean> {
try {
const { content } = await loadNote(meta);
return docStrokes(content.drawingStrokes).length > 0;
return allStrokes(content.drawingStrokes).length > 0;
} catch {
return false;
}
}
/** The drawing on its own, as a PNG on white paper. */
/**
* The drawing on its own, on white paper.
*
* One page is a PNG, as it always was. Several are a ZIP: a drawing spread over
* four pages is four pictures, and silently exporting only the first would lose
* the rest without saying so.
*/
export async function exportDrawingAsPng(meta: NoteMeta): Promise<void> {
const { content } = await loadNote(meta);
const png = await drawingToPng(content.drawingStrokes);
if (!png) throw new Error("This note has no drawing.");
download(bytesToBlob(png, "image/png"), `${safeFilename(meta.title)}.png`);
const name = safeFilename(meta.title);
const pages = await drawingPages(content.drawingStrokes, name);
if (pages.length === 0) throw new Error("This note has no drawing.");
if (pages.length === 1) {
download(bytesToBlob(pages[0]!.data, "image/png"), `${name}.png`);
return;
}
download(createZip(pages.map((p) => ({ name: p.name, data: p.data }))), `${name}.zip`);
}
export async function exportNoteAsMarkdown(meta: NoteMeta): Promise<void> {
const { content } = await loadNote(meta);
const doc = parseNoteContent(content.content, parsingExtensions);
const hasDrawing = docStrokes(content.drawingStrokes).length > 0;
const hasDrawing = allStrokes(content.drawingStrokes).length > 0;
const ids = attachmentIds(doc);
const md = noteToMarkdown(doc, {
@@ -173,8 +208,7 @@ export async function exportNoteAsMarkdown(meta: NoteMeta): Promise<void> {
const bytes = await fetchAttachment(id);
if (bytes) entries.push({ name: `${folder}-images/${id}.png`, data: bytes });
}
const png = await drawingToPng(content.drawingStrokes);
if (png) entries.push({ name: `${folder}-drawing.png`, data: png });
entries.push(...(await drawingPages(content.drawingStrokes, `${folder}-drawing`)));
download(createZip(entries), `${folder}.zip`);
}
@@ -220,10 +254,14 @@ export async function exportNotesAsPdf(
}
}
const png = parts === "text" ? null : await drawingToPng(content.drawingStrokes);
const drawing = png ? await blobToDataUrl(bytesToBlob(png, "image/png")) : null;
const drawings: string[] = [];
if (parts !== "text") {
for (const page of await drawingPages(content.drawingStrokes, "page")) {
drawings.push(await blobToDataUrl(bytesToBlob(page.data, "image/png")));
}
}
printable.push({ meta, content, drawing, images, includeText: parts !== "drawing" });
printable.push({ meta, content, drawings, images, includeText: parts !== "drawing" });
} catch {
/* an unreadable note is left out rather than aborting the whole print */
}
@@ -292,7 +330,7 @@ export async function exportNotesAsZip({
try {
const { content } = await loadNote(meta);
const doc = parseNoteContent(content.content, parsingExtensions);
const hasDrawing = docStrokes(content.drawingStrokes).length > 0;
const hasDrawing = allStrokes(content.drawingStrokes).length > 0;
entries.push({
name: `${name}.md`,
@@ -310,8 +348,7 @@ export async function exportNotesAsZip({
const bytes = await fetchAttachment(id);
if (bytes) entries.push({ name: `images/${id}.png`, data: bytes });
}
const png = await drawingToPng(content.drawingStrokes);
if (png) entries.push({ name: `drawings/${name}.png`, data: png });
entries.push(...(await drawingPages(content.drawingStrokes, `drawings/${name}`)));
} catch (err) {
// A note that cannot be read is named in the index rather than skipped in
// silence — an export that quietly drops things is worse than useless.
+5 -3
View File
@@ -145,8 +145,8 @@ const STYLES = `
export interface PrintableNote {
meta: NoteMeta;
content: NoteContent;
/** Drawing rendered as a data URL, if the note has one and it was asked for. */
drawing?: string | null;
/** The drawing's pages as data URLs, in order, if it was asked for. */
drawings?: string[] | null;
/** Attachment id → data URL, for images to survive into the print document. */
images?: Map<string, string>;
/** Omitted for a drawing-only export, where the text would be in the way. */
@@ -165,7 +165,9 @@ function articleFor(note: PrintableNote): string {
<h1>${esc(note.meta.title || "Untitled")}</h1>
${when ? `<div class="meta">${esc(when)}</div>` : ""}
${body}
${note.drawing ? `<div class="drawing"><img src="${note.drawing}" alt="Drawing"></div>` : ""}
${(note.drawings ?? [])
.map((src, i) => `<div class="drawing"><img src="${src}" alt="Drawing, page ${i + 1}"></div>`)
.join("\n ")}
</article>`;
}
+94 -7
View File
@@ -10,6 +10,15 @@ export const CANON_W = 1200,
CANON_H = 1600;
export const PREVIEW_DRAWING_W = 260; // history preview thumbnail
export const STROKES_VERSION = 1;
/* A drawing spread over more than one page.
*
* A note used to have exactly one sheet to draw on, which stopped making sense
* once the writing could run to four pages: the ink was stuck on the first one
* while the text carried on without it. Pages are numbered the way the PDF
* reader numbers them, and each keeps its own strokes in the same canonical
* space — so page 3 of a note means the same thing on a phone as on a desktop.
*/
export const SHEETS_VERSION = 2;
// Widths are canonical units, so a "Medium" pen covers the same fraction of the
// page everywhere. At the 760px desktop page width, M lands at roughly 3.5 CSS px.
@@ -41,10 +50,86 @@ export const strokeDocOf = (strokes, t) => ({
strokes,
...(t ? { t } : {}),
});
export const docStrokes = (doc) => (Array.isArray(doc?.strokes) ? doc.strokes : []);
export const sameStrokeDoc = (a, b) =>
a === b ||
(docStrokes(a).length === docStrokes(b).length && docStrokes(a).every((s, i) => s === docStrokes(b)[i]));
/** The strokes on one canvas — page one, for a drawing that has several. */
export const docStrokes = (doc) => sheetStrokes(doc, 1);
/**
* A drawing's pages, keyed by 1-based page number.
*
* Reads either shape, so a note drawn before there were pages needs no
* migration: its single list of strokes is page one, which is where it was.
* Empty pages are dropped rather than stored, so "which pages are drawn on" is
* simply the keys.
*/
export function noteSheets(doc) {
if (!doc || typeof doc !== "object") return {};
const source =
doc.sheets && typeof doc.sheets === "object"
? doc.sheets
: Array.isArray(doc.strokes)
? { 1: doc.strokes }
: {};
const out = {};
for (const [key, strokes] of Object.entries(source)) {
const page = Number(key);
if (!Number.isInteger(page) || page < 1) continue;
if (!Array.isArray(strokes) || strokes.length === 0) continue;
out[String(page)] = strokes;
}
return out;
}
export const sheetStrokes = (doc, page) => noteSheets(doc)[String(page)] ?? [];
/** Page numbers carrying ink, in reading order. */
export const drawnSheets = (doc) =>
Object.keys(noteSheets(doc))
.map(Number)
.sort((a, b) => a - b);
/** Every stroke in the drawing — for "is there anything here at all". */
export const allStrokes = (doc) => {
const sheets = noteSheets(doc);
return drawnSheets(doc).flatMap((page) => sheets[String(page)]);
};
/**
* Builds a stroke document from pages.
*
* A drawing that lives entirely on page one is written in the old single-page
* shape. Nothing needs it to be — but it means an ordinary note's drawing is
* stored exactly as it always was, and a client that has not been updated yet
* still shows it.
*/
export function sheetDocOf(sheets, t) {
const clean = noteSheets({ sheets });
const pages = Object.keys(clean);
if (pages.length === 0 || (pages.length === 1 && pages[0] === "1")) {
return strokeDocOf(clean["1"] ?? [], t);
}
return { v: SHEETS_VERSION, w: CANON_W, h: CANON_H, sheets: clean, ...(t ? { t } : {}) };
}
/** The pages of `doc` with one page replaced — an empty page is removed. */
export function sheetsWith(doc, page, strokes) {
const sheets = { ...noteSheets(doc) };
if (strokes?.length) sheets[String(page)] = strokes;
else delete sheets[String(page)];
return sheets;
}
/* Compared page by page. Flattening first would call a stroke moved from page
* two to page three "unchanged", and the move would never be saved. */
export const sameStrokeDoc = (a, b) => {
if (a === b) return true;
const x = noteSheets(a),
y = noteSheets(b);
const keys = Object.keys(x);
if (keys.length !== Object.keys(y).length) return false;
return keys.every(
(k) => y[k] && x[k].length === y[k].length && x[k].every((s, i) => s === y[k][i])
);
};
export function paintStroke(ctx, s) {
const p = s?.p;
@@ -77,8 +162,10 @@ export function paintStroke(ctx, s) {
ctx.globalCompositeOperation = "source-over";
}
// Replays a whole drawing into a context of targetW × targetH device pixels.
export function paintStrokeDoc(ctx, doc, targetW, targetH) {
// Replays one page of a drawing into a context of targetW × targetH device
// pixels. Single-page drawings are page one, which is what every caller that
// never heard of pages asks for.
export function paintStrokeDoc(ctx, doc, targetW, targetH, page = 1) {
const w = doc?.w || CANON_W,
h = doc?.h || CANON_H;
ctx.setTransform(1, 0, 0, 1, 0, 0);
@@ -86,7 +173,7 @@ export function paintStrokeDoc(ctx, doc, targetW, targetH) {
ctx.scale(targetW / w, targetH / h);
ctx.lineCap = "round";
ctx.lineJoin = "round";
for (const s of docStrokes(doc)) paintStroke(ctx, s);
for (const s of sheetStrokes(doc, page)) paintStroke(ctx, s);
ctx.setTransform(1, 0, 0, 1, 0, 0);
}
@@ -0,0 +1,141 @@
import { describe, it, expect } from "vitest";
import {
CANON_W,
CANON_H,
STROKES_VERSION,
SHEETS_VERSION,
allStrokes,
docStrokes,
drawnSheets,
noteSheets,
sameStrokeDoc,
sheetDocOf,
sheetStrokes,
sheetsWith,
strokeDocOf,
} from "./strokes.js";
/* `strokes.js` is untyped, so what comes back is a union TypeScript cannot see
* through. These two keep the tests about behaviour rather than about casts. */
interface Stroke {
label: string;
}
interface Doc {
v: number;
t?: string;
strokes?: Stroke[];
sheets?: Record<string, Stroke[]>;
}
const stroke = (label: string): Stroke =>
({ t: "pen", c: "#000", s: 3, p: [0, 0, 1, 1], label }) as Stroke;
const asDoc = (doc: unknown) => doc as Doc;
describe("reading a drawing", () => {
it("treats a note drawn before there were pages as page one", () => {
const a = stroke("a");
const doc = strokeDocOf([a]);
expect(noteSheets(doc)).toEqual({ 1: [a] });
expect(sheetStrokes(doc, 1)).toEqual([a]);
expect(sheetStrokes(doc, 2)).toEqual([]);
});
it("reads a drawing kept per page", () => {
const doc = { v: SHEETS_VERSION, w: CANON_W, h: CANON_H, sheets: { 1: [stroke("a")], 3: [stroke("c")] } };
expect(drawnSheets(doc)).toEqual([1, 3]);
expect(sheetStrokes(doc, 3)).toEqual([stroke("c")]);
});
it("has no pages at all for nothing, an empty list or rubbish", () => {
for (const doc of [null, undefined, {}, { strokes: [] }, { sheets: {} }, 7, "x"]) {
expect(drawnSheets(doc)).toEqual([]);
expect(allStrokes(doc)).toEqual([]);
}
});
it("ignores pages that are not pages", () => {
const good = stroke("good");
const doc = { sheets: { 1: [good], 0: [stroke("zero")], "-2": [stroke("neg")], x: [stroke("x")], 2: "no" } };
expect(drawnSheets(doc)).toEqual([1]);
expect(allStrokes(doc)).toEqual([good]);
});
it("puts every stroke in reading order, page ten after page nine", () => {
const doc = { sheets: { 10: [stroke("j")], 9: [stroke("i")], 1: [stroke("a")] } };
expect(allStrokes(doc).map((s: Stroke) => s.label)).toEqual(["a", "i", "j"]);
});
it("still answers page one for callers that never heard of pages", () => {
const a = stroke("a");
expect(docStrokes({ sheets: { 1: [a], 2: [stroke("b")] } })).toEqual([a]);
});
});
describe("writing a drawing", () => {
it("keeps the old single-page shape when everything is on page one", () => {
const doc = asDoc(sheetDocOf({ 1: [stroke("a")] }));
expect(doc.v).toBe(STROKES_VERSION);
expect(doc.strokes).toHaveLength(1);
expect(doc.sheets).toBeUndefined();
});
it("writes pages once there is more than the first", () => {
const doc = asDoc(sheetDocOf({ 1: [stroke("a")], 2: [stroke("b")] }));
expect(doc.v).toBe(SHEETS_VERSION);
expect(Object.keys(doc.sheets!)).toEqual(["1", "2"]);
expect(doc.strokes).toBeUndefined();
});
it("writes pages for ink that starts on page two", () => {
const doc = asDoc(sheetDocOf({ 2: [stroke("b")] }));
expect(doc.v).toBe(SHEETS_VERSION);
expect(sheetStrokes(doc, 2)).toHaveLength(1);
});
it("is an empty single-page drawing when nothing is drawn", () => {
const doc = asDoc(sheetDocOf({}));
expect(doc.strokes).toEqual([]);
expect(allStrokes(doc)).toEqual([]);
});
it("carries the save token through either shape", () => {
expect(asDoc(sheetDocOf({ 1: [stroke("a")] }, "tok")).t).toBe("tok");
expect(asDoc(sheetDocOf({ 1: [stroke("a")], 2: [stroke("b")] }, "tok")).t).toBe("tok");
});
it("drops a page that has been emptied rather than storing it", () => {
const doc = { sheets: { 1: [stroke("a")], 2: [stroke("b")] } };
expect(drawnSheets({ sheets: sheetsWith(doc, 2, []) })).toEqual([1]);
expect(drawnSheets({ sheets: sheetsWith(doc, 1, []) })).toEqual([2]);
});
it("replaces one page and leaves the others alone", () => {
const doc = { sheets: { 1: [stroke("a")], 2: [stroke("b")] } };
const next = sheetsWith(doc, 2, [stroke("c"), stroke("d")]) as Record<string, Stroke[]>;
expect(next["1"]).toEqual([stroke("a")]);
expect(next["2"]).toHaveLength(2);
});
});
describe("noticing a change", () => {
it("calls a drawing the same as itself", () => {
const doc = sheetDocOf({ 1: [stroke("a")] });
expect(sameStrokeDoc(doc, doc)).toBe(true);
});
it("sees the same strokes through either shape", () => {
const a = stroke("a");
expect(sameStrokeDoc(strokeDocOf([a]), sheetDocOf({ 1: [a] }))).toBe(true);
});
it("sees a stroke moved to another page", () => {
const a = stroke("a");
// Flattening first would call this unchanged, and the move would never save.
expect(sameStrokeDoc(sheetDocOf({ 2: [a] }), sheetDocOf({ 3: [a] }))).toBe(false);
});
it("sees a page added and a page emptied", () => {
const one = sheetDocOf({ 1: [stroke("a")] });
expect(sameStrokeDoc(one, sheetDocOf({ 1: [stroke("a")], 2: [stroke("b")] }))).toBe(false);
expect(sameStrokeDoc(one, sheetDocOf({}))).toBe(false);
});
});
+149 -75
View File
@@ -14,7 +14,11 @@ import {
MIN_POINT_DIST,
newSaveToken,
strokeDocOf,
docStrokes,
sheetDocOf,
noteSheets,
sheetStrokes,
sheetsWith,
drawnSheets,
paintStroke,
paintStrokeDoc,
sizeCanvas,
@@ -36,7 +40,6 @@ export function NoteEditor({
backlinks,
}) {
const [mode, setMode] = useState("text");
const canvasRef = useRef(null);
const [tool, setTool] = useState("pen");
const [penColor, setPenColor] = useState(dark ? "#e6edf3" : "#1a1a2e");
const [penSizeIdx, setPenSizeIdx] = useState(DEFAULT_PEN_SIZE);
@@ -63,16 +66,20 @@ export function NoteEditor({
const scrollerRef = useRef(null);
/* ── DRAWING STATE ──
* strokesRef is what the pointer handlers read and write synchronously;
* `strokes` mirrors it so React can repaint the read-only layer. */
const strokesRef = useRef([]);
const [strokes, setStrokes] = useState([]);
* sheetsRef is what the pointer handlers read and write synchronously;
* `sheets` mirrors it so React can repaint the read-only layers. Both are the
* drawing's pages: page number → strokes, with drawn-on pages only. */
const sheetsRef = useRef({});
const [sheets, setSheets] = useState({});
const liveRef = useRef(null); // stroke currently under the pointer
const drawingRef = useRef(false);
const committedRef = useRef(null); // offscreen canvas holding all finished strokes
const drawPageRef = useRef(1); // the page that stroke is being drawn on
const canvasesRef = useRef(new Map()); // page number → its canvas
const committedRef = useRef(null); // offscreen copy of the page being drawn on
const appliedDocRef = useRef(undefined);
const pendingTokenRef = useRef(null);
const hasStrokes = strokes.length > 0;
const drawnPages = useMemo(() => drawnSheets({ sheets }), [sheets]);
const hasStrokes = drawnPages.length > 0;
// Old notes may still carry a raster drawing. It stays visible underneath any
// new vector strokes so migrating a note never loses what was already drawn.
const legacyRaster = note.drawingData && !hideDrawing;
@@ -86,10 +93,11 @@ export function NoteEditor({
* be — and plain text is simply text. */
const needsPage = mode === "draw" || hasStrokes || !!legacyRaster;
const commitStrokes = (next) => {
strokesRef.current = next;
setStrokes(next);
const commitSheets = (next) => {
sheetsRef.current = next;
setSheets(next);
};
const pageStrokes = (page) => sheetsRef.current[String(page)] ?? [];
const T = useMemo(
() => ({
@@ -198,7 +206,7 @@ export function NoteEditor({
const saveDrawingNow = () => {
if (!dirtyRef.current) return;
const token = newSaveToken();
const doc = strokeDocOf(strokesRef.current, token);
const doc = sheetDocOf(sheetsRef.current, token);
appliedDocRef.current = doc; // ignore the echo coming straight back through props
pendingTokenRef.current = token; // ...and any server read still in flight
dirtyRef.current = false;
@@ -223,7 +231,11 @@ export function NoteEditor({
if (!el || !pageH) return;
const height = el.scrollHeight;
const scroller = scrollerRef.current;
const count = Math.max(1, Math.ceil(height / pageH));
/* Drawn pages count too. Ink put on page four of a two-page note is on the
* note, and a count that ignored it would report two pages while showing
* four — and the fourth would have no canvas to be redrawn on. */
const written = Math.max(1, Math.ceil(height / pageH));
const count = Math.max(written, drawnPages[drawnPages.length - 1] ?? 1);
/* The page under the middle of the viewport is the one you are reading —
* the same rule the PDF reader uses. Measuring from the top instead said
* "page 1" while you were looking at the foot of a two-page note, because
@@ -244,11 +256,14 @@ export function NoteEditor({
scroller?.removeEventListener("scroll", recount);
textEditor.off("transaction", recount);
};
}, [textEditor, stageSize.h, mode]);
}, [textEditor, stageSize.h, mode, drawnPages]);
/* ── DRAWING: painting ── */
/* ── DRAWING: painting ──
* Only the page under the pointer is repainted while a stroke is being drawn;
* the others cannot have changed, and repainting four canvases per pointer
* move on a phone is felt. */
const renderFrame = () => {
const c = canvasRef.current;
const c = canvasesRef.current.get(drawPageRef.current);
if (!c) return;
const ctx = c.getContext("2d");
ctx.setTransform(1, 0, 0, 1, 0, 0);
@@ -264,23 +279,30 @@ export function NoteEditor({
ctx.restore();
}
};
// Repaint the offscreen canvas from scratch, then show it. Cheap enough at
// stroke boundaries, and it keeps eraser strokes ordered correctly.
const repaintAll = () => {
const c = canvasRef.current;
// Repaint one page's offscreen canvas from scratch, then show it. Cheap enough
// at stroke boundaries, and it keeps eraser strokes ordered correctly.
const repaintPage = (page) => {
const c = canvasesRef.current.get(page);
if (!c) return;
const { bw, bh } = sizeCanvas(c, stageSize.w, stageSize.h);
let off = committedRef.current;
if (!off) {
off = document.createElement("canvas");
committedRef.current = off;
if (page === drawPageRef.current) {
let off = committedRef.current;
if (!off) {
off = document.createElement("canvas");
committedRef.current = off;
}
if (off.width !== bw || off.height !== bh) {
off.width = bw;
off.height = bh;
}
paintStrokeDoc(off.getContext("2d"), strokeDocOf(pageStrokes(page)), bw, bh);
renderFrame();
return;
}
if (off.width !== bw || off.height !== bh) {
off.width = bw;
off.height = bh;
}
paintStrokeDoc(off.getContext("2d"), strokeDocOf(strokesRef.current), bw, bh);
renderFrame();
paintStrokeDoc(c.getContext("2d"), strokeDocOf(pageStrokes(page)), bw, bh);
};
const repaintAll = () => {
for (const page of canvasesRef.current.keys()) repaintPage(page);
};
/* Adopt strokes arriving from props (note switch, or a sync push from another
@@ -297,7 +319,7 @@ export function NoteEditor({
pendingTokenRef.current = null; // our own save echoed back; in sync again
return;
}
commitStrokes(docStrokes(doc));
commitSheets(noteSheets(doc));
setUndoStack([]);
setRedoStack([]);
repaintAll();
@@ -305,11 +327,15 @@ export function NoteEditor({
useEffect(() => {
if (mode === "draw") repaintAll();
}, [mode, stageSize.w, stageSize.h]);
}, [mode, stageSize.w, stageSize.h, pages.count]);
/* ── DRAWING: input ── */
const canonPos = (e) => {
const r = canvasRef.current.getBoundingClientRect();
/* ── DRAWING: input ──
* Coordinates are canonical *within the page drawn on*, so a stroke at the top
* of page three is at the top of page three on every other screen too. */
const canonPos = (e, page) => {
const c = canvasesRef.current.get(page);
if (!c) return { x: 0, y: 0 };
const r = c.getBoundingClientRect();
const t = e.touches ? e.touches[0] : e;
return { x: (t.clientX - r.left) * (CANON_W / r.width), y: (t.clientY - r.top) * (CANON_H / r.height) };
};
@@ -321,10 +347,18 @@ export function NoteEditor({
};
const r1 = (n) => Math.round(n * 10) / 10;
const startDraw = (e) => {
const startDraw = (page) => (e) => {
if (mode !== "draw") return;
e.preventDefault();
const p = canonPos(e);
// The offscreen copy belongs to whichever page is being drawn on, so it has
// to be rebuilt before the first frame — otherwise the previous page's ink
// would appear on this one.
if (drawPageRef.current !== page) {
drawPageRef.current = page;
committedRef.current = null;
repaintPage(page);
}
const p = canonPos(e, page);
liveRef.current = { t: tool, c: penColor, s: r1(currentStrokeWidth()), p: [r1(p.x), r1(p.y)] };
drawingRef.current = true;
renderFrame();
@@ -332,7 +366,7 @@ export function NoteEditor({
const doDraw = (e) => {
if (!drawingRef.current) return;
e.preventDefault();
const p = canonPos(e),
const p = canonPos(e, drawPageRef.current),
live = liveRef.current,
n = live.p.length;
const dx = p.x - live.p[n - 2],
@@ -351,10 +385,11 @@ export function NoteEditor({
renderFrame();
return;
}
const page = drawPageRef.current;
pushUndo();
commitStrokes([...strokesRef.current, live]);
commitSheets(sheetsWith({ sheets: sheetsRef.current }, page, [...pageStrokes(page), live]));
dirtyRef.current = true;
repaintAll();
repaintPage(page);
scheduleDrawingSave();
};
@@ -388,12 +423,15 @@ export function NoteEditor({
setTableMenu({ x: e.clientX, y: e.clientY, items });
};
/* History holds whole drawings, not single pages. Undo after a stroke on page
* three has to be able to put page three back exactly as it was, and a stack
* kept per page would also make one Undo button mean four different things. */
const pushUndo = () => {
setUndoStack((p) => [...p.slice(-30), strokesRef.current]);
setUndoStack((p) => [...p.slice(-30), sheetsRef.current]);
setRedoStack([]);
};
const applyHistory = (next) => {
commitStrokes(next);
commitSheets(next);
dirtyRef.current = true;
repaintAll();
scheduleDrawingSave();
@@ -402,19 +440,21 @@ export function NoteEditor({
if (!undoStack.length) return;
const prev = undoStack[undoStack.length - 1];
setUndoStack((p) => p.slice(0, -1));
setRedoStack((p) => [...p, strokesRef.current]);
setRedoStack((p) => [...p, sheetsRef.current]);
applyHistory(prev);
};
const redo = () => {
if (!redoStack.length) return;
const next = redoStack[redoStack.length - 1];
setRedoStack((p) => p.slice(0, -1));
setUndoStack((p) => [...p, strokesRef.current]);
setUndoStack((p) => [...p, sheetsRef.current]);
applyHistory(next);
};
/* Clears the page being looked at, not the whole drawing. Emptying four pages
* because the bin was pressed on one of them is not what anyone means by it. */
const clearCanvas = () => {
pushUndo();
applyHistory([]);
applyHistory(sheetsWith({ sheets: sheetsRef.current }, pages.current, []));
};
/* Save drawing on mode switch and debounced stroke-end updates. */
@@ -482,6 +522,18 @@ export function NoteEditor({
</button>
);
/* While drawing there is always one blank page past the end.
*
* Pages otherwise come from how far the writing runs, which would leave a
* drawing with nowhere to go: to draw on page three you would first have to
* write three pages. The spare page costs an empty canvas and becomes a real
* page the moment something is drawn on it. */
const canvasPages = mode === "draw" ? pages.count + 1 : pages.count;
const registerCanvas = (page) => (el) => {
if (el) canvasesRef.current.set(page, el);
else canvasesRef.current.delete(page);
};
return (
<div
style={{
@@ -678,7 +730,12 @@ export function NoteEditor({
)}
</div>
<div style={{ flex: 1 }} />
<button style={{ ...tb(), color: "#e5534b" }} onClick={clearCanvas}>
<button
data-clear-page
title={`Clear page ${pages.current}`}
style={{ ...tb(), color: "#e5534b" }}
onClick={clearCanvas}
>
{I.trash}
</button>
</>
@@ -724,7 +781,7 @@ export function NoteEditor({
// The text caps its own width in the stylesheet, so this is only
// the page when there is one.
width: needsPage ? stageSize.w : "100%",
minHeight: needsPage ? stageSize.h : undefined,
minHeight: needsPage ? canvasPages * stageSize.h : undefined,
margin: "0 auto",
background: needsPage ? T.bg : "transparent",
border: needsPage && !isMobileViewport ? `1px solid ${T.border}` : "none",
@@ -735,7 +792,7 @@ export function NoteEditor({
{/* A hairline where each page ends, so the count in the toolbar
corresponds to something you can see. Drawn behind the text and
deaf to the pointer — it marks the paper, it is not part of it. */}
{pages.count > 1 && mode === "text" && (
{canvasPages > 1 && (
<div
aria-hidden
style={{
@@ -784,33 +841,50 @@ export function NoteEditor({
}}
/>
)}
{/* Saved drawing — re-rendered from vectors, so identical on every screen */}
{hasStrokes && mode !== "draw" && (
<DrawingLayer strokes={strokes} width={stageSize.w} height={stageSize.h} zIndex={2} />
)}
{mode === "draw" && (
<canvas
ref={canvasRef}
onMouseDown={startDraw}
onMouseMove={doDraw}
onMouseUp={stopDraw}
onMouseLeave={stopDraw}
onTouchStart={startDraw}
onTouchMove={doDraw}
onTouchEnd={stopDraw}
style={{
position: "absolute",
left: 0,
top: 0,
width: "100%",
height: stageSize.h,
touchAction: "none",
cursor: "crosshair",
background: "transparent",
zIndex: 3,
}}
/>
)}
{/* Saved drawing — re-rendered from vectors, so identical on every
screen. One layer per drawn page; a page nobody has drawn on
needs no canvas at all. */}
{mode !== "draw" &&
drawnPages.map((page) => (
<div
key={page}
data-drawn-page={page}
style={{ position: "absolute", left: 0, right: 0, top: (page - 1) * stageSize.h, height: stageSize.h }}
>
<DrawingLayer
strokes={sheetStrokes({ sheets }, page)}
width={stageSize.w}
height={stageSize.h}
zIndex={2}
/>
</div>
))}
{mode === "draw" &&
Array.from({ length: canvasPages }, (_, i) => i + 1).map((page) => (
<canvas
key={page}
ref={registerCanvas(page)}
data-draw-page={page}
onMouseDown={startDraw(page)}
onMouseMove={doDraw}
onMouseUp={stopDraw}
onMouseLeave={stopDraw}
onTouchStart={startDraw(page)}
onTouchMove={doDraw}
onTouchEnd={stopDraw}
style={{
position: "absolute",
left: 0,
top: (page - 1) * stageSize.h,
width: "100%",
height: stageSize.h,
touchAction: "none",
cursor: "crosshair",
background: "transparent",
zIndex: 3,
}}
/>
))}
</div>
</div>
</div>
+116 -11
View File
@@ -5,17 +5,26 @@
* same index that holds the notes — so one search box finds a sentence whether
* it was typed or imported.
*
* **The text is not sent to the server.** It is derivable from a file the server
* already has, so storing it again would cost space and a second thing to
* encrypt for no information gained. The price is that a PDF becomes searchable
* on a device once that device has opened it. That is the same shape as the
* limitation notes already have, and the honest trade for not duplicating data.
* **The reading happens on a device, never on the server.** The server has no
* PDF library and no way to open one: the bytes arrive encrypted and are stored
* that way. The device that imports the file does the work once, and the result
* is stored beside the file — encrypted at rest like everything else — so every
* other device can search a document it has never opened.
*
* It used to be kept only in the browser that had opened the file, on the
* reasoning that text derivable from bytes the server already holds should not
* be stored twice. That was true and useless: a PDF imported on the desktop was
* invisible to a search on the phone. Storing it is what makes an archive of
* scanned invoices findable from anywhere, and it costs a few hundred kilobytes
* against a file that is measured in megabytes.
*/
import * as db from "../../offlineStore";
import * as sync from "../../syncEngine";
import * as searchIndex from "../../search";
import type { NoteMeta } from "../../types";
import type { PDFDocumentProxy } from "./pdfjs";
import { extractPageText } from "./pdfjs";
import { extractPageText, openPdf } from "./pdfjs";
/** Cache key. Keyed by attachment, not note: the same file in two notes is extracted once. */
const keyFor = (attachmentId: string) => `pdftext:${attachmentId}`;
@@ -34,6 +43,92 @@ export async function cachedPdfText(attachmentId: string): Promise<PdfText | nul
return (await db.getMeta<PdfText>(keyFor(attachmentId))) ?? null;
}
const textOf = (pages: string[]): PdfText => ({
pages,
full: pages.join("\n").slice(0, MAX_TEXT_CHARS),
});
const cache = async (attachmentId: string, text: PdfText): Promise<void> => {
try {
await db.setMeta(keyFor(attachmentId), text);
} catch {
/* a full quota costs the cache, not the search */
}
};
/** Stores the text with the file if nothing is stored there yet. */
async function ensureStored(attachmentId: string, pages: string[]): Promise<void> {
try {
if (await sync.fetchPdfText(attachmentId)) return;
await sync.storePdfText(attachmentId, pages);
} catch {
/* offline, or the file is gone — either way, nothing to do about it here */
}
}
/**
* The text of a file, from wherever it can be had cheapest.
*
* Locally first, then from the server — which is where the device that imported
* the file left it. Null means nobody has read this document yet, and whoever
* has the bytes should.
*/
export async function loadPdfText(attachmentId: string): Promise<PdfText | null> {
const local = await cachedPdfText(attachmentId);
if (local) return local;
try {
const pages = await sync.fetchPdfText(attachmentId);
if (!pages) return null;
const text = textOf(pages);
await cache(attachmentId, text);
return text;
} catch {
// Offline, or the file is gone. Either way there is nothing to index and
// nothing worth interrupting the user about.
return null;
}
}
/**
* Reads a freshly imported PDF, so it is searchable everywhere immediately.
*
* Done on the device that has the file open anyway, rather than left until some
* device opens the document — which for an invoice filed and never looked at
* again may be never. Failure is silent on purpose: the import succeeded, and
* the text can still be extracted the next time the file is opened.
*/
export async function extractOnImport(file: Blob, attachmentId: string): Promise<void> {
try {
const { doc } = await openPdf(new Uint8Array(await file.arrayBuffer()));
await extractAndCache(doc, attachmentId);
} catch (err) {
console.warn(`[pdf] could not read the text of ${attachmentId} on import:`, err);
}
}
/**
* Fills the search index with the text of every PDF the account holds.
*
* Runs beside the note backfill after login. A document whose text nobody has
* extracted yet is left alone: this device may not have its bytes, and
* downloading every PDF to index it is not a thing to do in the background.
*/
export async function backfillPdfText(
notes: NoteMeta[],
signal?: { aborted: boolean }
): Promise<number> {
let indexed = 0;
for (const note of notes) {
if (signal?.aborted) break;
if (!note.pdfAttachmentId) continue;
const text = await loadPdfText(note.pdfAttachmentId);
if (!text) continue;
indexPdfText(note.id, note.title || "", text);
indexed++;
}
return indexed;
}
/**
* Extracts every page's text and remembers it.
*
@@ -47,7 +142,12 @@ export async function extractAndCache(
signal?: { aborted: boolean }
): Promise<PdfText | null> {
const existing = await cachedPdfText(attachmentId);
if (existing) return existing;
if (existing) {
// Extracted here before the text was ever stored with the file. Sending it
// now is what makes those documents findable on the other devices too.
void ensureStored(attachmentId, existing.pages);
return existing;
}
const pages: string[] = [];
let total = 0;
@@ -66,11 +166,16 @@ export async function extractAndCache(
await new Promise((r) => setTimeout(r, 0));
}
const result: PdfText = { pages, full: pages.join("\n").slice(0, MAX_TEXT_CHARS) };
const result = textOf(pages);
await cache(attachmentId, result);
// Left with the file, so the next device does not have to read it again — and
// can find the document without ever opening it. A failure here is not worth
// reporting: the text is cached locally, and the next device to open the file
// will store it.
try {
await db.setMeta(keyFor(attachmentId), result);
} catch {
/* a full quota costs the cache, not the search */
await sync.storePdfText(attachmentId, pages);
} catch (err) {
console.warn(`[pdf] could not store the text of ${attachmentId}:`, err);
}
return result;
}
+7
View File
@@ -4,6 +4,7 @@ import * as sync from "../syncEngine";
import * as searchIndex from "../search";
import { sameStrokeDoc } from "../features/drawing/strokes.js";
import { reindexEntries } from "../features/notes/noteSort";
import { extractOnImport } from "../features/pdf/pdfText";
import type { Backlink, NoteContent, NoteMeta, Notebook, StrokeDoc, User } from "../types";
/** How often the whole account is reconciled with the server. */
@@ -203,6 +204,12 @@ export function useNoteData(user: User | null, activeNote: string | null): NoteD
const title = file.name.replace(/\.pdf$/i, "") || "Dokument";
const note = await sync.createPdfNote(folderId, title, attachment.id);
setNotes((p) => [note, ...p]);
/* Read the document here, now, on the machine that already has it open.
*
* Waiting until someone opens it means a filed invoice may never be read at
* all, and until it is, no device can find it. Not awaited: the note exists
* and should appear at once — the text catches up. */
void extractOnImport(file, attachment.id);
return note;
}, []);
+7
View File
@@ -2,6 +2,7 @@ import { useEffect, useMemo, useRef, useState } from "react";
import * as sync from "../syncEngine";
import * as searchIndex from "../search";
import { backfillPdfText } from "../features/pdf/pdfText";
import type { NoteMeta, SearchHit, User } from "../types";
const DEBOUNCE_MS = 120;
@@ -52,6 +53,12 @@ export function useSearchIndex(user: User | null, notes: NoteMeta[], query: stri
},
});
if (!cancelled) setVersion((v) => v + 1);
/* PDFs last: their text is stored with the file rather than derived from a
* note body, so it has to be fetched separately — and a document nobody
* has read yet is simply skipped. Searching a note is more urgent than
* searching a scan, so this waits until the notes are in. */
const indexed = await backfillPdfText(notesRef.current, signal);
if (indexed && !cancelled) setVersion((v) => v + 1);
running.current = false;
})();
+50 -19
View File
@@ -29,6 +29,11 @@ export interface ImportProgress {
label: string;
}
/* How far to look for further pages of one note's drawing. A bound rather than
* a limit anyone should reach: without one, a malformed archive would have this
* loop asking for filenames forever. */
const MAX_DRAWING_PAGES = 200;
export interface ImportResult {
notes: NoteMeta[];
/** Notes whose drawing came back as a picture instead of editable strokes. */
@@ -140,14 +145,42 @@ async function importOne(
else result.missingImages++;
}
/* The drawing, if the archive carried one. Two layouts exist: a notebook
* export puts it in `zeichnungen/`, a single-note export writes it beside the
* file as `<name>-zeichnung.png`. */
/* The drawing, if the archive carried one.
*
* Four layouts exist. A notebook export files it under `drawings/`, a
* single-note export writes it beside the file as `<name>-drawing.png`, and
* both had German names — `zeichnungen/`, `-zeichnung` — before the interface
* was put into English. All four are read: an archive written last year is
* exactly the archive someone reaches for.
*
* Only the first two were being looked for until now, which meant a drawing
* exported by this very app came back as nothing at all.
*
* A drawing may run over several pages. Page one keeps the plain name and the
* rest are numbered after it, so they come back in order.
*/
const base = baseName(markdownPath);
const drawingPath = [`zeichnungen/${base}.png`, `${base}-zeichnung.png`].find((p) => sources.files.has(p));
const drawingId = drawingPath ? await uploadPicture(sources.files.get(drawingPath)!, drawingPath) : null;
if (drawingId) result.flattenedDrawings++;
const drawingIds: string[] = [];
for (const suffix of ["", ...Array.from({ length: MAX_DRAWING_PAGES - 1 }, (_, i) => `-${i + 2}`)]) {
const path = [
`drawings/${base}${suffix}.png`,
`${base}-drawing${suffix}.png`,
`zeichnungen/${base}${suffix}.png`,
`${base}-zeichnung${suffix}.png`,
].find((p) => sources.files.has(p));
const bytes = path ? sources.files.get(path) : null;
if (!path || !bytes) break; // pages are consecutive; the first gap is the end
const id = await uploadPicture(bytes, path);
if (id) drawingIds.push(id);
}
if (drawingIds.length) result.flattenedDrawings++;
const asPicture = (id: string) => ({
type: "image",
attrs: { attachmentId: id, alt: "Drawing", src: null },
});
let replaced = false;
let doc = mapNodes(parsed.doc, (n) => {
if (n.type === "image") {
const id = n.attrs?.src ? uploaded.get(n.attrs.src) : null;
@@ -158,22 +191,20 @@ async function importOne(
}
// The exporter leaves a note saying the drawing lies beside the file. Now it
// is back inside the note, so the note about it would be wrong.
if (drawingId && isDrawingMarker(n)) {
return { type: "image", attrs: { attachmentId: drawingId, alt: "Zeichnung", src: null } };
const first = drawingIds[0];
if (first && isDrawingMarker(n)) {
replaced = true;
return asPicture(first);
}
return n;
})!;
// No marker to replace (a hand-written archive, or an older export) — the
// drawing still belongs in the note, so it goes at the end.
if (drawingId && !JSON.stringify(doc).includes(drawingId)) {
doc = {
...doc,
content: [
...(doc.content ?? []),
{ type: "image", attrs: { attachmentId: drawingId, alt: "Zeichnung", src: null } },
],
};
// The marker stands for the drawing as a whole, so it can only become the
// first page; the others follow it at the end. With no marker at all (a
// hand-written archive, or an older export) every page goes there.
const trailing = replaced ? drawingIds.slice(1) : drawingIds;
if (trailing.length) {
doc = { ...doc, content: [...(doc.content ?? []), ...trailing.map(asPicture)] };
}
const note = await sync.createNote(folderId);
@@ -182,7 +213,7 @@ async function importOne(
note.id,
serializeNoteContent(doc),
[],
[...uploaded.values(), ...(drawingId ? [drawingId] : [])]
[...uploaded.values(), ...drawingIds]
);
return { ...note, title };
}
+16
View File
@@ -1127,6 +1127,22 @@ export async function loadPdfBytes(attachmentId: string): Promise<Uint8Array> {
return new Uint8Array(await blob.arrayBuffer());
}
/* ─── The text inside a PDF ───
*
* Read on a device, stored with the file, handed to every other device. The
* server never opens a PDF; it keeps this the way it keeps a note body —
* encrypted, and unread by it. */
/** The stored text of a PDF, page by page, or null if nobody has read it yet. */
export async function fetchPdfText(attachmentId: string): Promise<string[] | null> {
const r = await api.get<{ pages: string[] | null }>(`/attachments/${attachmentId}/text`);
return r.pages ?? null;
}
export async function storePdfText(attachmentId: string, pages: string[]): Promise<void> {
await api.put(`/attachments/${attachmentId}/text`, { pages });
}
/** Creates a note that holds a PDF. */
export async function createPdfNote(
folderId: string | null,
+72 -9
View File
@@ -333,14 +333,14 @@ docker compose -f docker-compose.dev.yml up -d --build # must be running
| `editor` | 31 | tables and their right-click menu, task lists, note links, backlinks — and that each survives a reload |
| `search-attachments` | 12 | body-only matches, ranking, umlauts; images as blob URLs, not inline bytes |
| `history` | 10 | a large deletion is snapshotted, restoring works, restoring is itself undoable |
| `drawing` | 7 | strokes survive the poller, and match across four viewport widths |
| `drawing` | 18 | strokes survive the poller, match across four viewport widths, and stay on the page they were drawn on |
| `pdf` | 91 | import, page proportions, lazy loading, and reading with the network off |
| `attachments-lifecycle` | 10 | a picture is kept while any version shows it, and freed once none does |
| `conflict` | 10 | a late write wins but the state it buries survives as a version |
| `export` | 21 | the downloaded bytes are real Markdown, ZIP and PNG, unpacked and read back |
| `organise` | 39 | a hand-made order survives a search; separators hold their place; a note exported and imported comes back whole |
| `offline` | 12 | a cold page load with the network off still opens notes and searches them |
| `sync` | 5 | a PDF written on one device — document, drawing and pinned notes — reaches a second one |
| `sync` | 7 | a PDF written on one device — document, drawing and pinned notes — reaches a second one, which can find it by its contents without opening it |
Each suite creates a throwaway account through the admin API and deletes it
afterwards, so runs leave the database as they found it. They need an admin to
@@ -399,8 +399,8 @@ and a test asserts none ever reaches the stored body. The current page is the on
under the *middle* of the window: measuring from the top said "page 1" while you
stood at the foot of a two-page note, because a window is shorter than a page.
The drawing still covers only the first page. On a PDF the strokes are kept per
page; on a note there is one sheet.
Each page can be drawn on — see [One drawing, as many pages as the
note](#one-drawing-as-many-pages-as-the-note).
### Alignment
@@ -473,6 +473,11 @@ fresh device with nothing to search. After login a background pass backfills the
missing ones into IndexedDB, slowly enough to stay out of the way, indexing each
as it lands. Searching therefore also works offline.
PDFs come after the notes in that pass. Their text is not derived from a note
body but stored with the file, by the device that imported it — see [The reading
happens on a device, never on the
server](#the-reading-happens-on-a-device-never-on-the-server).
Matching is AND across terms, case- and diacritic-insensitive (`kuhlschrank`
finds *Kühlschrank*), with title matches ranked above body matches. Results show
a snippet around the first match.
@@ -581,11 +586,34 @@ one search box finds a sentence whether it was typed or imported. Extraction run
*after* the document is on screen and yields between pages: it must never stand
between the reader and the first page.
**The extracted text is not sent to the server.** It is derivable from a file the
server already holds, so storing it again would cost space and a second thing to
encrypt for nothing gained. The price is that a PDF becomes searchable on a
device once that device has opened it — the same shape as the limitation notes
already have, and the honest trade for not duplicating data.
### The reading happens on a device, never on the server
The server has no PDF library and no way to open one: the bytes arrive encrypted
and are stored that way. **The device that imports the file reads it**, at import
rather than at first open, and the result is stored beside the file in
`attachments.text_index` — encrypted at rest like everything else, and opaque to
the server.
That is what lets a device search a document it has never opened. It is the
precondition for filing scanned invoices: they are imported, and then never
opened again, so waiting for someone to open one would mean waiting forever.
```
PUT /api/attachments/:id/text { "pages": ["page one text", "page two text"] }
GET /api/attachments/:id/text → { "pages": [...] } or { "pages": null }
```
`null` is not an error — it means nobody has read this file yet, and whichever
device holds the bytes should. After login the search backfill asks for the text
of every PDF the account holds, after the note bodies, and skips the ones nobody
has read: downloading every document just to index it is not a thing to do in the
background.
This was deliberately *not* stored for a while, on the reasoning that text
derivable from bytes the server already has should not be kept twice. True, and
useless: a PDF imported on the desktop was invisible to a search on the phone
until the phone had opened it. A few hundred kilobytes of text against a file
measured in megabytes is a trade worth making.
The minimal text-layer CSS is written out in `textLayer.css` rather than
importing `pdfjs-dist/web/pdf_viewer.css`, which carries the whole reference
@@ -993,6 +1021,41 @@ encrypted at rest like note content.
Notes drawn before this format still carry a PNG in `drawing_data_path`; it renders
underneath any newer strokes so nothing is lost.
### One drawing, as many pages as the note
A note used to have exactly one sheet to draw on, which stopped making sense once
the writing could run to four pages: the ink was stuck on the first one while the
text carried on without it. Every page is exported anyway, so every page can be
drawn on. Pages are numbered the way the PDF reader numbers them:
```json
{ "v": 2, "w": 1200, "h": 1600, "sheets": {
"1": [ { "t": "pen", "c": "#e94560", "s": 5.5, "p": [100,100, 200,180] } ],
"3": [ { "t": "highlighter", "c": "#f5c518", "s": 27, "p": [80,900, 900,910] } ]
}}
```
A page nobody has drawn on is not stored, so the keys *are* the drawn pages.
**A drawing that lives entirely on page one is still written in the old shape**,
with a plain `strokes` list and no `sheets` at all. Nothing requires that — it
means an ordinary note's drawing is stored exactly as it always was, and a device
running an older build still shows it.
While drawing there is always one blank page past the end. Pages otherwise come
from how far the writing runs, which would leave a drawing with nowhere to go: to
draw on page three you would first have to write three pages. Drawing on the
spare page makes it a real one, and the page count in the toolbar counts it.
The bin clears **the page being looked at**, not the whole drawing. Undo and redo
work on the drawing as a whole, because one Undo button that means four different
things depending on where you scrolled is not a button anyone can use.
In an export, page one keeps the name a drawing has always had —
`drawings/<note>.png`, or `<note>-drawing.png` beside a single note — and the
rest are numbered after it. A drawing of several pages exported on its own comes
out as a ZIP rather than as its first page alone.
---
## Conflict Resolution