Docker/InkNote
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Free unused attachments, and keep versions by count rather than by age

Browse Source 
Two rules governed version retention: the newest 50 per note, and a sweep
of anything past 90 days. The second was wrong. The count already bounds
what a note can accumulate, so age only ever deleted history nothing had
replaced — and it bit hardest exactly where history is most wanted, a
note left alone for a quarter of a year losing its whole past for having
done nothing. It is off unless VERSION_RETENTION_DAYS is set.

How many versions to keep is now a notebook's own setting, falling back
to the server default when left empty. A reference notebook can keep
fifty of everything while a scratch one keeps three.

That makes freeing attachments answerable. A picture taken out of a note
cannot be deleted straight away: the versions from before still show it,
and restoring one would find a hole. It is released once no current note
and no kept version refers to it — which, with retention by count, is
when the last version holding it is pushed out.

The server cannot look for itself, bodies being encrypted, so the client
reports a note's attachments on every save as it already does for links,
and a version inherits what the note held when it was snapshotted — by
definition what that version contains.

Two things make it safe rather than merely plausible. Silence is not
"uses nothing": a note written before this existed has no rows, which
looks identical to one with no pictures, so nothing is released for a
user while any of their notes has yet to report — the prefetch that fills
the search index reports for them on its first pass. And an upload has a
row before the note using it does, so recent attachments are left alone.

POST /api/admin/housekeeping runs the sweep on demand.

Caught while testing: the same block of markup had been pasted into the
create-notebook dialog as well, where the state it referenced did not
exist — the dialog crashed on open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
marcandClaude Opus 5 committed 2026-08-08 22:22:46 +02:00
1 parent a32d6eba99
commit 3acb8b66a4
12 files changed
+507 -28
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

backend/server.js
+176 -11
View File
@@ -201,6 +201,27 @@ db.exec(`
-- Past states of a note, so an accidental overwrite can be undone after the
-- fact. Content is encrypted here exactly as it is in the notes table.
/* Which attachments a note's current body uses, and which each kept version
* used. The server cannot read either — both are encrypted — so the client
* reports the note's set on every save, and a version inherits whatever the
* note held at the moment it was snapshotted, which is by definition what
* that version contains.
*
* Together these are what make freeing an attachment safe: a picture removed
* from a note is still in the versions that came before, and deleting the file
* would leave a hole in history that restoring cannot fill. */
CREATE TABLE IF NOT EXISTS note_attachments (
note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
attachment_id TEXT NOT NULL,
PRIMARY KEY (note_id, attachment_id)
);
CREATE TABLE IF NOT EXISTS version_attachments (
version_id INTEGER NOT NULL REFERENCES note_versions(id) ON DELETE CASCADE,
attachment_id TEXT NOT NULL,
PRIMARY KEY (version_id, attachment_id)
);
CREATE TABLE IF NOT EXISTS note_versions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
note_id TEXT NOT NULL REFERENCES notes(id) ON DELETE CASCADE,
@@ -225,6 +246,8 @@ db.exec(`
created_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_note_attachments_att ON note_attachments(attachment_id);
CREATE INDEX IF NOT EXISTS idx_version_attachments_att ON version_attachments(attachment_id);
CREATE INDEX IF NOT EXISTS idx_notes_user ON notes(user_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_attachments_owner_hash ON attachments(user_id, sha256);
CREATE INDEX IF NOT EXISTS idx_note_versions ON note_versions(note_id, id DESC);
@@ -299,6 +322,23 @@ if (!tableColumns("notes").includes("drawing_strokes")) {
// A PDF note points at the attachment holding its file. The old pdf_path column
// stored a filesystem path instead, which is why it never survived a container
// rebuild; attachments are content-addressed, encrypted and deduplicated.
// How many versions a notebook keeps per note. Null falls back to the global
// default, so a notebook only carries a number once someone has chosen one.
/* Whether this note's client has told us which attachments it uses.
*
* Notes written before that existed have no reference rows, which is
* indistinguishable from a note using no pictures at all — and freeing on that
* basis would delete images that are still on screen. Nothing is released for a
* user until every one of their notes has reported, which the background
* prefetch takes care of on the first pass after an upgrade. */
if (!tableColumns("notes").includes("attachments_reported")) {
db.exec("ALTER TABLE notes ADD COLUMN attachments_reported INTEGER NOT NULL DEFAULT 0");
}
if (!tableColumns("notebooks").includes("max_versions")) {
db.exec("ALTER TABLE notebooks ADD COLUMN max_versions INTEGER");
}
if (!tableColumns("notes").includes("pdf_attachment_id")) {
db.exec("ALTER TABLE notes ADD COLUMN pdf_attachment_id TEXT");
}
@@ -673,6 +713,26 @@ app.put("/api/admin/users/:id/password", authMiddleware, requireAdmin, async (re
}
});
/**
* Runs the housekeeping sweep now instead of waiting for its timer.
*
* `graceSeconds` lowers the delay that protects an upload still in flight — its
* row exists before the client has had a chance to report the note that uses it.
* Only an administrator can ask, and only on their own instance; the default
* leaves the protection alone.
*/
app.post("/api/admin/housekeeping", authMiddleware, requireAdmin, (req, res) => {
const grace =
req.body?.graceSeconds === undefined
? ORPHAN_GRACE_MS
: Math.max(0, Number(req.body.graceSeconds) || 0) * 1000;
pruneSyncLog();
pruneOldVersions();
pruneUnreferencedAttachments(grace);
pruneOrphanAttachments(grace);
res.json({ success: true });
});
app.delete("/api/admin/users/:id", authMiddleware, requireAdmin, (req, res) => {
const target = db.prepare("SELECT * FROM users WHERE id = ?").get(req.params.id);
if (!target) return res.status(404).json({ error: "User not found" });
@@ -710,15 +770,28 @@ app.post("/api/notebooks", authMiddleware, (req, res) => {
});
app.put("/api/notebooks/:id", authMiddleware, (req, res) => {
const { name, coverIndex, sortOrder } = req.body;
const { name, coverIndex, sortOrder, maxVersions } = req.body;
const nb = db.prepare("SELECT * FROM notebooks WHERE id = ? AND user_id = ?").get(req.params.id, req.userId);
if (!nb) return res.status(404).json({ error: "Not found" });
// null clears the notebook's own setting and puts it back on the default;
// undefined leaves whatever it had.
let versions;
if (maxVersions === null) versions = null;
else if (maxVersions !== undefined) {
const n = Number(maxVersions);
if (!Number.isFinite(n) || n < 1) return res.status(400).json({ error: "maxVersions must be at least 1" });
versions = Math.min(Math.round(n), MAX_VERSIONS_CEILING);
}
db.prepare("UPDATE notebooks SET name = COALESCE(?, name), cover_index = COALESCE(?, cover_index), sort_order = COALESCE(?, sort_order), updated_at = datetime('now'), version = version + 1 WHERE id = ?")
.run(name, coverIndex, sortOrder, req.params.id);
if (maxVersions !== undefined) {
db.prepare("UPDATE notebooks SET max_versions = ? WHERE id = ?").run(versions, req.params.id);
}
broadcastToUser(req.userId, { type: "notebook_updated", data: { id: req.params.id, name, coverIndex } }, req.deviceId);
res.json({ success: true });
res.json({ success: true, defaultMaxVersions: MAX_VERSIONS_PER_NOTE });
});
app.delete("/api/notebooks/:id", authMiddleware, (req, res) => {
@@ -743,16 +816,37 @@ const num = (name, fallback) => {
};
const VERSION_COALESCE_MS = num("VERSION_COALESCE_MINUTES", 5) * 60 * 1000;
const MAX_VERSIONS_PER_NOTE = num("MAX_VERSIONS_PER_NOTE", 50);
const VERSION_RETENTION_DAYS = num("VERSION_RETENTION_DAYS", 90);
/* A per-notebook setting is a number a user types; it needs an upper bound. */
const MAX_VERSIONS_CEILING = 500;
/* Zero means "never expire by age", and that is the default.
*
* Versions used to be dropped after 90 days as well as being capped per note.
* The cap already bounds how much a note can accumulate, so the age rule only
* ever deleted history that nothing had replaced — and it hit hardest exactly
* where history is most wanted: a note untouched for a quarter of a year lost
* its whole past, having done nothing but sit there. Set the variable to a
* number of days to bring it back. */
const VERSION_RETENTION_DAYS = num("VERSION_RETENTION_DAYS", 0);
const sqliteTimeToMs = (t) => (t ? Date.parse(`${String(t).replace(" ", "T")}Z`) : 0);
/** How many versions this note's notebook keeps — its own setting, or the default. */
function versionLimitFor(noteId) {
const row = db
.prepare(
"SELECT b.max_versions AS limit_ FROM notes n LEFT JOIN notebooks b ON b.id = n.notebook_id WHERE n.id = ?"
)
.get(noteId);
const own = Number(row?.limit_);
return Number.isFinite(own) && own > 0 ? Math.min(own, MAX_VERSIONS_CEILING) : MAX_VERSIONS_PER_NOTE;
}
function pruneNoteVersions(noteId) {
db.prepare(`
DELETE FROM note_versions
WHERE note_id = ? AND id NOT IN (
SELECT id FROM note_versions WHERE note_id = ? ORDER BY id DESC LIMIT ?
)`).run(noteId, noteId, MAX_VERSIONS_PER_NOTE);
)`).run(noteId, noteId, versionLimitFor(noteId));
}
function snapshotNote(userId, noteId, deviceId, { force = false, reason = "edit" } = {}) {
@@ -768,9 +862,18 @@ function snapshotNote(userId, noteId, deviceId, { force = false, reason = "edit"
if (last && Date.now() - sqliteTimeToMs(last.created_at) < VERSION_COALESCE_MS) return;
}
db.prepare(`INSERT INTO note_versions (note_id, user_id, title, content_encrypted, drawing_strokes, device_id, reason)
const inserted = db.prepare(`INSERT INTO note_versions (note_id, user_id, title, content_encrypted, drawing_strokes, device_id, reason)
VALUES (?, ?, ?, ?, ?, ?, ?)`)
.run(noteId, userId, note.title, note.content_encrypted, note.drawing_strokes, deviceId, reason);
/* The version inherits the note's attachments as they stand right now, which
* is exactly what the body being snapshotted refers to — the snapshot is taken
* before the incoming write is applied, and so is this. Without it, removing a
* picture from a note would free the file while older versions still showed it. */
db.prepare(
"INSERT OR IGNORE INTO version_attachments (version_id, attachment_id) SELECT ?, attachment_id FROM note_attachments WHERE note_id = ?"
).run(inserted.lastInsertRowid, noteId);
pruneNoteVersions(noteId);
}
@@ -871,8 +974,24 @@ const replaceNoteLinks = db.transaction((userId, sourceId, links) => {
}
});
/* An upper bound on what one note may claim, so a malformed client cannot
* write an unbounded number of rows. */
const MAX_ATTACHMENTS_PER_NOTE = 500;
/** Records which attachments a note's body currently uses. */
const replaceNoteAttachments = db.transaction((noteId, ids) => {
db.prepare("DELETE FROM note_attachments WHERE note_id = ?").run(noteId);
db.prepare("UPDATE notes SET attachments_reported = 1 WHERE id = ?").run(noteId);
const insert = db.prepare(
"INSERT OR IGNORE INTO note_attachments (note_id, attachment_id) VALUES (?, ?)"
);
for (const id of ids.slice(0, MAX_ATTACHMENTS_PER_NOTE)) {
if (typeof id === "string" && id) insert.run(noteId, id);
}
});
app.put("/api/notes/:id", authMiddleware, (req, res) => {
const { title, content, notebookId, starred, baseUpdatedAt, links } = req.body;
const { title, content, notebookId, starred, baseUpdatedAt, links, attachments } = req.body;
const note = db.prepare("SELECT * FROM notes WHERE id = ? AND user_id = ?").get(req.params.id, req.userId);
if (!note) return res.status(404).json({ error: "Not found" });
@@ -905,6 +1024,7 @@ app.put("/api/notes/:id", authMiddleware, (req, res) => {
.run(title, encContent, notebookId, starred !== undefined ? (starred ? 1 : 0) : null, req.deviceId, req.params.id);
if (Array.isArray(links)) replaceNoteLinks(req.userId, req.params.id, links);
if (Array.isArray(attachments)) replaceNoteAttachments(req.params.id, attachments);
const fresh = db.prepare("SELECT updated_at FROM notes WHERE id = ?").get(req.params.id);
broadcastToUser(req.userId, { type: "note_updated", data: { id: req.params.id, title } }, req.deviceId);
@@ -1376,9 +1496,10 @@ function pruneSyncLog() {
} catch (err) { console.error("[prune] failed:", err.message); }
}
/* Per-note count is capped on write; this only clears out versions of notes
* that stopped being edited long ago. */
/* Off unless VERSION_RETENTION_DAYS is set — the per-note count is what bounds
* growth, and see the note on that constant for why age is a poor second rule. */
function pruneOldVersions() {
if (VERSION_RETENTION_DAYS <= 0) return;
try {
const r = db.prepare(`DELETE FROM note_versions WHERE created_at < datetime('now', '-${VERSION_RETENTION_DAYS} days')`).run();
if (r.changes) console.log(` 🧹 pruned ${r.changes} note versions older than ${VERSION_RETENTION_DAYS} days`);
@@ -1391,7 +1512,51 @@ function pruneOldVersions() {
* Only files that have sat untouched for an hour are considered, so an upload
* in flight — written to disk just before its row is inserted — is never hit. */
const ORPHAN_GRACE_MS = 60 * 60 * 1000;
function pruneOrphanAttachments() {
/**
* Deletes attachments nothing refers to any more.
*
* An attachment is free once no current note body and no kept version mentions
* it. Both halves matter: a picture taken out of a note is still in the versions
* from before it was removed, and freeing it then would leave a hole that
* restoring cannot fill.
*
* So a deleted image does not vanish at once. It goes when the last version
* holding it is pushed out by newer ones — which is the point at which nothing
* can ask for it back.
*
* Rows are only considered once the note has actually reported its attachments;
* a client that has never sent the list would otherwise look like a note using
* nothing at all.
*/
function pruneUnreferencedAttachments(graceMs = ORPHAN_GRACE_MS) {
try {
const rows = db
.prepare(
`SELECT a.id, a.sha256 FROM attachments a
WHERE a.created_at < datetime('now', ?)
AND NOT EXISTS (SELECT 1 FROM note_attachments na WHERE na.attachment_id = a.id)
AND NOT EXISTS (SELECT 1 FROM version_attachments va WHERE va.attachment_id = a.id)
-- Nothing is released while any of this user's notes has yet to say
-- what it uses: silence there is not the same as "uses nothing".
AND NOT EXISTS (
SELECT 1 FROM notes n
WHERE n.user_id = a.user_id AND n.deleted_at IS NULL AND n.attachments_reported = 0
)`
)
.all(`-${Math.round(graceMs / 1000)} seconds`);
if (rows.length === 0) return;
const del = db.prepare("DELETE FROM attachments WHERE id = ?");
for (const row of rows) del.run(row.id);
console.log(` 🧹 released ${rows.length} attachment(s) no note or version refers to`);
// The files behind them go in the sweep below, which only removes blobs no
// row points at — including copies still shared with another user's row.
} catch (err) {
console.error("[prune] attachment references failed:", err.message);
}
}
function pruneOrphanAttachments(graceMs = ORPHAN_GRACE_MS) {
try {
if (!fs.existsSync(ATTACHMENT_DIR)) return;
const referenced = new Set(db.prepare("SELECT sha256 FROM attachments").all().map((r) => r.sha256));
@@ -1400,7 +1565,7 @@ function pruneOrphanAttachments() {
if (referenced.has(name)) continue;
const full = path.join(ATTACHMENT_DIR, name);
try {
if (Date.now() - fs.statSync(full).mtimeMs < ORPHAN_GRACE_MS) continue;
if (Date.now() - fs.statSync(full).mtimeMs < graceMs) continue;
fs.unlinkSync(full);
removed++;
} catch { /* leave anything we cannot inspect alone */ }
@@ -1409,7 +1574,7 @@ function pruneOrphanAttachments() {
} catch (err) { console.error("[prune] attachments failed:", err.message); }
}
const housekeeping = () => { pruneSyncLog(); pruneOldVersions(); pruneOrphanAttachments(); };
const housekeeping = () => { pruneSyncLog(); pruneOldVersions(); pruneUnreferencedAttachments(); pruneOrphanAttachments(); };
housekeeping();
setInterval(housekeeping, 24 * 60 * 60 * 1000).unref();