feat: canonical SQL DDL + schema validator + migration tool
- schema/canonical.sql: 29 tables across 3 databases, CHECK constraints, foreign keys, 13 indexes, WAL mode, schema_version tracking - tools/validate-schema.ts: applies DDL to in-memory SQLite, extracts PRAGMA table_info + sqlite_master metadata as JSON - tools/migrate-db.ts: CLI for Tauri→Electrobun data migration with atomic transaction, version fencing, INSERT OR IGNORE - docs/SWITCHING.md: migration guide with prerequisites and troubleshooting
This commit is contained in:
Hibryda
parent
0f75cb8e32
commit
631fc2efc8
4 changed files with 718 additions and 0 deletions
91
docs/SWITCHING.md
Normal file
91
docs/SWITCHING.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
# Switching from Tauri to Electrobun
|
||||
|
||||
This guide covers migrating your data when switching from the Tauri v2/v3 build to the Electrobun build of AGOR.
|
||||
|
||||
## Overview
|
||||
|
||||
Both stacks use SQLite for persistence but store databases in different locations with slightly different schemas. The `migrate-db` tool copies data from a Tauri source database into an Electrobun target database using the canonical schema as the contract.
|
||||
|
||||
**This is a one-way migration.** The source database is opened read-only and never modified. The target database receives copies of all compatible data. Running the tool multiple times is safe (uses `INSERT OR IGNORE`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Bun** >= 1.0 (ships with bun:sqlite)
|
||||
- A working Tauri installation with existing data in `~/.local/share/agor/sessions.db`
|
||||
- The Electrobun build installed (creates `~/.config/agor/settings.db` on first run)
|
||||
|
||||
## Database Locations
|
||||
|
||||
| Database | Tauri path | Electrobun path |
|
||||
|----------|-----------|-----------------|
|
||||
| Settings + sessions | `~/.local/share/agor/sessions.db` | `~/.config/agor/settings.db` |
|
||||
| btmsg + bttask | `~/.local/share/agor/btmsg.db` | `~/.local/share/agor/btmsg.db` (shared) |
|
||||
| FTS5 search | `~/.local/share/agor/search.db` | `~/.local/share/agor/search.db` (shared) |
|
||||
|
||||
The btmsg and search databases are already shared between stacks -- no migration needed for those.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Stop both applications
|
||||
|
||||
Close the Tauri app and the Electrobun app if either is running. SQLite WAL mode handles concurrent reads, but stopping both avoids partial writes.
|
||||
|
||||
### 2. Back up your data
|
||||
|
||||
```bash
|
||||
cp ~/.local/share/agor/sessions.db ~/.local/share/agor/sessions.db.bak
|
||||
cp ~/.config/agor/settings.db ~/.config/agor/settings.db.bak 2>/dev/null
|
||||
```
|
||||
|
||||
### 3. Run the migration
|
||||
|
||||
```bash
|
||||
# Migrate settings/sessions from Tauri to Electrobun
|
||||
bun tools/migrate-db.ts \
|
||||
--from ~/.local/share/agor/sessions.db \
|
||||
--to ~/.config/agor/settings.db
|
||||
```
|
||||
|
||||
The tool will:
|
||||
- Open the source database read-only
|
||||
- Apply the canonical schema to the target if needed
|
||||
- Copy all matching tables using `INSERT OR IGNORE` (existing rows are preserved)
|
||||
- Report per-table row counts
|
||||
- Write a version fence to `schema_version`
|
||||
|
||||
### 4. Verify
|
||||
|
||||
Launch the Electrobun app and confirm your projects, settings, and session history appear correctly.
|
||||
|
||||
### 5. (Optional) Validate the schema
|
||||
|
||||
```bash
|
||||
bun tools/validate-schema.ts | jq '.tableCount'
|
||||
# Should output the number of tables defined in canonical.sql
|
||||
```
|
||||
|
||||
## Version Fence
|
||||
|
||||
After migration, the target database's `schema_version` table contains:
|
||||
|
||||
| Column | Value |
|
||||
|--------|-------|
|
||||
| `version` | `1` |
|
||||
| `migration_source` | `migrate-db` |
|
||||
| `migration_timestamp` | ISO-8601 timestamp of the migration |
|
||||
|
||||
This fence prevents accidental re-application of older schemas and provides an audit trail.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"Source database not found"** -- Verify the Tauri data directory path. On some systems, `XDG_DATA_HOME` may override `~/.local/share`.
|
||||
|
||||
**"Schema application failed"** -- The canonical.sql file may be out of sync with a newer database. Pull the latest version and retry.
|
||||
|
||||
**"Migration failed on table X"** -- The migration runs inside a single transaction. If any table fails, all changes are rolled back. Check the error message for column mismatches, which typically mean the canonical schema needs updating.
|
||||
|
||||
## What is NOT Migrated
|
||||
|
||||
- **FTS5 search indexes** -- These are virtual tables that cannot be copied via `SELECT *`. Rebuild the index from the Electrobun app (Ctrl+Shift+F, then rebuild).
|
||||
- **Layout state** -- The Electrobun UI uses a different layout system. Your layout preferences will reset.
|
||||
- **SSH key files** -- Only the SSH connection metadata (host, port, username) is migrated. Private key files remain on disk at their original paths.
|
||||
299
schema/canonical.sql
Normal file
299
schema/canonical.sql
Normal file
|
|
@ -0,0 +1,299 @@
|
|||
-- canonical.sql — Authoritative DDL for all AGOR SQLite databases.
|
||||
-- Both Tauri (Rust) and Electrobun (Bun) stacks MUST match this schema.
|
||||
-- DBs: settings.db, btmsg.db, search.db. See docs/SWITCHING.md.
|
||||
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
PRAGMA busy_timeout = 5000;
|
||||
|
||||
-- ── VERSION TRACKING ──────────────────────────────────────────────────
|
||||
-- Schema version fence. One row per database file.
|
||||
CREATE TABLE IF NOT EXISTS schema_version (
|
||||
version INTEGER NOT NULL,
|
||||
migration_source TEXT, -- e.g. 'tauri', 'electrobun', 'migrate-db'
|
||||
migration_timestamp TEXT -- ISO-8601 when last migrated
|
||||
);
|
||||
|
||||
-- ── settings.db TABLES ────────────────────────────────────────────────
|
||||
|
||||
-- Key-value application settings (theme, fonts, shell, cwd, etc.)
|
||||
CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||
|
||||
-- Project configurations stored as JSON blobs.
|
||||
CREATE TABLE IF NOT EXISTS projects (id TEXT PRIMARY KEY, config TEXT NOT NULL);
|
||||
|
||||
-- Workspace groups (sidebar tabs).
|
||||
CREATE TABLE IF NOT EXISTS groups (
|
||||
id TEXT PRIMARY KEY, name TEXT NOT NULL,
|
||||
icon TEXT NOT NULL, position INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- User-created custom themes.
|
||||
CREATE TABLE IF NOT EXISTS custom_themes (
|
||||
id TEXT PRIMARY KEY, name TEXT NOT NULL,
|
||||
palette TEXT NOT NULL -- JSON blob (color map)
|
||||
);
|
||||
|
||||
-- User-customized keyboard shortcuts.
|
||||
CREATE TABLE IF NOT EXISTS keybindings (id TEXT PRIMARY KEY, chord TEXT NOT NULL);
|
||||
|
||||
-- Agent sessions per project (provider-agnostic).
|
||||
CREATE TABLE IF NOT EXISTS agent_sessions (
|
||||
project_id TEXT NOT NULL,
|
||||
session_id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'idle'
|
||||
CHECK (status IN ('idle','running','error','stopped','completed')),
|
||||
cost_usd REAL NOT NULL DEFAULT 0,
|
||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
model TEXT NOT NULL DEFAULT '',
|
||||
error TEXT,
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_sessions_project ON agent_sessions(project_id);
|
||||
|
||||
-- Individual agent messages within a session.
|
||||
CREATE TABLE IF NOT EXISTS agent_messages (
|
||||
session_id TEXT NOT NULL,
|
||||
msg_id TEXT NOT NULL,
|
||||
role TEXT NOT NULL,
|
||||
content TEXT NOT NULL DEFAULT '',
|
||||
tool_name TEXT,
|
||||
tool_input TEXT,
|
||||
timestamp INTEGER NOT NULL,
|
||||
cost_usd REAL NOT NULL DEFAULT 0,
|
||||
input_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
output_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (session_id, msg_id),
|
||||
FOREIGN KEY (session_id) REFERENCES agent_sessions(session_id) ON DELETE CASCADE
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_messages_session ON agent_messages(session_id, timestamp);
|
||||
|
||||
-- Historical session metrics (cost, tokens, turns) per project.
|
||||
CREATE TABLE IF NOT EXISTS session_metrics (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
project_id TEXT NOT NULL,
|
||||
session_id TEXT NOT NULL,
|
||||
start_time INTEGER NOT NULL,
|
||||
end_time INTEGER NOT NULL,
|
||||
peak_tokens INTEGER DEFAULT 0,
|
||||
turn_count INTEGER DEFAULT 0,
|
||||
tool_call_count INTEGER DEFAULT 0,
|
||||
cost_usd REAL DEFAULT 0,
|
||||
model TEXT,
|
||||
status TEXT NOT NULL,
|
||||
error_message TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_metrics_project ON session_metrics(project_id);
|
||||
|
||||
-- Session anchors — preserved turns surviving compaction.
|
||||
CREATE TABLE IF NOT EXISTS session_anchors (
|
||||
id TEXT PRIMARY KEY,
|
||||
project_id TEXT NOT NULL,
|
||||
message_id TEXT NOT NULL,
|
||||
anchor_type TEXT NOT NULL
|
||||
CHECK (anchor_type IN ('auto','pinned','promoted')),
|
||||
content TEXT NOT NULL,
|
||||
estimated_tokens INTEGER NOT NULL,
|
||||
turn_index INTEGER NOT NULL DEFAULT 0,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_anchors_project ON session_anchors(project_id);
|
||||
|
||||
-- Remote relay machine connections.
|
||||
CREATE TABLE IF NOT EXISTS remote_machines (
|
||||
id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL,
|
||||
url TEXT NOT NULL,
|
||||
token TEXT NOT NULL,
|
||||
auto_connect INTEGER NOT NULL DEFAULT 0,
|
||||
spki_pins TEXT NOT NULL DEFAULT '[]', -- JSON array of SPKI hashes
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- Legacy v2 layout state (single-row).
|
||||
CREATE TABLE IF NOT EXISTS layout_state (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
preset TEXT NOT NULL DEFAULT '1-col', pane_ids TEXT NOT NULL DEFAULT '[]'
|
||||
);
|
||||
|
||||
-- Legacy v2 terminal sessions.
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
type TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
shell TEXT,
|
||||
cwd TEXT,
|
||||
args TEXT,
|
||||
created_at INTEGER NOT NULL,
|
||||
last_used_at INTEGER NOT NULL,
|
||||
group_name TEXT DEFAULT '',
|
||||
project_id TEXT DEFAULT ''
|
||||
);
|
||||
|
||||
-- SSH connection profiles.
|
||||
CREATE TABLE IF NOT EXISTS ssh_sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
host TEXT NOT NULL,
|
||||
port INTEGER NOT NULL DEFAULT 22,
|
||||
username TEXT NOT NULL,
|
||||
key_file TEXT DEFAULT '',
|
||||
folder TEXT DEFAULT '',
|
||||
color TEXT DEFAULT '#89b4fa',
|
||||
created_at INTEGER NOT NULL,
|
||||
last_used_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- ── btmsg.db TABLES — inter-agent messaging & task board ──────────────
|
||||
|
||||
-- Registered agents (Tier 1 management + Tier 2 project).
|
||||
CREATE TABLE IF NOT EXISTS agents (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
role TEXT NOT NULL,
|
||||
group_id TEXT NOT NULL,
|
||||
tier INTEGER NOT NULL DEFAULT 2,
|
||||
model TEXT,
|
||||
cwd TEXT,
|
||||
system_prompt TEXT,
|
||||
status TEXT DEFAULT 'stopped',
|
||||
last_active_at TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Agent-to-agent visibility ACL.
|
||||
CREATE TABLE IF NOT EXISTS contacts (
|
||||
agent_id TEXT NOT NULL, contact_id TEXT NOT NULL,
|
||||
PRIMARY KEY (agent_id, contact_id)
|
||||
);
|
||||
|
||||
-- Direct messages between agents.
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
from_agent TEXT NOT NULL,
|
||||
to_agent TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
read INTEGER DEFAULT 0,
|
||||
reply_to TEXT,
|
||||
group_id TEXT NOT NULL,
|
||||
sender_group_id TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_messages_to ON messages(to_agent, read);
|
||||
CREATE INDEX IF NOT EXISTS idx_messages_from ON messages(from_agent);
|
||||
|
||||
-- Broadcast channels within a group.
|
||||
CREATE TABLE IF NOT EXISTS channels (
|
||||
id TEXT PRIMARY KEY, name TEXT NOT NULL, group_id TEXT NOT NULL,
|
||||
created_by TEXT NOT NULL, created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Channel membership (many-to-many).
|
||||
CREATE TABLE IF NOT EXISTS channel_members (
|
||||
channel_id TEXT NOT NULL, agent_id TEXT NOT NULL,
|
||||
joined_at TEXT DEFAULT (datetime('now')),
|
||||
PRIMARY KEY (channel_id, agent_id)
|
||||
);
|
||||
|
||||
-- Messages posted to channels.
|
||||
CREATE TABLE IF NOT EXISTS channel_messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
channel_id TEXT NOT NULL,
|
||||
from_agent TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_channel_messages ON channel_messages(channel_id, created_at);
|
||||
|
||||
-- Agent liveness heartbeats (unix epoch seconds).
|
||||
CREATE TABLE IF NOT EXISTS heartbeats (agent_id TEXT PRIMARY KEY, timestamp INTEGER NOT NULL);
|
||||
|
||||
-- Undeliverable messages (recipient not found, etc.)
|
||||
CREATE TABLE IF NOT EXISTS dead_letter_queue (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
from_agent TEXT NOT NULL,
|
||||
to_agent TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
error TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Audit trail for agent actions.
|
||||
CREATE TABLE IF NOT EXISTS audit_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
agent_id TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
detail TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Per-session message acknowledgment (prevents re-processing).
|
||||
CREATE TABLE IF NOT EXISTS seen_messages (
|
||||
session_id TEXT NOT NULL, message_id TEXT NOT NULL,
|
||||
seen_at INTEGER NOT NULL DEFAULT (unixepoch()),
|
||||
PRIMARY KEY (session_id, message_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_seen_messages_session ON seen_messages(session_id);
|
||||
|
||||
-- Kanban task board.
|
||||
CREATE TABLE IF NOT EXISTS tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
status TEXT DEFAULT 'todo'
|
||||
CHECK (status IN ('todo','progress','review','done','blocked')),
|
||||
priority TEXT DEFAULT 'medium'
|
||||
CHECK (priority IN ('low','medium','high')),
|
||||
assigned_to TEXT,
|
||||
created_by TEXT NOT NULL,
|
||||
group_id TEXT NOT NULL,
|
||||
parent_task_id TEXT,
|
||||
sort_order INTEGER DEFAULT 0,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
updated_at TEXT DEFAULT (datetime('now')),
|
||||
version INTEGER DEFAULT 1 -- optimistic locking
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_tasks_group ON tasks(group_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks(status);
|
||||
|
||||
-- Task discussion comments.
|
||||
CREATE TABLE IF NOT EXISTS task_comments (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL,
|
||||
agent_id TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_task_comments_task ON task_comments(task_id);
|
||||
|
||||
-- ── search.db TABLES — FTS5 full-text search ─────────────────────────
|
||||
|
||||
-- Agent message search index.
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS search_messages USING fts5(
|
||||
session_id,
|
||||
role,
|
||||
content,
|
||||
timestamp
|
||||
);
|
||||
|
||||
-- Task search index.
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS search_tasks USING fts5(
|
||||
task_id,
|
||||
title,
|
||||
description,
|
||||
status,
|
||||
assigned_to
|
||||
);
|
||||
|
||||
-- Inter-agent message search index.
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS search_btmsg USING fts5(
|
||||
message_id,
|
||||
from_agent,
|
||||
to_agent,
|
||||
content,
|
||||
channel_name
|
||||
);
|
||||
219
tools/migrate-db.ts
Normal file
219
tools/migrate-db.ts
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
#!/usr/bin/env bun
|
||||
/**
|
||||
* migrate-db.ts — Migrate AGOR data from a Tauri (source) database to an
|
||||
* Electrobun (target) database using the canonical schema.
|
||||
*
|
||||
* Usage:
|
||||
* bun tools/migrate-db.ts --from <source.db> --to <target.db>
|
||||
* bun tools/migrate-db.ts --from ~/.local/share/agor/sessions.db \
|
||||
* --to ~/.config/agor/settings.db
|
||||
*
|
||||
* Behavior:
|
||||
* - Opens source DB read-only (never modifies it).
|
||||
* - Creates/opens target DB, applies canonical.sql if schema_version absent.
|
||||
* - Copies rows for every table present in BOTH source and target.
|
||||
* - Wraps all inserts in a single transaction (atomic rollback on failure).
|
||||
* - Reports per-table row counts.
|
||||
* - Writes migration fence to schema_version in target.
|
||||
*/
|
||||
|
||||
import { Database } from "bun:sqlite";
|
||||
import { readFileSync, existsSync } from "fs";
|
||||
import { join, resolve } from "path";
|
||||
|
||||
// ── CLI args ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function usage(): never {
|
||||
console.error("Usage: bun tools/migrate-db.ts --from <source.db> --to <target.db>");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
let fromPath = "";
|
||||
let toPath = "";
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === "--from" && args[i + 1]) fromPath = args[++i];
|
||||
else if (args[i] === "--to" && args[i + 1]) toPath = args[++i];
|
||||
else if (args[i] === "--help" || args[i] === "-h") usage();
|
||||
}
|
||||
|
||||
if (!fromPath || !toPath) usage();
|
||||
|
||||
fromPath = resolve(fromPath);
|
||||
toPath = resolve(toPath);
|
||||
|
||||
if (!existsSync(fromPath)) {
|
||||
console.error(`Source database not found: ${fromPath}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Load canonical DDL ────────────────────────────────────────────────────────
|
||||
|
||||
const schemaPath = join(import.meta.dir, "..", "schema", "canonical.sql");
|
||||
let ddl: string;
|
||||
try {
|
||||
ddl = readFileSync(schemaPath, "utf-8");
|
||||
} catch (err) {
|
||||
console.error(`Failed to read canonical schema: ${err}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Open databases ────────────────────────────────────────────────────────────
|
||||
|
||||
const sourceDb = new Database(fromPath, { readonly: true });
|
||||
const targetDb = new Database(toPath);
|
||||
|
||||
// Apply pragmas to target
|
||||
targetDb.exec("PRAGMA journal_mode = WAL");
|
||||
targetDb.exec("PRAGMA foreign_keys = OFF"); // Disable during migration for insert order flexibility
|
||||
targetDb.exec("PRAGMA busy_timeout = 5000");
|
||||
|
||||
// Apply canonical schema to target if needed
|
||||
const hasVersion = (() => {
|
||||
try {
|
||||
const row = targetDb
|
||||
.query<{ cnt: number }, []>("SELECT COUNT(*) AS cnt FROM schema_version")
|
||||
.get();
|
||||
return (row?.cnt ?? 0) > 0;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})();
|
||||
|
||||
if (!hasVersion) {
|
||||
console.log("Applying canonical schema to target database...");
|
||||
targetDb.exec(ddl);
|
||||
}
|
||||
|
||||
// ── Discover migratable tables ────────────────────────────────────────────────
|
||||
|
||||
/** Get regular (non-virtual, non-internal) table names from a database. */
|
||||
function getTableNames(db: Database): Set<string> {
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT name FROM sqlite_master
|
||||
WHERE type = 'table'
|
||||
AND name NOT LIKE 'sqlite_%'
|
||||
AND name NOT LIKE '%_content'
|
||||
AND name NOT LIKE '%_data'
|
||||
AND name NOT LIKE '%_idx'
|
||||
AND name NOT LIKE '%_config'
|
||||
AND name NOT LIKE '%_docsize'
|
||||
ORDER BY name`,
|
||||
)
|
||||
.all() as Array<{ name: string }>;
|
||||
return new Set(rows.map((r) => r.name));
|
||||
}
|
||||
|
||||
const sourceTables = getTableNames(sourceDb);
|
||||
const targetTables = getTableNames(targetDb);
|
||||
|
||||
// Only migrate tables present in both source and target
|
||||
const migratable = [...sourceTables].filter((t) => targetTables.has(t));
|
||||
|
||||
if (migratable.length === 0) {
|
||||
console.log("No overlapping tables found between source and target.");
|
||||
sourceDb.close();
|
||||
targetDb.close();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log(`\nMigrating ${migratable.length} tables from:`);
|
||||
console.log(` source: ${fromPath}`);
|
||||
console.log(` target: ${toPath}\n`);
|
||||
|
||||
// ── Migrate data ──────────────────────────────────────────────────────────────
|
||||
|
||||
interface MigrationResult {
|
||||
table: string;
|
||||
rows: number;
|
||||
skipped: boolean;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
const results: MigrationResult[] = [];
|
||||
|
||||
const migrate = targetDb.transaction(() => {
|
||||
for (const table of migratable) {
|
||||
// Skip schema_version — we write our own fence
|
||||
if (table === "schema_version") {
|
||||
results.push({ table, rows: 0, skipped: true });
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
// Read all rows from source
|
||||
const rows = sourceDb.prepare(`SELECT * FROM "${table}"`).all();
|
||||
|
||||
if (rows.length === 0) {
|
||||
results.push({ table, rows: 0, skipped: false });
|
||||
continue;
|
||||
}
|
||||
|
||||
// Get column names from the first row
|
||||
const columns = Object.keys(rows[0] as Record<string, unknown>);
|
||||
const placeholders = columns.map(() => "?").join(", ");
|
||||
const colList = columns.map((c) => `"${c}"`).join(", ");
|
||||
|
||||
const insertStmt = targetDb.prepare(
|
||||
`INSERT OR IGNORE INTO "${table}" (${colList}) VALUES (${placeholders})`,
|
||||
);
|
||||
|
||||
let count = 0;
|
||||
for (const row of rows) {
|
||||
const values = columns.map((c) => (row as Record<string, unknown>)[c] ?? null);
|
||||
insertStmt.run(...values);
|
||||
count++;
|
||||
}
|
||||
|
||||
results.push({ table, rows: count, skipped: false });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
results.push({ table, rows: 0, skipped: false, error: msg });
|
||||
throw new Error(`Migration failed on table '${table}': ${msg}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
migrate();
|
||||
} catch (err) {
|
||||
console.error(`\nMIGRATION ROLLED BACK: ${err}`);
|
||||
sourceDb.close();
|
||||
targetDb.close();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Write version fence ───────────────────────────────────────────────────────
|
||||
|
||||
const timestamp = new Date().toISOString();
|
||||
targetDb.exec("DELETE FROM schema_version");
|
||||
targetDb
|
||||
.prepare(
|
||||
"INSERT INTO schema_version (version, migration_source, migration_timestamp) VALUES (?, ?, ?)",
|
||||
)
|
||||
.run(1, "migrate-db", timestamp);
|
||||
|
||||
// Re-enable foreign keys
|
||||
targetDb.exec("PRAGMA foreign_keys = ON");
|
||||
|
||||
// ── Report ────────────────────────────────────────────────────────────────────
|
||||
|
||||
console.log("Table Rows Status");
|
||||
console.log("─".repeat(50));
|
||||
|
||||
let totalRows = 0;
|
||||
for (const r of results) {
|
||||
const status = r.skipped ? "skipped" : r.error ? `ERROR: ${r.error}` : "ok";
|
||||
const rowStr = String(r.rows).padStart(8);
|
||||
console.log(`${r.table.padEnd(25)}${rowStr} ${status}`);
|
||||
totalRows += r.rows;
|
||||
}
|
||||
|
||||
console.log("─".repeat(50));
|
||||
console.log(`Total: ${totalRows} rows migrated across ${results.filter((r) => !r.skipped && !r.error).length} tables`);
|
||||
console.log(`Version fence: v1 at ${timestamp}`);
|
||||
|
||||
sourceDb.close();
|
||||
targetDb.close();
|
||||
109
tools/validate-schema.ts
Normal file
109
tools/validate-schema.ts
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
#!/usr/bin/env bun
|
||||
/**
|
||||
* validate-schema.ts — Apply canonical.sql to an in-memory SQLite database
|
||||
* and extract structural metadata for CI comparison.
|
||||
*
|
||||
* Usage: bun tools/validate-schema.ts
|
||||
* Output: JSON to stdout with tables, columns, indexes, and schema version.
|
||||
*/
|
||||
|
||||
import { Database } from "bun:sqlite";
|
||||
import { readFileSync } from "fs";
|
||||
import { join } from "path";
|
||||
|
||||
// ── Load canonical DDL ────────────────────────────────────────────────────────
|
||||
|
||||
const schemaPath = join(import.meta.dir, "..", "schema", "canonical.sql");
|
||||
let ddl: string;
|
||||
try {
|
||||
ddl = readFileSync(schemaPath, "utf-8");
|
||||
} catch (err) {
|
||||
console.error(`Failed to read ${schemaPath}: ${err}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Apply to in-memory DB ─────────────────────────────────────────────────────
|
||||
|
||||
const db = new Database(":memory:");
|
||||
|
||||
try {
|
||||
db.exec(ddl);
|
||||
} catch (err) {
|
||||
console.error(`Schema application failed: ${err}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// ── Extract metadata ──────────────────────────────────────────────────────────
|
||||
|
||||
interface ColumnInfo {
|
||||
cid: number;
|
||||
name: string;
|
||||
type: string;
|
||||
notnull: number;
|
||||
dflt_value: string | null;
|
||||
pk: number;
|
||||
}
|
||||
|
||||
interface TableMeta {
|
||||
name: string;
|
||||
type: string; // 'table' | 'virtual'
|
||||
columns: ColumnInfo[];
|
||||
indexes: string[];
|
||||
}
|
||||
|
||||
// Get all tables and virtual tables from sqlite_master
|
||||
const masterRows = db
|
||||
.prepare(
|
||||
`SELECT name, type FROM sqlite_master
|
||||
WHERE type IN ('table', 'virtual table')
|
||||
AND name NOT LIKE 'sqlite_%'
|
||||
AND name NOT LIKE '%_content'
|
||||
AND name NOT LIKE '%_data'
|
||||
AND name NOT LIKE '%_idx'
|
||||
AND name NOT LIKE '%_config'
|
||||
AND name NOT LIKE '%_docsize'
|
||||
ORDER BY name`,
|
||||
)
|
||||
.all() as Array<{ name: string; type: string }>;
|
||||
|
||||
const tables: TableMeta[] = [];
|
||||
|
||||
for (const { name, type } of masterRows) {
|
||||
// Get column info (not available for FTS5 virtual tables)
|
||||
let columns: ColumnInfo[] = [];
|
||||
try {
|
||||
columns = db
|
||||
.prepare(`PRAGMA table_info('${name}')`)
|
||||
.all() as ColumnInfo[];
|
||||
} catch {
|
||||
// FTS5 tables don't support table_info
|
||||
}
|
||||
|
||||
// Get indexes for this table
|
||||
const indexRows = db
|
||||
.prepare(
|
||||
`SELECT name FROM sqlite_master
|
||||
WHERE type = 'index' AND tbl_name = ?
|
||||
ORDER BY name`,
|
||||
)
|
||||
.all(name) as Array<{ name: string }>;
|
||||
|
||||
tables.push({
|
||||
name,
|
||||
type: type === "table" ? "table" : "virtual",
|
||||
columns,
|
||||
indexes: indexRows.map((r) => r.name),
|
||||
});
|
||||
}
|
||||
|
||||
// ── Output ────────────────────────────────────────────────────────────────────
|
||||
|
||||
const output = {
|
||||
schemaFile: "schema/canonical.sql",
|
||||
version: 1,
|
||||
tableCount: tables.length,
|
||||
tables,
|
||||
};
|
||||
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
db.close();
|
||||
Loading…
Add table
Add a link
Reference in a new issue