gtfs.zone/coloring-book

Fork 0

Improve load experience with refresh page with existing feed #127

New issue

Closed

opened 2026-05-24 22:05:37 +00:00 by maxtkc · 0 comments

maxtkc commented

2026-05-24 22:05:37 +00:00

Owner

Summary

Refreshing the page with a large feed (think MBTA, BART, NYC subway) freezes the tab for many seconds with no visual feedback — the user thinks the app is broken. The frozen time is dominated by main-thread work that has nothing to do with rendering: re-parsing CSV blobs out of IndexedDB, walking every row in the validator, and rebuilding the same in-memory indexes twice (once from blobs, again from snapshot + patches).

We will (1) add the existing FeedProgressIndicator to the refresh path so the user sees that something is happening, (2) replace CSV blobs with JSON blobs to make restoreDataFromDatabase 3–10× faster and eliminate the redundant processParsedData pass, (3) move the parse off the main thread into a worker, (4) parallelize independent IDB reads, (5) drop the startup validator (its results are not displayed anyway), and (6) eliminate the snapshot-vs-blob redundancy by tracking the version a blob was last persisted at.

Scope is deliberately limited to the restore + first-paint path. The route renderer's createRouteFeatures is also slow but stays out of scope — we want the user to at least see the page before the map indexes are built.

Tradeoffs:

Switching blob format to JSON bumps the DB schema version, so existing locally-stored feeds will be wiped (users will be prompted to export, then must re-import). Per CLAUDE.md ("No backwards-compatibility hacks") this is acceptable.
Removing startup validation means a feed with bad data won't show validator output until the user explicitly opens the validation panel. That output currently goes nowhere (InfoDisplay is not used in the new UI structure), so this is strictly a wash.

Relevant context

Architecture

The app boots via GTFSEditor.init() in src/index.ts:127. Module instantiation is synchronous in the constructor; async work happens in init(). There is no central state — modules own their state and the canonical store is IndexedDB (GTFSDatabase).

Current restore path (the slow path on refresh)

GTFSEditor.init() (src/index.ts:127) — orchestrates everything below, sequentially with await.
GTFSParser.initialize() (src/modules/gtfs-parser.ts:459):
- Calls gtfsDatabase.initialize() — opens IndexedDB.
- Seeds gtfsData with header-only CSVs for every known file (cheap).
- Calls restoreDataFromDatabase() (src/modules/gtfs-parser.ts:483):
  - Iterates ALL_GTFS_FILES sequentially with await. For each .txt file:
    - await this.gtfsDatabase.getTableBlob(tableName) — IDB read.
    - Papa.parse(csv, { header: true, skipEmptyLines: true }) — synchronous, main thread. For a 100 MB stop_times.txt this is multi-second.
    - processParsedData(parsed.data) — .map() over every row, walks every field, runs parseFieldValue (a long if-tree on field names) to coerce types. Allocates a brand new object per row.
    - setupVirtual(tableName, data) — builds byId Map (composite key per row) + any fieldMaps (e.g. stop_times indexed by trip_id and stop_id).
PatchManager.initialize() (src/modules/patch-manager.ts:42):
- Reads getVersions(), getLatestSnapshot().
- If a snapshot exists: decompresses it (gzip via DecompressionStream), JSON.parse, then for each table: db.clearTable(table) (wipes the just-built virtual table) → db.insertRows(table, rows) (rebuilds it) → parser.setInMemoryFileData(...) (calls setupVirtual yet again with the new array). This is pure duplicate work whenever the blob is already current.
- Replays patches from snapshot.version + 1 to currentVersion via applyPatchForward (each patch is one virtual-table mutation).
MapController.initialize() (src/modules/map-controller.ts:85):
- Creates MapLibre map + LayerManager + RouteRenderer + InteractionHandler + BasemapControl. RouteRenderer kicks off layer-init in the background; main-thread cost here is small.
validateAndUpdateInfo() (src/index.ts:365):
- Runs gtfsValidator.validateFeed() synchronously. This iterates every row of stop_times and runs per-field validation (regex-backed validateValue calls). For a 2M-row feed this is many seconds of main-thread work — and the comment at src/index.ts:369 says the result is not displayed anywhere.
pageStateManager.initializeFromURL() + processURLCommands + mapController.updateMap():
- updateMap() → routeRenderer.renderRoutes() → createRouteFeatures() (src/modules/route-renderer.ts:202). This walks every shape point, every stop_time, every trip, every route. Also expensive, but out of scope for this plan — it runs after the first paint and the user has already seen the UI.
browseNavigation.refresh() — renders the sidebar list. Fast.

Key files & functions

src/index.ts:127 — GTFSEditor.init() (orchestration)
src/modules/gtfs-parser.ts:459 — GTFSParser.initialize()
src/modules/gtfs-parser.ts:483 — GTFSParser.restoreDataFromDatabase() (the main hot path)
src/modules/gtfs-parser.ts:407 — GTFSParser.persistDirtyBlobs() (debounced write path; needs to switch to JSON too)
src/modules/gtfs-parser.ts:425 — GTFSParser.generateCSVFromRows() (delete after switch)
src/modules/gtfs-parser.ts:128 — GTFSParser.processParsedData() (delete after switch — only used for parsing CSV from blob, which goes away)
src/modules/gtfs-parser.ts:64 — GTFSParser.parseFieldValue() (delete after switch — same reason)
src/modules/gtfs-database.ts:1027 — saveTableBlob(tableName, csv: string) — change signature to JSON
src/modules/gtfs-database.ts:1038 — getTableBlob(tableName): string | null — change signature to JSON
src/modules/gtfs-database.ts:213 — GTFSDatabase.initialize() (DB version bump)
src/modules/gtfs-database.ts:155 (approx) — file_blobs store schema
src/modules/feed-progress-indicator.ts — already exists, just needs to be used during restore
src/modules/patch-manager.ts:42 — PatchManager.initialize() (snapshot vs blob dedup)
src/modules/gtfs-validator.ts:448 — validateStopTimes() (offending row loop)
src/workers/gtfs-parser.worker.ts — existing import worker; we will add a sibling gtfs-restore.worker.ts (see Phase 3 gotcha)

Invariants to preserve

Copy-on-read (CLAUDE.md): all virtual-table getAll/getById/query calls must return shallow copies.
Shared-array invariant (gtfs-parser.ts:160): gtfsData[fileName].data must be the same array reference as the virtual table's flat array. After restoration, both must point at the same in-memory array.
There is always a feed: if no rows after restore, initializeEmpty() runs (src/index.ts:336). The new path must still satisfy getAllFileNames()/getFileDataSync() checks.
Patch replay correctness: virtual-table insert dedups by byId, delete is no-op when missing, so replaying patches that were already applied (because blob was current) remains safe. This invariant lets us skip patch replay as an optimization (Phase 5) but doesn't require us to.
DB version bump: gtfs-database.ts:213 already handles version upgrades with an "Export & Continue / Clear & Continue" modal — we get the user-facing migration UX for free.

What we are not doing

Not changing the new-feed parseFile() import path. It already uses a worker and already drives FeedProgressIndicator. Only minor signature changes (writing JSON blobs instead of CSV blobs).
Not touching RouteRenderer.createRouteFeatures() or LayerManager.addStopsLayer(). They are slow but happen after the UI is interactive.
Not adding incremental rendering to the editor table or browse navigation. Out of scope.
Not refactoring the patch system. The snapshot/blob dedup is a small additive change, not a rewrite.

Phase 1 — Show progress during restore

The user's primary complaint is "the page hangs." Even before we make the restore faster, showing that work is happening reframes the experience entirely. This is the smallest, lowest-risk change and ships value on its own.

Goal: FeedProgressIndicator is visible from the moment GTFSEditor.init() runs and updates through restore phases.

In src/index.ts:127 GTFSEditor.init(), call feedProgressIndicator.startLoading('boot', 'Opening database...') as the very first statement of the try block (before tabLock.init()).
After await this.gtfsParser.initialize(), call feedProgressIndicator.updateProgress('boot', 60, 'Restoring patches...').
After await this.patchManager.initialize(), call feedProgressIndicator.updateProgress('boot', 80, 'Building map...').
After await this.mapController.updateMap(), call feedProgressIndicator.finishLoading('boot').
Ensure finishLoading('boot') is also called from the catch block so a failed init doesn't leave the bar stuck.
Inside GTFSParser.restoreDataFromDatabase() (src/modules/gtfs-parser.ts:483), after each table is restored, call feedProgressIndicator.updateProgress('boot', 5 + (idx / total) * 50, 'Restored ${tableName}...'). The 5–55 % band leaves room for Phases 2–3 to subdivide further.
Add an early-skip in restoreDataFromDatabase if the table blob is empty/null so the progress bar advances even for absent files.

Gotcha: FeedProgressIndicator is keyed on operation name; 'boot' is a new key — make sure it doesn't collide with the existing 'parseFile' key, which can fire concurrently if the user has a URL command (#load=) during refresh.

Gotcha: feedProgressIndicator is a module-level singleton (src/modules/feed-progress-indicator.ts:105), so it constructs and appends its DOM on first import. That happens during GTFSEditor construction (transitively) — verify the indicator's DOM is in place before init() runs (it should be, but worth a console.log in the first call).

Phase 2 — Drop startup validator

Easiest measurable win. validateAndUpdateInfo() loops every stop_times row, never displays the result, and is called twice (once in init() and once as a callback passed to uiController.initialize).

Goal: zero validator work during boot. Validator class stays callable for future on-demand use.

In src/index.ts:324, delete the this.validateAndUpdateInfo(); call.
In src/index.ts:157, the validateAndUpdateInfo.bind(this) is passed to uiController.initialize as the onValidate callback. Trace where uiController invokes it — if it's only edit-driven (not load-driven), leave the callback intact. If it's never used, remove the parameter from the uiController.initialize signature.
Keep GTFSEditor.validateAndUpdateInfo() (src/index.ts:365) so manual callers still work, but add a // Not called on startup — invoke manually if the validation panel is opened. comment (the one allowed comment per CLAUDE.md rules: the WHY is non-obvious).
If uiController.initialize ends up not needing the callback, also remove the unused validator field if nothing else references it.

Gotcha: don't delete the validator module — it's still imported in tests / could be wired up later. Only delete the startup invocation.

Gotcha: search for any other callers of validateAndUpdateInfo or validator.validateFeed to make sure removing the startup call doesn't break a code path that depended on the side-effect of the validator running first (it shouldn't — the validator is pure).

Phase 3 — Switch blob format from CSV to JSON, and parse off the main thread

This is the biggest perceived win. CSV blobs require Papa.parse + processParsedData — the first is single-threaded text parsing, the second is per-row per-field allocation. JSON blobs go through native JSON.parse (highly optimized in V8) and the rows come back already type-coerced, so processParsedData disappears entirely.

This phase has two halves: change what we persist, and move the parse into a worker.

3a. Change persisted blob format to JSON

In src/modules/gtfs-database.ts, change the file_blobs store schema (around line 156 / line 278) from { tableName, csv } to { tableName, json }. Update the FileBlob interface.
Change saveTableBlob(tableName: string, csv: string) (src/modules/gtfs-database.ts:1027) to saveTableBlob(tableName: string, json: string). Same body, just stores under json key.
Change getTableBlob(tableName: string): Promise<string | null> (src/modules/gtfs-database.ts:1038) to return the json field.
In src/modules/gtfs-parser.ts:407 persistDirtyBlobs(), replace const csv = this.generateCSVFromRows(fileName, rows) with const json = JSON.stringify(rows).
In src/modules/gtfs-parser.ts:639 parseFile() (the new-import path), replace await this.gtfsDatabase.saveTableBlob(tableName, fileResult.rawContent) (which writes the raw CSV) with await this.gtfsDatabase.saveTableBlob(tableName, JSON.stringify(fileResult.data)).
Delete GTFSParser.generateCSVFromRows() (src/modules/gtfs-parser.ts:425) and GTFSParser.processParsedData() (src/modules/gtfs-parser.ts:128) and GTFSParser.parseFieldValue() (src/modules/gtfs-parser.ts:64). Search for references first — they're also called from updateFileInMemory (src/modules/gtfs-parser.ts:755) and updateFileContent (src/modules/gtfs-parser.ts:699), which do still need to parse CSV (user pastes CSV into the editor). For those call sites, keep parseFieldValue/processParsedData — only delete generateCSVFromRows if no caller remains. (Revise as needed during implementation; the goal is "no CSV in the blob round-trip" not "no CSV code at all".)
In src/modules/gtfs-database.ts:213 initialize(), bump this.dbVersion. The existing upgrade flow already handles wiping + offering export.

3b. Move blob restore into a worker

The new-import path already uses a worker (src/workers/gtfs-parser.worker.ts). Restore is structurally similar (parse strings → emit row arrays) so we add a new message type rather than a new file.

In src/workers/gtfs-parser.worker.ts, add a new inbound message type { type: 'restore'; blobs: { tableName: string; json: string }[] } and a corresponding handler.
The handler iterates blobs, calls JSON.parse(json) for each, and posts { type: 'progress', progress, status: 'Restoring ${tableName}...' } between tables, then a final { type: 'done-restore', tables: { [tableName]: rows[] } }.
In GTFSParser.restoreDataFromDatabase() (src/modules/gtfs-parser.ts:483), rewrite the body:
1. Promise.all over ALL_GTFS_FILES to fetch all getTableBlob(...) calls in parallel (independent IDB reads).
2. Filter out null/empty blobs and .geojson files (those still go through their existing path).
3. Spawn the worker, post { type: 'restore', blobs }, hook onmessage to forward progress to feedProgressIndicator and resolve on done-restore.
4. On resolve, iterate tables and for each: set this.gtfsData[fileName] = { content: '', data: rows, errors: [] } and this.setupVirtual(tableName, rows). Preserve the shared-array invariant (rows passed to setupVirtual must be the same array stored in gtfsData[fileName].data).
5. Terminate the worker.
Keep the .geojson handling unchanged — those files are tiny and per-table.

Gotcha (critical): the shared-array invariant. setupVirtual's flat parameter is stored as a live reference (gtfs-parser.ts:163 doc comment). After the worker returns, the deserialized array from JSON.parse is a fresh array — that's fine, but gtfsData[fileName].data and the flat passed to setupVirtual must be the same reference, not copies. Mistake would corrupt byId silently on the next edit.

Gotcha: workers don't share memory; the JSON strings get structured-cloned across the boundary. For very large blobs (100 MB+ JSON for stop_times) the clone has measurable cost. Mitigate by transferring ArrayBuffers instead: store blobs as Uint8Array containing UTF-8 JSON bytes, transfer with postMessage(msg, [buffer]). Defer this optimization — start with plain string transfer, measure, optimize only if string-cloning is a bottleneck.

Gotcha: the IDB upgrade modal at gtfs-database.ts:231 shows an "Export & Continue" option which calls exportCurrentBlobsAsZip. If that exporter assumes the csv field on file_blobs, it will break for users upgrading from the old schema. The export reads the old schema before the upgrade fires, so it should still see csv-keyed records — verify this is true (the peekVersion call at gtfs-database.ts:229 opens the DB without triggering upgrade). If exportCurrentBlobsAsZip opens a new DB connection that triggers upgrade-and-wipe before reading, fix it to read the raw blobs via a non-upgrading open.

Phase 4 — Defer non-critical post-restore work

After the restore finishes, init() does a lot of sequential work before the user sees anything: mapController.updateMap(), browseNavigation.refresh(), uiController.updateFileList(). Reordering these to favor first paint reduces perceived load time even when total time is unchanged.

Goal: the user sees the breadcrumb / sidebar shell as soon as possible, even if the map is still rendering.

After pageStateManager.initializeFromURL() (src/index.ts:330), call browseNavigation.refresh() immediately (don't wait for map) and uiController.updateFileList() right after. These are fast and give the user something to look at.
Wrap mapController.updateMap() in a requestIdleCallback (or requestAnimationFrame fallback) so the browser can paint the sidebar before kicking off the heavy index build.
Remove the trailing if (this.browseNavigation) { this.browseNavigation.refresh(); } block (src/index.ts:347) — redundant after the reorder.
The exportBtn.disabled = false enable (src/index.ts:355) should happen as soon as gtfsParser.initialize() resolves, not at the end of init() — the user can export the moment data is in memory.

Gotcha: browseNavigation.refresh() calls getPageStateManager().getBreadcrumbs() which depends on pageStateManager.initializeFromURL() having run. Order: initializeFromURL → browseNavigation.refresh() → idle-callback for updateMap(). Keep that order.

Gotcha: requestIdleCallback isn't on Safari < 17. Fall back to setTimeout(fn, 0) if unavailable. Keep it simple — a 4-line runWhenIdle helper at the top of index.ts.

Phase 5 — Eliminate snapshot/blob redundancy

When both a snapshot and a current blob exist, PatchManager.initialize() discards the blob's data (clearTable + insertRows + setInMemoryFileData) and rebuilds from the snapshot — then replays patches on top. If the blob was already current (i.e. persisted after the last patch), all this work is wasted.

Goal: track the version each blob was persisted at; skip snapshot/patch replay when blob is current.

Add a blobVersion field to the meta store: { key: 'blobVersion', version: number }. After persistDirtyBlobs() (src/modules/gtfs-parser.ts:407) completes, write the current patch version into meta.blobVersion. The PatchManager exposes version via patchManager.version (src/modules/patch-manager.ts:413); pass it into persistDirtyBlobs as an argument so the parser doesn't need a back-reference.
In PatchManager.initialize() (src/modules/patch-manager.ts:42), after reading currentVersion/headVersion, also read meta.blobVersion. If blobVersion === currentVersion, skip the entire snapshot-restore + patch-replay block. The in-memory state from GTFSParser.initialize() is already correct.
Otherwise, fall through to the existing snapshot + patch-replay path unchanged.
In applyPatchForward (src/modules/patch-manager.ts:92), no changes needed — replay correctness relies on virtual-table dedup which is unchanged.
In parseFile() (src/modules/gtfs-parser.ts:556 — new-import path), after the loop that writes blobs, also write meta.blobVersion = 0 (a fresh import has no patches yet). Otherwise the next refresh will think the blob is stale.
In initializeEmpty() (src/modules/gtfs-parser.ts:525), set meta.blobVersion = 0 after persistDirtyBlobs().

Gotcha: there is a 3-second debounce on blob persistence. So a user who edits then immediately refreshes will have blobVersion < currentVersion and will correctly fall into the snapshot+replay path. The optimization only kicks in when at least 3 seconds have passed since the last edit. That's fine — it's the common case (refresh after the user has been idle).

Gotcha: if persistDirtyBlobs writes partial blobs (some tables in blobDirty, some not), blobVersion should only advance if every dirty table flushed successfully. Wrap the write + version update in the same operation, and on partial failure leave blobVersion untouched so the next load falls back to snapshot+replay (which is correct).

Gotcha: snapshots run via maybeSnapshot() (src/modules/patch-manager.ts:480) every CONFIG.SNAPSHOT_INTERVAL (50) patches. They are not blob-aware. With this optimization, snapshots are only consulted on the slow path — which is fine; they remain a correctness fallback. No change needed to the snapshot cadence.

Phase 6 — Verify and measure

Easy to feel good about a "performance" patch that doesn't measure. Add the timing logs we need to confirm the win on a real large feed.

Add console.time('[boot] total') at the start of GTFSEditor.init() and console.timeEnd at the end (both success and failure paths).
Add console.time around each major stage: gtfs-parser.initialize, patch-manager.initialize, map-controller.updateMap, browse-navigation.refresh.
Inside restoreDataFromDatabase, log per-table parse times for any table over 1 MB JSON.
Manually test with three feeds: a small one (BART), a medium one (MBTA), and a large one (NYC subway or whatever the user has loaded). Confirm the progress bar appears, advances, and the page is interactive in a reasonable time.
Confirm IDB upgrade modal fires for users on the old schema and that "Export & Continue" yields a valid GTFS zip.
Confirm that after a normal load, a refresh with no edits is "blob-current" fast (Phase 5 should kick in).
Confirm that a refresh within 3 seconds of an edit still produces correct state (Phase 5 should not kick in; snapshot+replay path runs).

Gotcha: don't leave the console.time calls in the shipped code. Either gate them behind a DEBUG_BOOT flag in CONFIG or remove before merging.

## Summary Refreshing the page with a large feed (think MBTA, BART, NYC subway) freezes the tab for many seconds with no visual feedback — the user thinks the app is broken. The frozen time is dominated by main-thread work that has nothing to do with rendering: re-parsing CSV blobs out of IndexedDB, walking every row in the validator, and rebuilding the same in-memory indexes twice (once from blobs, again from snapshot + patches). We will (1) add the existing `FeedProgressIndicator` to the refresh path so the user sees that something is happening, (2) replace CSV blobs with JSON blobs to make `restoreDataFromDatabase` 3–10× faster and eliminate the redundant `processParsedData` pass, (3) move the parse off the main thread into a worker, (4) parallelize independent IDB reads, (5) drop the startup validator (its results are not displayed anyway), and (6) eliminate the snapshot-vs-blob redundancy by tracking the version a blob was last persisted at. Scope is deliberately limited to the restore + first-paint path. The route renderer's `createRouteFeatures` is also slow but stays out of scope — we want the user to at least *see* the page before the map indexes are built. Tradeoffs: - Switching blob format to JSON bumps the DB schema version, so existing locally-stored feeds will be wiped (users will be prompted to export, then must re-import). Per CLAUDE.md ("No backwards-compatibility hacks") this is acceptable. - Removing startup validation means a feed with bad data won't show validator output until the user explicitly opens the validation panel. That output currently goes nowhere (`InfoDisplay is not used in the new UI structure`), so this is strictly a wash. ## Relevant context ### Architecture The app boots via `GTFSEditor.init()` in `src/index.ts:127`. Module instantiation is synchronous in the constructor; async work happens in `init()`. There is no central state — modules own their state and the canonical store is IndexedDB (`GTFSDatabase`). ### Current restore path (the slow path on refresh) 1. **`GTFSEditor.init()`** (`src/index.ts:127`) — orchestrates everything below, sequentially with `await`. 2. **`GTFSParser.initialize()`** (`src/modules/gtfs-parser.ts:459`): - Calls `gtfsDatabase.initialize()` — opens IndexedDB. - Seeds `gtfsData` with header-only CSVs for every known file (cheap). - Calls `restoreDataFromDatabase()` (`src/modules/gtfs-parser.ts:483`): - Iterates `ALL_GTFS_FILES` **sequentially** with `await`. For each `.txt` file: - `await this.gtfsDatabase.getTableBlob(tableName)` — IDB read. - `Papa.parse(csv, { header: true, skipEmptyLines: true })` — **synchronous, main thread**. For a 100 MB `stop_times.txt` this is multi-second. - `processParsedData(parsed.data)` — `.map()` over every row, walks every field, runs `parseFieldValue` (a long if-tree on field names) to coerce types. Allocates a brand new object per row. - `setupVirtual(tableName, data)` — builds `byId` Map (composite key per row) + any `fieldMaps` (e.g. `stop_times` indexed by `trip_id` and `stop_id`). 3. **`PatchManager.initialize()`** (`src/modules/patch-manager.ts:42`): - Reads `getVersions()`, `getLatestSnapshot()`. - **If a snapshot exists**: decompresses it (gzip via `DecompressionStream`), `JSON.parse`, then for each table: `db.clearTable(table)` (wipes the just-built virtual table) → `db.insertRows(table, rows)` (rebuilds it) → `parser.setInMemoryFileData(...)` (calls `setupVirtual` *yet again* with the new array). **This is pure duplicate work whenever the blob is already current.** - Replays patches from `snapshot.version + 1` to `currentVersion` via `applyPatchForward` (each patch is one virtual-table mutation). 4. **`MapController.initialize()`** (`src/modules/map-controller.ts:85`): - Creates MapLibre map + LayerManager + RouteRenderer + InteractionHandler + BasemapControl. RouteRenderer kicks off layer-init in the background; main-thread cost here is small. 5. **`validateAndUpdateInfo()`** (`src/index.ts:365`): - Runs `gtfsValidator.validateFeed()` synchronously. This iterates **every** row of `stop_times` and runs per-field validation (regex-backed `validateValue` calls). For a 2M-row feed this is many seconds of main-thread work — and the comment at `src/index.ts:369` says the result is not displayed anywhere. 6. **`pageStateManager.initializeFromURL()`** + `processURLCommands` + `mapController.updateMap()`: - `updateMap()` → `routeRenderer.renderRoutes()` → `createRouteFeatures()` (`src/modules/route-renderer.ts:202`). This walks every shape point, every stop_time, every trip, every route. Also expensive, but **out of scope for this plan** — it runs after the first paint and the user has already seen the UI. 7. **`browseNavigation.refresh()`** — renders the sidebar list. Fast. ### Key files & functions - `src/index.ts:127` — `GTFSEditor.init()` (orchestration) - `src/modules/gtfs-parser.ts:459` — `GTFSParser.initialize()` - `src/modules/gtfs-parser.ts:483` — `GTFSParser.restoreDataFromDatabase()` (the main hot path) - `src/modules/gtfs-parser.ts:407` — `GTFSParser.persistDirtyBlobs()` (debounced write path; needs to switch to JSON too) - `src/modules/gtfs-parser.ts:425` — `GTFSParser.generateCSVFromRows()` (delete after switch) - `src/modules/gtfs-parser.ts:128` — `GTFSParser.processParsedData()` (delete after switch — only used for parsing CSV from blob, which goes away) - `src/modules/gtfs-parser.ts:64` — `GTFSParser.parseFieldValue()` (delete after switch — same reason) - `src/modules/gtfs-database.ts:1027` — `saveTableBlob(tableName, csv: string)` — change signature to JSON - `src/modules/gtfs-database.ts:1038` — `getTableBlob(tableName): string | null` — change signature to JSON - `src/modules/gtfs-database.ts:213` — `GTFSDatabase.initialize()` (DB version bump) - `src/modules/gtfs-database.ts:155` (approx) — `file_blobs` store schema - `src/modules/feed-progress-indicator.ts` — already exists, just needs to be used during restore - `src/modules/patch-manager.ts:42` — `PatchManager.initialize()` (snapshot vs blob dedup) - `src/modules/gtfs-validator.ts:448` — `validateStopTimes()` (offending row loop) - `src/workers/gtfs-parser.worker.ts` — existing import worker; we will add a sibling `gtfs-restore.worker.ts` (see Phase 3 gotcha) ### Invariants to preserve - **Copy-on-read** (CLAUDE.md): all virtual-table `getAll`/`getById`/`query` calls must return shallow copies. - **Shared-array invariant** (`gtfs-parser.ts:160`): `gtfsData[fileName].data` must be the *same* array reference as the virtual table's `flat` array. After restoration, both must point at the same in-memory array. - **There is always a feed**: if no rows after restore, `initializeEmpty()` runs (`src/index.ts:336`). The new path must still satisfy `getAllFileNames()/getFileDataSync()` checks. - **Patch replay correctness**: virtual-table `insert` dedups by `byId`, `delete` is no-op when missing, so replaying patches that were already applied (because blob was current) remains safe. This invariant lets us skip patch replay as an optimization (Phase 5) but doesn't require us to. - **DB version bump**: `gtfs-database.ts:213` already handles version upgrades with an "Export & Continue / Clear & Continue" modal — we get the user-facing migration UX for free. ### What we are *not* doing - Not changing the new-feed `parseFile()` import path. It already uses a worker and already drives `FeedProgressIndicator`. Only minor signature changes (writing JSON blobs instead of CSV blobs). - Not touching `RouteRenderer.createRouteFeatures()` or `LayerManager.addStopsLayer()`. They are slow but happen after the UI is interactive. - Not adding incremental rendering to the editor table or browse navigation. Out of scope. - Not refactoring the patch system. The snapshot/blob dedup is a small additive change, not a rewrite. --- ## Phase 1 — Show progress during restore The user's primary complaint is "the page hangs." Even before we make the restore faster, showing that work is happening reframes the experience entirely. This is the smallest, lowest-risk change and ships value on its own. **Goal**: `FeedProgressIndicator` is visible from the moment `GTFSEditor.init()` runs and updates through restore phases. - [ ] In `src/index.ts:127` `GTFSEditor.init()`, call `feedProgressIndicator.startLoading('boot', 'Opening database...')` as the very first statement of the `try` block (before `tabLock.init()`). - [ ] After `await this.gtfsParser.initialize()`, call `feedProgressIndicator.updateProgress('boot', 60, 'Restoring patches...')`. - [ ] After `await this.patchManager.initialize()`, call `feedProgressIndicator.updateProgress('boot', 80, 'Building map...')`. - [ ] After `await this.mapController.updateMap()`, call `feedProgressIndicator.finishLoading('boot')`. - [ ] Ensure `finishLoading('boot')` is also called from the `catch` block so a failed init doesn't leave the bar stuck. - [ ] Inside `GTFSParser.restoreDataFromDatabase()` (`src/modules/gtfs-parser.ts:483`), after each table is restored, call `feedProgressIndicator.updateProgress('boot', 5 + (idx / total) * 50, 'Restored ${tableName}...')`. The 5–55 % band leaves room for Phases 2–3 to subdivide further. - [ ] Add an early-skip in `restoreDataFromDatabase` if the table blob is empty/null so the progress bar advances even for absent files. **Gotcha**: `FeedProgressIndicator` is keyed on operation name; `'boot'` is a new key — make sure it doesn't collide with the existing `'parseFile'` key, which can fire concurrently if the user has a URL command (`#load=`) during refresh. **Gotcha**: `feedProgressIndicator` is a module-level singleton (`src/modules/feed-progress-indicator.ts:105`), so it constructs and appends its DOM on first import. That happens during `GTFSEditor` construction (transitively) — verify the indicator's DOM is in place before `init()` runs (it should be, but worth a `console.log` in the first call). --- ## Phase 2 — Drop startup validator Easiest measurable win. `validateAndUpdateInfo()` loops every `stop_times` row, never displays the result, and is called twice (once in `init()` and once as a callback passed to `uiController.initialize`). **Goal**: zero validator work during boot. Validator class stays callable for future on-demand use. - [ ] In `src/index.ts:324`, delete the `this.validateAndUpdateInfo();` call. - [ ] In `src/index.ts:157`, the `validateAndUpdateInfo.bind(this)` is passed to `uiController.initialize` as the `onValidate` callback. Trace where `uiController` invokes it — if it's only edit-driven (not load-driven), leave the callback intact. If it's never used, remove the parameter from the `uiController.initialize` signature. - [ ] Keep `GTFSEditor.validateAndUpdateInfo()` (`src/index.ts:365`) so manual callers still work, but add a `// Not called on startup — invoke manually if the validation panel is opened.` comment (the one allowed comment per CLAUDE.md rules: the WHY is non-obvious). - [ ] If `uiController.initialize` ends up not needing the callback, also remove the unused `validator` field if nothing else references it. **Gotcha**: don't delete the validator module — it's still imported in tests / could be wired up later. Only delete the *startup invocation*. **Gotcha**: search for any other callers of `validateAndUpdateInfo` or `validator.validateFeed` to make sure removing the startup call doesn't break a code path that depended on the side-effect of the validator running first (it shouldn't — the validator is pure). --- ## Phase 3 — Switch blob format from CSV to JSON, and parse off the main thread This is the biggest perceived win. CSV blobs require `Papa.parse` + `processParsedData` — the first is single-threaded text parsing, the second is per-row per-field allocation. JSON blobs go through native `JSON.parse` (highly optimized in V8) and the rows come back already type-coerced, so `processParsedData` disappears entirely. This phase has two halves: change what we *persist*, and move the *parse* into a worker. ### 3a. Change persisted blob format to JSON - [ ] In `src/modules/gtfs-database.ts`, change the `file_blobs` store schema (around line 156 / line 278) from `{ tableName, csv }` to `{ tableName, json }`. Update the `FileBlob` interface. - [ ] Change `saveTableBlob(tableName: string, csv: string)` (`src/modules/gtfs-database.ts:1027`) to `saveTableBlob(tableName: string, json: string)`. Same body, just stores under `json` key. - [ ] Change `getTableBlob(tableName: string): Promise<string | null>` (`src/modules/gtfs-database.ts:1038`) to return the `json` field. - [ ] In `src/modules/gtfs-parser.ts:407` `persistDirtyBlobs()`, replace `const csv = this.generateCSVFromRows(fileName, rows)` with `const json = JSON.stringify(rows)`. - [ ] In `src/modules/gtfs-parser.ts:639` `parseFile()` (the new-import path), replace `await this.gtfsDatabase.saveTableBlob(tableName, fileResult.rawContent)` (which writes the raw CSV) with `await this.gtfsDatabase.saveTableBlob(tableName, JSON.stringify(fileResult.data))`. - [ ] Delete `GTFSParser.generateCSVFromRows()` (`src/modules/gtfs-parser.ts:425`) and `GTFSParser.processParsedData()` (`src/modules/gtfs-parser.ts:128`) and `GTFSParser.parseFieldValue()` (`src/modules/gtfs-parser.ts:64`). Search for references first — they're also called from `updateFileInMemory` (`src/modules/gtfs-parser.ts:755`) and `updateFileContent` (`src/modules/gtfs-parser.ts:699`), which *do* still need to parse CSV (user pastes CSV into the editor). For those call sites, keep `parseFieldValue`/`processParsedData` — only delete `generateCSVFromRows` if no caller remains. (Revise as needed during implementation; the goal is "no CSV in the blob round-trip" not "no CSV code at all".) - [ ] In `src/modules/gtfs-database.ts:213` `initialize()`, bump `this.dbVersion`. The existing upgrade flow already handles wiping + offering export. ### 3b. Move blob restore into a worker The new-import path already uses a worker (`src/workers/gtfs-parser.worker.ts`). Restore is structurally similar (parse strings → emit row arrays) so we add a new message type rather than a new file. - [ ] In `src/workers/gtfs-parser.worker.ts`, add a new inbound message type `{ type: 'restore'; blobs: { tableName: string; json: string }[] }` and a corresponding handler. - [ ] The handler iterates `blobs`, calls `JSON.parse(json)` for each, and posts `{ type: 'progress', progress, status: 'Restoring ${tableName}...' }` between tables, then a final `{ type: 'done-restore', tables: { [tableName]: rows[] } }`. - [ ] In `GTFSParser.restoreDataFromDatabase()` (`src/modules/gtfs-parser.ts:483`), rewrite the body: 1. `Promise.all` over `ALL_GTFS_FILES` to fetch all `getTableBlob(...)` calls in parallel (independent IDB reads). 2. Filter out null/empty blobs and `.geojson` files (those still go through their existing path). 3. Spawn the worker, post `{ type: 'restore', blobs }`, hook `onmessage` to forward progress to `feedProgressIndicator` and resolve on `done-restore`. 4. On resolve, iterate `tables` and for each: set `this.gtfsData[fileName] = { content: '', data: rows, errors: [] }` and `this.setupVirtual(tableName, rows)`. Preserve the shared-array invariant (`rows` passed to `setupVirtual` must be the *same* array stored in `gtfsData[fileName].data`). 5. Terminate the worker. - [ ] Keep the `.geojson` handling unchanged — those files are tiny and per-table. **Gotcha (critical)**: the shared-array invariant. `setupVirtual`'s `flat` parameter is stored as a live reference (`gtfs-parser.ts:163` doc comment). After the worker returns, the deserialized array from `JSON.parse` is a fresh array — that's fine, but `gtfsData[fileName].data` and the `flat` passed to `setupVirtual` must be the **same reference**, not copies. Mistake would corrupt `byId` silently on the next edit. **Gotcha**: workers don't share memory; the JSON strings get structured-cloned across the boundary. For very large blobs (100 MB+ JSON for `stop_times`) the clone has measurable cost. Mitigate by transferring `ArrayBuffer`s instead: store blobs as `Uint8Array` containing UTF-8 JSON bytes, transfer with `postMessage(msg, [buffer])`. **Defer this optimization** — start with plain string transfer, measure, optimize only if string-cloning is a bottleneck. **Gotcha**: the IDB upgrade modal at `gtfs-database.ts:231` shows an "Export & Continue" option which calls `exportCurrentBlobsAsZip`. If that exporter assumes the `csv` field on `file_blobs`, it will break for users upgrading from the old schema. The export reads the *old* schema before the upgrade fires, so it should still see `csv`-keyed records — verify this is true (the `peekVersion` call at `gtfs-database.ts:229` opens the DB without triggering upgrade). If `exportCurrentBlobsAsZip` opens a *new* DB connection that triggers upgrade-and-wipe before reading, fix it to read the raw blobs via a non-upgrading open. --- ## Phase 4 — Defer non-critical post-restore work After the restore finishes, `init()` does a lot of sequential work before the user sees anything: `mapController.updateMap()`, `browseNavigation.refresh()`, `uiController.updateFileList()`. Reordering these to favor first paint reduces perceived load time even when total time is unchanged. **Goal**: the user sees the breadcrumb / sidebar shell as soon as possible, even if the map is still rendering. - [ ] After `pageStateManager.initializeFromURL()` (`src/index.ts:330`), call `browseNavigation.refresh()` immediately (don't wait for map) and `uiController.updateFileList()` right after. These are fast and give the user something to look at. - [ ] Wrap `mapController.updateMap()` in a `requestIdleCallback` (or `requestAnimationFrame` fallback) so the browser can paint the sidebar before kicking off the heavy index build. - [ ] Remove the trailing `if (this.browseNavigation) { this.browseNavigation.refresh(); }` block (`src/index.ts:347`) — redundant after the reorder. - [ ] The `exportBtn.disabled = false` enable (`src/index.ts:355`) should happen as soon as `gtfsParser.initialize()` resolves, not at the end of `init()` — the user can export the moment data is in memory. **Gotcha**: `browseNavigation.refresh()` calls `getPageStateManager().getBreadcrumbs()` which depends on `pageStateManager.initializeFromURL()` having run. Order: `initializeFromURL` → `browseNavigation.refresh()` → idle-callback for `updateMap()`. Keep that order. **Gotcha**: `requestIdleCallback` isn't on Safari < 17. Fall back to `setTimeout(fn, 0)` if unavailable. Keep it simple — a 4-line `runWhenIdle` helper at the top of `index.ts`. --- ## Phase 5 — Eliminate snapshot/blob redundancy When both a snapshot and a current blob exist, `PatchManager.initialize()` discards the blob's data (`clearTable` + `insertRows` + `setInMemoryFileData`) and rebuilds from the snapshot — then replays patches on top. If the blob was already current (i.e. persisted *after* the last patch), all this work is wasted. **Goal**: track the version each blob was persisted at; skip snapshot/patch replay when blob is current. - [ ] Add a `blobVersion` field to the `meta` store: `{ key: 'blobVersion', version: number }`. After `persistDirtyBlobs()` (`src/modules/gtfs-parser.ts:407`) completes, write the current patch version into `meta.blobVersion`. The `PatchManager` exposes `version` via `patchManager.version` (`src/modules/patch-manager.ts:413`); pass it into `persistDirtyBlobs` as an argument so the parser doesn't need a back-reference. - [ ] In `PatchManager.initialize()` (`src/modules/patch-manager.ts:42`), after reading `currentVersion`/`headVersion`, also read `meta.blobVersion`. If `blobVersion === currentVersion`, skip the entire snapshot-restore + patch-replay block. The in-memory state from `GTFSParser.initialize()` is already correct. - [ ] Otherwise, fall through to the existing snapshot + patch-replay path unchanged. - [ ] In `applyPatchForward` (`src/modules/patch-manager.ts:92`), no changes needed — replay correctness relies on virtual-table dedup which is unchanged. - [ ] In `parseFile()` (`src/modules/gtfs-parser.ts:556` — new-import path), after the loop that writes blobs, also write `meta.blobVersion = 0` (a fresh import has no patches yet). Otherwise the next refresh will think the blob is stale. - [ ] In `initializeEmpty()` (`src/modules/gtfs-parser.ts:525`), set `meta.blobVersion = 0` after `persistDirtyBlobs()`. **Gotcha**: there is a 3-second debounce on blob persistence. So a user who edits then immediately refreshes will have `blobVersion < currentVersion` and will correctly fall into the snapshot+replay path. The optimization only kicks in when at least 3 seconds have passed since the last edit. That's fine — it's the common case (refresh after the user has been idle). **Gotcha**: if `persistDirtyBlobs` writes partial blobs (some tables in `blobDirty`, some not), `blobVersion` should *only* advance if every dirty table flushed successfully. Wrap the write + version update in the same operation, and on partial failure leave `blobVersion` untouched so the next load falls back to snapshot+replay (which is correct). **Gotcha**: snapshots run via `maybeSnapshot()` (`src/modules/patch-manager.ts:480`) every `CONFIG.SNAPSHOT_INTERVAL` (50) patches. They are *not* blob-aware. With this optimization, snapshots are only consulted on the slow path — which is fine; they remain a correctness fallback. No change needed to the snapshot cadence. --- ## Phase 6 — Verify and measure Easy to feel good about a "performance" patch that doesn't measure. Add the timing logs we need to confirm the win on a real large feed. - [ ] Add `console.time('[boot] total')` at the start of `GTFSEditor.init()` and `console.timeEnd` at the end (both success and failure paths). - [ ] Add `console.time` around each major stage: `gtfs-parser.initialize`, `patch-manager.initialize`, `map-controller.updateMap`, `browse-navigation.refresh`. - [ ] Inside `restoreDataFromDatabase`, log per-table parse times for any table over 1 MB JSON. - [ ] Manually test with three feeds: a small one (BART), a medium one (MBTA), and a large one (NYC subway or whatever the user has loaded). Confirm the progress bar appears, advances, and the page is interactive in a reasonable time. - [ ] Confirm IDB upgrade modal fires for users on the old schema and that "Export & Continue" yields a valid GTFS zip. - [ ] Confirm that after a normal load, a refresh with no edits is "blob-current" fast (Phase 5 should kick in). - [ ] Confirm that a refresh within 3 seconds of an edit still produces correct state (Phase 5 should *not* kick in; snapshot+replay path runs). **Gotcha**: don't leave the `console.time` calls in the shipped code. Either gate them behind a `DEBUG_BOOT` flag in `CONFIG` or remove before merging.