WebAssembly in Salesforce LWC: 500K Record Engine & Net-Zero DB
“When the platform caps storage, heap, and CPU, stop fighting the caps - change where the data lives and where the work runs.”
Independent research on a familiar ceiling: half a million catalog rows inside a sandboxed CRM component. Gzip transport, Wasm-assisted decoding, and an in-memory query engine - with edits stored as tiny deltas instead of a database.
WEBASSEMBLYRUSTLWCOBJECT STORAGEAST QUERYDELTA MERGE
500K-row synthetic fixture cold-loads, decodes, and indexes in the high tens of seconds on broadband in my tests - dominated by download, not parsing
What Made the Problem Difficult
Independent work, synthetic data only. I have watched large quoting and catalog problems hit the same three walls: CRM storage priced per row, server heaps capped in single-digit megabytes, and browsers asked to parse hundreds of megabytes of row-oriented JSON on the main thread. I wanted a reproducible answer to a narrow question - how far can you get with compression, direct-to-browser streaming, and in-memory indexing - so I built the pipeline end to end on weekends and measured it.
Core Platform Constraint
Systemic FrictionThe walls, stated plainly:
Half a million catalog rows do not belong in per-row CRM storage. The cost and the import time both hurt.
A 100MB+ JSON payload cannot pass through a 6–12MB execution heap. It fails loudly.
Catalog providers rate-limit. Synchronous pulls during busy hours return 429s.
Row-oriented JSON with repeated keys compresses badly and parses slowly.
Hundreds of thousands of object allocations trigger long GC pauses.
Price overrides and margin tweaks look like they need RDS - with bills and ops to match.
- !Stay inside the CRM component model: LWC under Lightning Web Security, no external portal redirect
- !Cold open in tens of seconds, warm filtering instant: full load in under ~30s on broadband, queries in milliseconds
- !No provisioned database for edits: overrides must work without RDS or similar
- !No raw catalog rows in CRM storage: object storage holds the snapshot, CRM holds at most pointers
- !Nothing heavy through the server heap: the browser streams bytes directly from storage
What System Was Designed
The pipeline I built and measured: 1. Offline compaction: a small worker reshapes synthetic catalog JSON into columnar arrays and gzip-compresses it. On my fixtures this lands around 2–3% of the original byte size, staged immutably on S3-compatible storage. 2. Signed-URL streaming: the server side mints a short-lived signed URL and steps aside. The component fetches bytes directly - the CRM heap never sees the payload. 3. Wasm-assisted decode with JS fallback: a small Rust/Wasm decoder expands the stream into linear memory; where Wasm is unavailable the same path falls back to Pako. Either way the result lands in typed arrays, not object soup. 4. In-memory query engine: the json-query-lite lineage - lexer, parser, AST over typed arrays and inverted indexes - evaluates nested AND/OR/NOT filters in milliseconds. 5. Delta files for edits: overrides serialize to a tiny compressed delta (~KBs) beside the base snapshot and merge in memory at load. No database process exists anywhere. 6. Virtualized grid: roughly four dozen DOM rows in a sliding window, so scrolling half a million logical rows stays smooth.
Pivotal Architectural Choices
Key Architectural Decisions
Wasm-assisted decode vs. main-thread JSON parsing
Expanding a multi-megabyte gzip into hundreds of thousands of JS objects on the main thread means seconds of GC pauses.
- Server-side chunking through Apex
- Main-thread loop with setTimeout slices
- Web Worker running plain JSON.parse()
Decoding outside the GC heap kept the UI alive during ingest; the fallback keeps the same code path working where Wasm is restricted.
Direct signed-URL streaming vs. relaying through the server
Anything that touches the CRM heap inherits its limits.
- Buffering chunks through server objects
- Proxying the stream through middleware
- Moving quoting to an external portal
The payload never enters server memory. The server stays an auth gate, which is all it should be here.
Delta files vs. a provisioned database for edits
Overrides need durability, but not necessarily a running database.
- Hosted Postgres cluster with sync API
- Document store with provisioned throughput
- Storing edits as CRM rows
For single-editor quoting sessions this removes the entire database tier. Concurrent editing needs explicit merge handling - stated openly below.
Typed-array AST engine vs. Array.filter() per keystroke
Naive per-keystroke scans across 500K rows lag Noticeably - a few hundred milliseconds per keypress.
- Plain Array.filter() with string matching
- IndexedDB queries
- Embedded SQLite in Wasm
The same engine family as my json-query-lite work. Millisecond queries without leaving the tab.
What Was Actually Built
Built the Wasm decoder from a minimal Rust crate (wasm-bindgen surface, streaming inflate) with an identical Pako JS fallback behind one interface, loaded as a static resource.
Kept authorization server-side and narrow: short-expiry signed URLs scoped to one snapshot prefix; no credentials reach the client.
Reshaped fixtures to columnar form before compression - shared keys factored out - which is where most of the byte savings actually come from.
Reused the json-query-lite query core (lexer, recursive-descent parser, BM25-aware AST) over typed arrays; nested AND/OR/NOT plus LIKE/IN/BETWEEN work the same in Node and in the component.
Implemented delta merge as a version-stamped hash join at load: base rows plus override rows reconcile before first paint, in milliseconds on test fixtures.
Built the grid as a sliding window of a few dozen rows positioned with transforms, so scroll cost is constant regardless of row count.
Deliberate Architectural Compromises
Trade-offs & Mitigations
⚖Browser memory vs. server spend
No database or compute tier to operate for browsing and editing snapshots.
The tab holds the working set - on the order of 150–180MB for the largest fixture I tested.
Columnar typed arrays (strings as dictionary IDs, money as integer minor units) plus prompt release of parse buffers.
⚖Wasm capability vs. sandbox reality
Near-native decode speed outside the GC heap.
Sandboxed environments restrict instantiation modes and shared memory.
Conservative Wasm feature set and a zero-config JS fallback; the slow path still works.
⚖Delta simplicity vs. concurrent-write safety
No lock management, no connection pooling, full history for free.
Concurrent edits to one quote need optimistic concurrency, not row locks.
Version-stamped deltas with hash checks and last-writer-wins plus manual re-merge surfaced in the UI - documented as a limit, not hidden.
Verified Outcomes
500K-row synthetic fixture cold-loads, decodes, and indexes in the high tens of seconds on broadband in my tests - dominated by download, not parsing
Snapshot bytes shrink to roughly 2–3% of the row-oriented JSON on test fixtures via columnar layout plus gzip
Interactive filtering over the full set returns in milliseconds per keystroke in the harness - no server involved
Edits persist as KB-scale delta files with no database process to run or pay for
Scrolling cost stays flat: constant DOM size regardless of logical row count
Published lineage as open source: json-query-lite (TS) and a Rust/Wasm search sibling, both MIT
System Schematic & Data Flow
Compressed snapshots stream direct to the browser, decode beside the GC heap, and merge with tiny edit deltas at render time.
Text alternative for screen readers: Architecture flow: Catalog API (throttled) to Offline Compactor via Async Ingest & Columnar Gzip; Object Store Snapshot to Component via Signed URL via Direct Streaming, No Server Heap; Wasm Decoder to AST Query over Typed Arrays via Decode Once, Filter in ms; Delta File (KBs) to In-Memory Merge via Edits Without a Database
Architectural Conclusion
“Platform limits tell you where the default path ends. Compression, direct streaming, and in-memory indexing are often enough to move the ceiling without new infrastructure.”
- •Measure the heap you actually have before designing the pipeline - it decides everything downstream.
- •Columnar layout plus gzip beats clever parsing of row-oriented JSON every time.
- •Delta files are a legitimate database replacement for single-editor snapshots - and an honest limitation for concurrent ones.
- •Direct-to-browser streaming removes a whole class of middle-tier bottlenecks.