/ 2min

Resumable uploads for multi-gigabyte files

A chunked upload protocol built on Elysia.js, and what it takes to survive a connection that drops halfway through a 4 GB driver pack.

Driver packs are not small. A single package can run to several gigabytes, and the people uploading them are often on an office connection that drops. A plain multipart/form-data POST is the wrong tool: when the connection dies at 80 percent, the whole upload starts again.

Split, record, resume

The protocol splits the file into fixed-size chunks on the client. Each chunk is uploaded on its own with three pieces of identity: the upload session, the chunk index, and the total count. The server writes the chunk to a temporary location and records it.

The important part is that the record is what makes resuming possible. When a client reconnects, it asks which chunks the server already has. It only sends the missing ones. Nothing is retried blindly, and nothing is sent twice.

const received = await db.uploadChunk.findMany({
  where: { sessionId },
  select: { index: true },
});
// The client sends only the gaps.

Finalising is a separate call. It verifies the expected chunk count is present, assembles the parts, and only then moves the file into place. An interrupted upload leaves temporary parts behind, never a half-written driver pack that looks complete.

Cleaning up after the failures

The failure path deserves as much attention as the success path. Sessions that never finalise accumulate chunks, so a scheduled job expires sessions past their deadline and deletes their parts. Without it the temporary store grows forever, and the growth is invisible until the disk is full.

Expiry is measured from the last chunk received, not from when the session started. A slow but healthy upload should not be reaped while it is still making progress.

Search that holds up

The catalog side of the same system had a different problem: finding a driver by series, model, SKU, category, or hardware ID across a large table. ILIKE '%term%' does not use an index and degrades quickly.

A trigram index does, and it handles the partial-match queries people actually type. Combined with keyset pagination instead of OFFSET, the result pages stay fast at any depth. OFFSET 40000 makes the database read and discard forty thousand rows; keyset pagination reads only what it returns.

Neither piece is clever. Both are the difference between a system that works on a demo dataset and one that works on the real one.

More posts

2 min

Failover without breaking the stream

Moving a request to a healthy provider is easy. Moving it after the first token has already reached the client is the hard part.

  • Rust
  • Streaming
  • Reliability
Read article

2 min

One route, five windows

How a single SvelteKit route became a workspace, a dock, a desktop-locked board, and two on-demand windows, all reading the same local database.

  • Tauri
  • Svelte
  • Desktop
Read article

2 min

An agent that edits files should ask first

Buffering every write until a human approves the plan, and why the approval gate belongs in the runtime rather than the prompt.

  • AI
  • Architecture
  • Permissions
Read article