File Sync

Dropbox and OneDrive don’t upload a whole file again when you change it, and neither does this. Shiny.AspNet.FileSync is the server, mapped into any ASP.NET Core app. Shiny.Net.HttpServer.FileSync.Client keeps a local folder in sync with it in both directions, and runs its transfers through Shiny.Net.Http, so they continue in the background on iOS and Android. The client has no server dependency — it is the same package whichever server it talks to. Trim- and AOT-clean.

GitHub
Downloads

Frameworks

ASP.NET

dotnet add package Shiny.AspNet.FileSync                  # the server
dotnet add package Shiny.Net.HttpServer.FileSync.Client   # the app

// server
using Shiny.AspNet.FileSync;

app.MapFileSync("/sync", o => o.Store = new DiskFileSyncStore(Path.Combine(dataDir, "sync")))
   .RequireAuthorization();

// app (MAUI)
builder.Services.AddFileSyncClient(o =>
{
    o.ServerUri = new("https://files.example.com/sync");
    o.LocalPath = Path.Combine(FileSystem.AppDataDirectory, "Documents");
    o.Headers["Authorization"] = "Bearer " + token;
});

// then, from a page or a background job:
var result = await sync.SyncAsync();     // one pass
await sync.RunAsync(cancellationToken);  // or keep the folder in sync until cancelled

How only the changed bytes move

Files are split into content-defined chunks with FastCDC, 64 KB–1 MB with a 256 KB average. Each chunk ends where a rolling hash of the bytes before it hits a pattern, so chunk boundaries depend on the content, not on fixed offsets. An insertion in the middle of a file changes only the chunk it lands in. With fixed-size blocks, every block after the insertion would change.

The server stores each chunk once, keyed by its SHA-256. A file is a list of chunk hashes.

Direction	How it works
Upload	the client chunks the changed file and asks the server which chunks it lacks (POST chunks/missing). It sends only those, grouped into packs of up to 8 MB, then commits the new chunk list. Renaming or copying a file uploads nothing. Inserting into the middle of an 8 MB file uploads a few hundred KB.
Download	the client fetches the new chunk list and takes every chunk it already holds from local files (the old version of that file, or any other synced file). It downloads the rest as packs, verifies every chunk’s hash, assembles the file in a staging area, and moves it into place in one step. A half-written file is never visible.

FileSyncResult.BytesTransferred and BytesReused report how much of each pass crossed the network and how much didn’t need to.

Background transfers

Packs move through Shiny.Net.Http’s IHttpTransferManager:

Platform	Details
iOS	an NSURLSession background session.
Android	a foreground service.
Everywhere else	a managed loop that resumes downloads with Range (the server supports Range: bytes=N- on packs).
Uploads	sent over tus by default (UseTusUploads), to the server’s built-in tus endpoint at {prefix}/uploads, so a dropped upload resumes rather than restarting.

AddFileSyncClient registers Shiny transfers itself when the app hasn’t. If the app also uses Shiny.Net.Http for its own transfers, register those with its own delegate before AddFileSyncClient, and the sync client shares them.

Transfer identifiers come from each pack’s content. If the app is killed mid-transfer, the next pass finds the transfer still running, or finished, and waits on it or uses it instead of starting again. The whole design is safe to interrupt at any point, because everything is content-addressed and every commit names the revision it was based on.

Note

Without a transfer manager (plain .NET on macOS or Linux, unless you call AddHttpClientTransfers and register an IConnectivity), or with UseBackgroundTransfers = false, packs go directly through HttpClient while the app is running. Outside a Shiny host, call Start() on HttpClientHttpTransferManager yourself. A MAUI app using UseShiny() does this automatically.

Sync semantics

Every change on the server gets the next number from one counter. That number is the version’s revision, and also the cursor of the change feed. The client records, for each path, the revision and the local size and modified time it last agreed on, and judges each pass against that record:

Here	There	Result
changed	unchanged	Upload, committed against the known revision
unchanged	changed	Download
deleted	unchanged	Delete on the server (leaves a tombstone)
unchanged	deleted	Delete locally, and remove folders left empty
changed	deleted	The edit wins: uploaded as a new file
deleted	changed	The change wins: downloaded again
changed	changed, same bytes	Adopted, nothing transferred
changed	changed	Conflict: the server’s version keeps the name, and the local edit becomes name (conflicted copy {DeviceName} {date}).ext and is uploaded too

Nothing is ever silently overwritten. A commit whose base revision is out of date gets a 409 with the current version, and the client turns that into the conflict above. The same happens if a local file changes while a download for it is being assembled: that path is left for the next pass.

RunAsync runs a pass, then waits for whichever comes first:

• a server long-poll (GET changes/wait, Dropbox’s longpoll) reporting a change,
• a FileSystemWatcher event on platforms that have one (not iOS),
• ScanInterval (30 s).

Failed passes back off, up to five minutes. The Activity event reports each path as it is synced, and SyncCompleted reports each pass.

Paths compare without case, as on macOS and Windows, and names Windows would refuse are rejected everywhere. The client never syncs .DS_Store, Thumbs.db, desktop.ini, Office ~$ lock files, *.tmp, *.partial, or its own .shinysync state folder. Add your own rules with Ignore.

Per-user stores

One store serves everyone. For one tree per account, as Dropbox does it, pick a store per request:

var stores = new ConcurrentDictionary<string, DiskFileSyncStore>();

app.MapFileSync("/sync", o => o.StoreSelector = ctx =>
        stores.GetOrAdd(ctx.User.Identity!.Name!, name => new DiskFileSyncStore(Path.Combine(root, name))))
   .RequireAuthorization();

Chunks are never shared between stores. Otherwise one user could learn whether another holds a particular file by asking whether its chunks are missing.

Storage and garbage collection

DiskFileSyncStore keeps chunks as files (chunks/ab/abcdef…) and the file tree as an append-only journal, journal.jsonl. Each commit is flushed to disk before the client is told it succeeded. When the journal is replayed, a line torn by a crash is ignored, and the journal is compacted when it has grown well past the live tree. Implement IFileSyncStore to keep chunks in blob storage or entries in a database. CommitAsync is the one place that must be atomic.

Chunks of deleted files and replaced versions stay until you collect them:

var sync = app.MapFileSync("/sync", o => { ... });
// from a timer or a BackgroundService:
var removed = await sync.CollectGarbageAsync();

Chunks younger than GarbageGracePeriod (1 day) are kept even when nothing refers to them, because they may belong to a commit that is still on its way. Packs whose tus upload was started and abandoned are swept after a day by the tus endpoint’s own timer; sync.RemoveExpiredUploadsAsync() sweeps them now.

Request size

Kestrel limits a request body to 30 MB by default. The sync routes set their own: a plain PUT chunks/pack is bounded by MaxPackSize (64 MB), and the tus upload routes by what is left of the pack being uploaded. A RequestSizeLimitAttribute you put on the endpoint with WithMetadata takes precedence.

Routes

All routes are under the prefix; every GET also answers HEAD:

Route
GET changes?cursor=&limit=	What changed after a cursor, oldest first
GET changes/wait?cursor=&timeout=	Long-poll until something changes
GET files/manifest?path=	A file’s current version and chunk list
PUT files/manifest?path=	Commit a version: 409 when the base revision is stale, 422 listing missing chunks
DELETE files/manifest?path=&revision=	Delete, leaving a tombstone
GET files/content?path=	The whole file, for a browser or a script
POST chunks/missing	Which of these chunks the server lacks
PUT chunks/pack	Upload a pack in one request
GET chunks/pack?h=a,b,c	Download chunks as a pack, resumable with Range
GET chunks/{hash}	One chunk
uploads	tus 1.0.0, for resumable pack uploads

MapFileSync returns a FileSyncEndpointConventionBuilder, an ordinary IEndpointConventionBuilder over the route group all of them live in — the tus routes are a nested group inside it — so RequireAuthorization, RequireCors, RequireRateLimiting and every other ASP.NET convention covers them all. The routes are excluded from the OpenAPI document. Errors are problem details, written through IProblemDetailsService when the app registered one. OnChangedAsync runs after every commit or delete.

Options

Property	Default	Notes
Store / StoreSelector	one required	One store for everyone, or one per request
MaxChunkSize	4 MB	Largest chunk accepted (the client’s maximum is 1 MB)
MaxPackSize	64 MB	Largest pack, through tus or a plain PUT
MaxChunksPerDownload	256	Most chunks one pack download may name
MaxChangesPerPage	1000	Most entries per page of the change feed
MaxWaitTimeout	90 s	Longest a long-poll is held open
EnableTusUploads	true	Map the tus endpoint at {prefix}/uploads
TusStore	{root}/uploads	Where packs stay while their tus upload is in progress
GarbageGracePeriod	1 day	How long an unreferenced chunk is kept
OnChangedAsync	null	Runs after every committed change
TimeProvider	system	The clock garbage collection runs on

Not included

• Version history: chunks of old versions are collectable, and there are no routes for listing or restoring them.
• Folder entries: empty folders are not synced.
• Sharing between accounts.
• Selective sync.
• A browser client.

The file hash in a commit is the client’s own claim. Chunk hashes are verified on every upload and download. Tested end to end on Kestrel with two client folders against one server, over HttpClient and through Shiny.Net.Http’s transfer manager with tus uploads, on HTTP/1.1.

AI Coding Assistant

Claude CodeGitHub Copilot

Step 1 — Add the marketplace:

claude plugin marketplace add shinyorg/skills

Step 2 — Install the plugin:

claude plugin install shiny@shiny

One plugin installs all 48 Shiny skills. Your agent loads only the skill relevant to what you're building, so there's no cost to having them all available.

Step 1 — Add the marketplace:

copilot plugin marketplace add https://github.com/shinyorg/skills

Step 2 — Install the plugin:

copilot plugin install shiny@shiny

One plugin installs all 48 Shiny skills. Your agent loads only the skill relevant to what you're building, so there's no cost to having them all available.

View shiny-aspnet Skill