How It Works

Fusionaly uses SQLite with buffering and backpressure. It needs no separate database cluster. Every layer allows eventual consistency: the browser SDK, the HTTP ingress, the WAL-backed writes, and the background aggregation. During a burst, writes slow down instead of overwhelming the database. Dashboards can lag by up to a minute during a spike. Raw events stay safe on disk the whole time.

Key guarantees:

• Retries at every layer stop events from being lost (browser to server to database)
• Ingestion writes straight to disk. There is no volatile buffer.
• Background jobs turn raw events into hourly aggregates
• Backpressure causes delays, not failed writes

Retry Strategy

Every layer retries failed operations. Events do not get lost.

Layer	Retry Behavior
Browser SDK	Retries 3 times, with delays of 1s, 2s, and 4s. Caches up to 100 events in localStorage when offline.
HTTP Ingress	Returns 503 + Retry-After when saturated. The SDK respects this.
Database Writes	PerformWrite retries up to 10 times, with exponential backoff, on SQLITE_BUSY.
Background Jobs	Failed batches stay in the queue. No row advances until it is processed.

The browser SDK uses navigator.sendBeacon() for page unload events. These survive even if the user closes the tab.

Data Flow

Browser SDK (200ms batches, 100-event localStorage cache)
        |
        v
/x/api/v1/events (HTTP ingress + concurrency limiter)
        |
        v
ingested_events table (durable queue)
        |
        v  Background job (100-row batches)
events table (authoritative log)
        |
        v  Aggregate upserts
site_stats / page_stats / ref_stats / ... (hourly buckets)
        |
        v
Dashboards (eventually consistent)

This design works with SQLite’s single-writer model, not against it. Buffers are explicit. Backpressure is intentional. Aggregates can lag briefly without any loss of correctness.

Storage Layer

Fusionaly uses SQLite in WAL mode, with busy_timeout = 5000 and a connection pool (10 open, 5 idle). The PerformWrite helper retries each transaction up to 10 times, with exponential backoff.

Three queueing layers:

1. HTTP handler — A concurrency limiter returns 503 + Retry-After when saturated
2. Connection pool — Goroutines wait for a free connection instead of overloading the database
3. WAL queue — Serializes writes while it still allows concurrent reads

Processing Pipeline

1. Browser SDK — Batches events every 200ms, up to 10 per batch. Retries with backoff. Caches up to 100 events in localStorage when offline.
2. Ingestion API — Validates events, normalizes URLs, filters excluded IPs, and writes them to ingested_events.
3. Background processor — Runs every 60 seconds. Drains pending rows in batches of 100, into events.
4. Aggregate upserts — Runs INSERT ... ON CONFLICT into hourly tables (site_stats, page_stats, and others).

No row is dropped. Every failure triggers a retry before the pipeline moves on.

Hot vs Cold Data

Path	Tables	Consistency	Use case
Hot	ingested_events, events	Synchronous	Visitor Transparency, exact queries
Cold	*_stats tables	Eventually consistent	Dashboards, reports

Cold tables store hourly buckets. Even a multi-year query reads only about 24 rows per day, per dimension.

Burst Handling

During a spike of 1,000 events per second:

1. Browsers coalesce requests, so only about 100 HTTP requests per second reach the server.
2. Ingestion accepts every event into ingested_events and responds fast.
3. The background processor drains the queue in chunks of 100 rows, and loops until it is empty.
4. Aggregates lag briefly, but stay in order.

Performance

Reads: Most dashboard queries take under 10ms. Hourly aggregates keep queries cheap.

Writes: WAL mode, batch transactions (100 events per commit), and client and server retries work together. Fusionaly needs no RAM buffer.

-- Typical dashboard query (single-digit ms)
SELECT strftime('%Y-%m-%d', hour) AS day, SUM(visitors) AS visitors
FROM site_stats
WHERE website_id = ? AND hour BETWEEN datetime('now', '-30 days') AND datetime('now')
GROUP BY day;

Storage Estimates

Each event uses about 240 bytes, including indexes and aggregate overhead.

Traffic	Daily Events	Daily Growth	1-year	5-year
1K visits/day	~3,750	~0.7 MB	~0.3 GB	~2 GB
10K visits/day	~37,500	~7 MB	~2.6 GB	~19 GB
100K visits/day	~375,000	~72 MB	~26 GB	~190 GB

Storage grows in a straight line with time. It does not grow faster during traffic spikes.

On SQLite limits: SQLite supports databases up to 281 TB in theory. You will never reach this. A site with 100K daily visits stores about 26 GB per year. At that rate, reaching SQLite’s limit would take about 10,000 years. Plan for disk space, not for a database limit.

Summary

SQLite works well here for three reasons. Hot tables stay small. Dashboards query compact hourly aggregates. Writes never block reads. A modest server handles about 400 requests per second — enough for hundreds of thousands of events per day.