Storage

Everything the recorder keeps lives in one directory, set with --dir (default flowscope-data). The HTTP server serves this directory read-only.

Directory layout

flowscope-data/
├── BinanceUsdm_BTCUSDT/
│   ├── 2026-10-03.fsd
│   └── 2026-10-04.fsd
├── BinanceUsdm_BTCUSDT.fsh
├── Okx_BTC-USDT-SWAP/
│   └── 2026-10-04.fsd
├── Okx_BTC-USDT-SWAP.fsh
├── hl/
│   ├── BTC/
│   │   ├── grid
│   │   ├── 2026-10-04.levels
│   │   └── 2026-10-04.fp
│   └── ETH/
│       └── ...
└── options/
    ├── BTC/
    │   ├── 2026-10-03.opt
    │   └── 2026-10-04.opt
    └── ETH/
        └── ...

Market keys

A market’s key is <Venue>_<SYMBOL>, with any character in the symbol other than ASCII letters, digits and - replaced by _. Examples: BinanceUsdm_BTCUSDT, Okx_BTC-USDT-SWAP, Hyperliquid_BTC.

/v1/markets lists every directory in the data directory whose name is a valid key, except hl and options.

Day segments

A day segment (.fsd) holds one market’s history for one UTC day:

• Minute footprints: per minute, OHLC, buy and sell volume, print count, buy and sell trade counts, buy and sell volume in each of the 11 trade-size bands, and buy/sell volume per price row.
• Minute context: per minute, the last open interest, the last funding rate, long and short liquidations, and book depth within ±0.1, 0.25, 0.5, 1, 2, 2.5, 5 and 10 % of the mid.

It is a compact little-endian binary format with a versioned header and an FNV-1a checksum. Only minutes that hold prints are written. The format still has a candle section, written empty: bars are built from the minutes. The desktop keeps no segments; it loads them from the hub. The byte layout is in Segment format.

A busy BTC perpetual takes about 0.2 MB per day raw; a quiet altcoin about 0.1 MB, where the fixed per-minute records dominate.

How segments are written

• Changed segments are written every --save-secs (60 s by default), and on exit.
• Each write goes to a temporary file that is synced and then renamed over the old file. A reader never sees a half-written segment.
• What is in memory is merged with what is on disk first, so a short session never overwrites a fuller one. The minute still filling is saved as Partial.
• Today’s segment is rewritten on every save; the first save after midnight can still write the previous day’s last minutes.

Heatmap snapshots

The .fsh file holds a market’s raw tape and liquidity runs, so the recorder’s own heatmap state survives a restart. The recorder does not serve it, so it writes it only on exit.

Hyperliquid files

grid

A text file with two numbers: the level bucket size and the footprint row size in price units, for example:

20 5

The grid is chosen from the price on the first sample and then stays fixed for the coin, so every day file and every answer share one grid. Payloads written with a different grid are skipped when answers are built.

.levels and .fp day files

Day files are positions codec payloads appended one after another:

• .levels: level runs that ended that day, appended on every save.
• .fp: footprint minutes of that day, appended once each minute has closed.

Each payload is self-delimiting and carries its own checksum. A reader stops at the first damaged or truncated payload, so a crash in the middle of a write loses only the last append. Appends are synced to disk.

Answers combine these files, for everything memory no longer holds completely, with memory, for the rest. In memory the recorder keeps up to 64 MB of level runs and two days of footprint minutes per coin.

Option chain files

With --options-coins set (BTC and ETH by default), the recorder asks Deribit once a minute for the option trades since its last request and for every listed option of each coin, and appends the chain with each option’s net taker flow, as one snapshot, to that coin’s file for the UTC day.

An .opt day file is records written one after another, oldest first:

u32 length | snapshot (length bytes)

The length is little-endian. Each snapshot is exactly what /v1/options/ serves: the time it was taken, the index price and every option’s expiry, strike, call or put, open interest, mark volatility, underlying price and flow, compressed, with its own checksum. The flow is not stored anywhere else: after a restart the recorder goes on from the newest snapshot’s. A reader stops at the first record whose length runs past the end of the file, so a crash in the middle of an append loses only that record.

A BTC snapshot of about 950 options is about 10 to 12 kB, so a day file is about 14 to 17 MB. The newest snapshot of each coin is also kept in memory; older ones are read from the day files when asked for.

Retention

Leftover temporary files from an interrupted write are removed when segments are pruned.

Sizing the disk

Heatmap snapshots depend on how busy the market is and how long the recorder ran before exit.

Backups

All files are safe to copy while the recorder runs: segments are replaced atomically, and Hyperliquid and option chain day files only grow by whole appended payloads (a copy taken mid-append ends in a torn payload that readers ignore).