Files

The data hub serves two directories: Derive’s output, prepared from the raw archive, and the archive’s auxiliary directory, where Capture keeps Hyperliquid positions and option chains. Neither is served as a directory; the hub’s API is the only way to them.

Derived history

data/
├── archive-source.json
├── BinanceUsdm_BTCUSDT/
│   ├── 2026-10-03.fsd
│   ├── 2026-10-04.fsd
│   ├── heat/2026-10-04/13.book.fht
│   ├── heat/2026-10-04/13.book.5s.fht
│   ├── heat/2026-10-04/13.liq.fht
│   └── chart-hours/…
└── Okx_BTC-USDT-SWAP/
    └── …

Everything here can be derived again from the archive, after a fix or a new format; nothing in it is the only copy of anything.

Market keys

A market’s key is <Venue>_<SYMBOL>, with any character in the symbol other than ASCII letters, digits and - replaced by _. Examples: BinanceUsdm_BTCUSDT, Okx_BTC-USDT-SWAP, Hyperliquid_BTC.

Day segments

A day segment (.fsd) holds one market’s history for one UTC day:

• Minute footprints: per minute, OHLC, buy and sell volume, print count, buy and sell trade counts, buy and sell volume in each of the 11 trade-size bands, and buy/sell volume per price row.
• Minute context: per minute, the last open interest, the last funding rate, long and short liquidations, and book depth within ±0.1, 0.25, 0.5, 1, 2, 2.5, 5 and 10 % of the mid.

It is a compact little-endian binary format with a versioned header, DEFLATE-compressed, with an FNV-1a checksum. Only minutes that hold prints are written. The candle section is written empty: bars are built from the minutes. Derive writes each segment to a temporary file, syncs it and renames it over the old one, so a reader never sees a half-written segment, and merges so the more complete minute wins. The byte layout is in Segment format.

A busy BTC perpetual takes about 0.04 MB per day compressed.

Heatmap tiles

Each tile holds one market’s runs of one kind for one hour: start, duration, price bucket, size and side, stored column by column, DEFLATE-compressed, with a checksum. The coarser versions snap runs to 5 s, 1 min, 5 min or 1 h and keep the largest size per bucket and slot, so a zoomed-out heatmap looks the same with far fewer runs. A busy BTC perpetual takes about 125 MB of tiles a day.

Hyperliquid files

Under auxiliary/hl/<COIN>/ in the raw archive.

grid

A text file with two numbers: the level bucket size and the footprint row size in price units, for example:

20 5

The grid is chosen from the price on the first sample and then stays fixed for the coin, so every day file and every answer share one grid. Payloads written with a different grid are skipped when answers are built.

.levels and .fp day files

Day files are positions codec payloads appended one after another:

• .levels: level runs that ended that day, appended on every save.
• .fp: footprint minutes of that day, appended once each minute has closed.

Each payload is self-delimiting and carries its own checksum. A reader stops at the first damaged or truncated payload, so a crash in the middle of a write loses only the last append. Appends are synced to disk.

Option chain files

Under auxiliary/options/<COIN>/ in the raw archive. Once a minute Capture asks Deribit for the option trades since its last request and for every listed option of each configured coin, and appends the chain with each option’s net taker flow, as one snapshot, to that coin’s file for the UTC day.

An .opt day file is records written one after another, oldest first:

u32 length | snapshot (length bytes)

The length is little-endian. Each snapshot is exactly what /v1/options/ serves: the time it was taken, the index price and every option’s expiry, strike, call or put, open interest, mark volatility, underlying price and flow, compressed, with its own checksum. The flow and its trade cursor are checkpointed together before each snapshot is appended, so a restart goes on exactly where it stopped. A reader stops at the first record whose length runs past the end of the file, so a crash in the middle of an append loses only that record.

A BTC snapshot of about 950 options is about 10 to 12 kB, so a day file is about 14 to 17 MB. The hub keeps the newest snapshot of each coin in memory and reads older ones from the day files when asked for.