Caching and CORS
Which recorder answers can be cached and for how long, and how browsers can call the API.
The recorder sets Cache-Control and Access-Control-Allow-Origin on every answer. The cache policy is what makes it cheap to serve history to many clients: past days never change, so caches keep them forever.
Cache policy
| Answer | Cache-Control | Why |
|---|---|---|
| Segment of a past UTC day | public, max-age=31536000, immutable | A closed day is final |
| Segment of today (UTC) | no-cache | Rewritten every save interval |
404 not recorded for a segment | no-cache | The day may appear later |
Day lists (/v1/segments/<key>/) | no-cache | Grow every day |
/v1/markets | no-cache | Grows when markets are added |
Everything under /v1/hl/, errors included | no-cache | Answers end at the newest sample |
Everything under /v1/options/, errors included, while option chains are recorded | no-cache | The newest snapshot changes every minute |
404 option chains not recorded | no-store | Option recording is off |
/health | no-store | Must reach the recorder |
Other errors: 400 and unknown-path 404 outside /v1/hl/ and /v1/options/, 405, 431 | no-store | Errors |
“Today” is decided by the recorder’s clock at the moment of the request: a segment whose day is before the current UTC day is a past day.
no-cache allows a cache to store the answer but requires it to revalidate before reuse. The recorder sends no ETag or Last-Modified, so revalidation fetches the full answer again. In practice, treat no-cache answers as uncached.
Nothing is backfilled from venue trade history, but the first save after midnight still writes the previous day’s last minutes, so a segment fetched in the first minute of a day can miss them. If you need every late minute, purge yesterday’s segments from your cache once in the morning, or cap their lifetime at the proxy.
Putting a cache in front
Any cache that honours Cache-Control does the right thing without configuration:
- Past days are stored once and served from the cache from then on.
- Everything else passes through to the recorder.
A client opening a week of 1-minute history fetches seven segments: six immutable, one for today. With a cache in front, the recorder sees only the request for today.
An nginx example with proxy_cache is in Deployment.
Compression
The recorder does not compress answers. Segments are raw binary and compress well (zstd about 3–4× on these records), so enable gzip or zstd at the proxy or CDN for application/octet-stream and text/plain.
Conditional and partial requests
| Feature | Supported |
|---|---|
HEAD | Yes |
If-None-Match / ETag | No |
If-Modified-Since / Last-Modified | No |
Range | No; the whole file is sent |
| Keep-alive | No; one request per connection |
CORS
Every answer carries:
Access-Control-Allow-Origin: *So a page on any origin can read the API:
const res = await fetch("https://data.example.com/v1/segments/BinanceUsdm_BTCUSDT/2026-10-03.fsd");
const bytes = new Uint8Array(await res.arrayBuffer());What works and what does not:
| Request | Result |
|---|---|
GET or HEAD without custom headers | Works; no preflight is needed |
| A request that triggers a preflight (custom headers, other methods) | The OPTIONS preflight answers 405, so the browser blocks it |
Requests with credentials (credentials: "include") | Blocked by the browser: a wildcard origin does not allow credentials |
Keep browser calls to plain fetch(url). If you need authentication, add it at a reverse proxy and have the proxy answer preflights itself.
The Flowscope browser app reads this history through the data hub, which runs a recorder and serves the same paths, on its own website at /hub. The open CORS policy lets your own web tools read a recorder’s API.
Recommended proxy settings
| Setting | Recommendation |
|---|---|
| Cache key | The full URL, including the query string (it matters for /v1/hl/…) |
| Cache size | Enough for all past days of the markets you serve; about 0.2 MB per BTC market-day |
| Stale answers | Serve stale on recorder errors, so past days stay available during a restart |
| Compression | gzip or zstd for application/octet-stream and text/plain |
| CORS | Leave it to the recorder, or replace it with your own policy if you restrict origins |