Strike48 Labs Strike48 Labs

Deployment

Alpha Preview — "I'm Logging it."

Target architecture

This is the target deployment topology. The on-node hot layer and the separate meta-query tier with in-memory / NVMe caches are the design direction, not the current shipped deployment.

A very high-level view of how McLogs is deployed. Sources feed **ingest nodes** that keep recent data in an on-node **hot layer** and land everything else in **S3** (Parquet), with a **Postgres catalog** as the file index. A separate, horizontally-scaling **meta-query tier** answers queries by reading the hot layer for recent data and S3 for anything older — accelerated by in-memory and NVMe caches.

Topology

Meta-query tier - scales horizontally locations hot reads hot reads older than hot layer Source 1 Ingest node 1hot layer Source 2 Ingest node 2hot layer Postgres catalog S3 bucketParquet - cold Meta query In-memory cache NVMe cache
Meta-query tier - scales horizontally locations hot reads hot reads older than hot layer Source 1 Ingest node 1hot layer Source 2 Ingest node 2hot layer Postgres catalog S3 bucketParquet - cold Meta query In-memory cache NVMe cache

How it fits

  • Ingest nodes — each source lands on an ingest node. Recent data stays in that node's hot layer (fast, local); as it ages past the hot window it's written to S3 as Parquet and registered in the Postgres catalog. Add ingest nodes to take on more sources / throughput.
  • Postgres catalog — the index and metadata store. The meta-query tier reads it to find where data lives (which node's hot layer, or which S3 objects / row groups).
  • S3 bucket — the cold tier. It's linked into the meta-query tier and serves any read older than the hot layer; recent reads never touch S3.
  • Meta-query tier — answers queries by fanning out: hot reads hit the ingest nodes directly, cold reads come from S3, and results are stitched together. It scales horizontally — add meta-query nodes for more concurrent query load.
  • Caches — each meta-query node fronts storage with an in-memory cache (hottest results) and an NVMe cache (larger, still-fast spill), so repeat and adjacent queries avoid round-trips to S3.