Telemetry Pipeline

Alpha Preview — "I'm Logging it."

Strike48's telemetry pipeline (McLogs) is a single high-performance Rust service that ingests logs, metrics, and traces, normalizes and routes them in-flight, and delivers them durably to a streaming layer (for real-time consumers and replay) and a lakehouse (DuckLake, for analytics).

The three signals have different dominant query shapes, so each is headed for a purpose-built storage path. Today, all three land in DuckLake — Parquet on object storage, registered in a Postgres catalog — delivered at-least-once, with a streaming copy for real-time consumers.

Each sink has its own write-ahead log — the append happens as events cross from the parser to that sink, so the two destinations buffer independently. The WAL holds events until the sink acknowledges and replays unacked events on crash, giving at-least-once delivery to both.

Logs — the firehose path

• Fewer, larger Parquet files — flushing at target_file_size (~128 MB) rather than on every small batch means better query planning than thousands of tiny files.
• Planned — sub-second visibility — recent logs would live inline in the catalog DB, queryable immediately (no flush lag).
• Planned — secondary indexes on trace_id, request_id, user_id, maintained in Postgres while rows are still inline. Not implemented today.

Metrics — the tiered-rollup path

• Typed columns, not JSON — values are stored as native typed metrics (gauge/counter/histogram), avoiding the per-row json_extract cost that dominates analytical queries.
• Transactional rollups — raw, 1m, 5m, 1h, and 1d resolutions are written in a single transaction; a query automatically reads the coarsest resolution that answers it.
• Retention by tier — raw data ages out; rollups are kept indefinitely.

Traces — catalog as index

• Point lookup by trace_id — Postgres index seek → row-group coordinates → a single object-store GET.
• Analytics stay columnar — percentiles, service maps, and error rates run over the span table.
• Hot traces (last hour) live inline in the catalog for sub-second visibility; sampling stays at the collector, not the storage layer.

Distributed DuckLake: one catalog, two sides

DuckLake is normally single-process DuckDB. McLogs makes it distributed by putting the catalog in Postgres and the data in an object store — so many writers and readers share one lake and coordinate only through catalog transactions. Crucially, ingest and query are separate parts: the write path and read path never talk to each other; they meet only at the shared catalog and storage, and scale independently.

• Ingest (write path) — buffers events → Arrow → Parquet (atomic write-then-rename) → registers files in the catalog → acks. Multiple ingestors write concurrently.
• Query (read path) — a separate service (mclogs-query, Arrow Flight + HTTP) resolves file/row-group locations from the catalog, then reads only the needed Parquet row groups. Adding query capacity never touches ingest.
• The catalog is the contract — a Postgres-native DuckLake catalog (no separate catalog service) is the single coordination point and file index for the lake. (The per-signal hot-tier and secondary-index designs above build on this catalog but are not yet shipped.)