Axiom

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Build me log storage that replaces Axiom — and be honest about why this is ALMOST.

What you are paying for is **storage economics**: ingest everything, keep it for months, and query it fast enough to be worth opening during an incident. Reproducing that means a columnar store, compression, and object storage, not a table in a relational database. It is buildable with existing open pieces, and it is genuinely more operational work than it looks. Read the sizing section before starting.

STACK
- ClickHouse for storage and querying, or Loki if your queries are mostly label filters and greps. Do not write a storage engine
- Vector as the shipper and the transform layer. Do not write log agents
- Node 20+ with Fastify only for the small pieces: the query interface, alerts and access control
- Object storage for cold data
- Caddy in front, on a private network

THE SHAPE
- Ingest endpoints speaking the common protocols so existing shippers work unchanged: HTTP with JSON lines, syslog, and whatever your platform already emits
- One wide table per dataset with a timestamp, a set of low-cardinality attributes as real columns, and the rest as a map. Query on the columns; the map is for the long tail
- Partition by day, order by the columns you filter on most, and set a compression codec deliberately. Those three choices decide whether a month of data is queryable in a second or in a minute
- Tiering: hot on local disk, older partitions on object storage, with a stated retention per dataset

WHAT DECIDES THE BILL
- Volume, and it is always higher than estimated. Measure bytes per day per service before choosing hardware
- Cardinality: an attribute with millions of distinct values — a request id, a user id — as a real column will destroy the index. Keep those in the map or in a dedicated field, and write the rule down where people adding logs will read it
- Sampling is the thing the paid product exists to avoid, so if you find yourself sampling, compare the cost honestly against the subscription
- Structured logs, one JSON object per line, with a level, a timestamp, a service and a request id. A log line that is a sentence cannot be queried, and this is the single biggest lever on usefulness

QUERYING
- The store's own query language, with saved queries and a few starting points for the questions asked during an incident: errors by service, slowest requests, everything for one request id
- A time-range picker, a live tail, and a share link that carries the query and the range
- Fast enough that somebody opens it at three in the morning rather than grepping a server

ALERTS
- A saved query, an interval, a threshold, and a channel. Evaluated by a small worker, with deduplication and a recovery message
- Alert on symptoms with an action attached; delete any alert nobody acts on
- The alerting path must not depend on the infrastructure being watched

ACCESS AND SAFETY
- Per-dataset permissions, and redaction at ingest: secrets, tokens and personal data stripped by key and by pattern before storage. A log store is where data leaks to and stays
- Retention enforced and provable, because a log containing personal data is subject to the same deletion requests as anything else

OPERATIONS
- Compose files, the shipper configuration, retention policies, and a disk watchdog that sheds ingest rather than filling the volume
- Backups for the configuration and the alert definitions; the logs themselves have a retention, and that is the policy
- Health endpoint reporting ingest lag and free space

WHAT MATTERS MOST
Cardinality discipline and structured logs at the source. Get those wrong and no amount of storage tuning saves you; get them right and a small machine will hold more than you expect.