It's a side-project from our consultancy work. We're two deep technologists and so far entertaining the notion that we're very bad at (product) sales. But we're trying to learn that now.

https://logging24.com/landing_a/

The basic idea is to make Regex-scans so fast/cheap that "a metric" can be anything numeric in the text and "tracing" is useless because you can just log (and filter) more things. Turns out Regex at >200GB/s solves a lot of problems.

Metric cardinality explosion is immediately a non-issue, histograms have arbitrary resolution, and you can get from histogram pixels back to the underlying logs. And no need to instrument everything thrice for logs, metrics and traces.

The next big feature I'm aiming for is needle-in-a-haystack searches. The data block headers support it already, but the scan engine doesn't yet use it.

Nah, it's fine. Storage of raw logs is pretty cheap (and I think this is widely assumed). For querying, two problems arise:

1. Query latency, i.e. we need enough CPUs to quickly return a result. This is solved by horizontal scaling. All the idle time can be amortized across customers in the SaaS setting (not everyone is looking at the same time).

2. Query cost, i.e. the total amount of CPU time (and other resources) spent per data scanned must be reasonable. This ultimately depends on the speed of the regex engine. We're currently at $0.05/TB scanned. And metric queries on multi-TB datasets can usually be sampled without impacting result quality much.

> We believe raw data based approach will transform how we use observability data and extract value from it. Yep. We have built quuxLogging on the same premise, but with more emphasis on "raw": Instead of parsing events (wide or not), we treat it fundamentally as a very large set of (usually text) lines and optimized hard on the querying-lots-of-text part. Basically a horizontally scaled (extremely fast) regex engine with data aggregation support.

Having a decent way to get metrics from logs ad-hoc completely solves the metric cardinality explosion.

FWIW, I'm currently rebuilding it as a web-app, if you want to try: https://quuxtodo.com/

  |dump =*clap
  [ "pa" |dump /tsch < > 0.0 ] ==slow slow clap
  # [
  #   "pa"
  #   
  #   "tsch"
  #   
  #   +0.0e0
  # ]

Theoretically speaking, I need at least the deff and defq distinction, otherwise { } could not be invoked during "parse" time. The defv case could in principle be removed, but as * needs to execute (otherwise nothing ever will), execution would need to be the default or I would need to decide based on dynamic type. The first option would mean a lot of superfluous {} (or | actually) around normal data variables, the second one would destroy the nice similarities between arrays, strings and functions with integer domains.

{ /foo dump } /f defv { /bar dump } /g defv

f g ; /h deff

h # "foo" "bar"

{ 2 } /putTwoOnStack deff

{ 2 } /putClosureOnStack defv

putTwoOnStack dump # 2

putClosureOnStack dump #

putClosureOnStack * dump # 2

Comments URL: https://news.ycombinator.com/item?id=9795314

Points: 86

# Comments: 23

:(

Comments URL: https://news.ycombinator.com/item?id=664871

Points: 5

# Comments: 0