Yet these are quite comparable accidents.

I wonder what the reason is?

Comments URL: https://news.ycombinator.com/item?id=45491591

Points: 2

# Comments: 0

The utility is designed for monitoring small, medium and even large networks using Netflow/IPFIX/sFlow protocols.

It allows you to build different reports on IP addresses, protocols, services, GeoIP, Autonomous Systems and various fields from Netflow/IPFIX/sFlow. Reports can be shown in Grafana.

The utility is not resource-intensive, small networks can be monitored even on Raspberry Pi (with an external hard drive, of course).

In addition to reports, the collector uses moving averages and can run scripts when thresholds are exceeded or when traffic falls below thresholds. We use this to create BGP Flowspec or Blackhole announces during DoS/DDoS attacks.

The utility is in beta state, so if you notice any bug or error in the documentation - feel free to drop a line.

Any feedback is welcome!

Comments URL: https://news.ycombinator.com/item?id=43083439

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=38179653

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=35525945

Points: 2

# Comments: 0

Library was initially written for use with Netflow collector. More precisely for storing filtered and aggregated traffic information. Typical load for such applications is: a lot of updates in "hot" dataset, fewer inserts, deletions are rare and mostly batched, no needs for fast sequential scan (user can wait for a few seconds or even more to get report). Radix trees are perfectly fits this requirements.

Later we made some tests and decided to use it for IP (plus some other information) lookups without underlying file ("RAM-only" mode).

We have to support some old network hardware (for example almost abandoned Tilera TILE-Gx), that's why we're trying to avoid using OS and platform-specific features in library.

Radix trees has disadvantages, the most notable is that it's hard to implement user-defined sorting order for keys and they required more memory for sparse nodes compared to B-trees (however, this statement is not true for on-disk TKVDB database, we don't store empty pointers)

Briefly speaking, the above projects are way more mature, full-featured and use different data structures for storing data. BDB, LMDB are B-tree-based, LevelDB (and forks), SQLite4 LSM are using log-structured merge-trees.

TKVDB is a lot simpler, a bit more low-level, uses Radix trees and not tested as well as other key-value engines.

> I read this as: only use it single threaded.

If you protect access to data with your OS locks, everything will be fine.

We're using TKVDB in multi threaded environment (in RAM-only mode) for IP lookups with Netmap and DPDK. On 40Gbe full line rate there is only few hundred CPU cycles for decision on network packet, so the locking tricks becomes important.

Intel has guarantees about writing properly aligned data - if one core is writing 1, 2, 4 or 8 bytes, it will be written atomically. We're exploiting this feature when updating our tree - final update operation is 8 byte (4 byte for 32 bit CPU's) aligned pointer update.

So, in some cases you don't need to use locks at all (you may add HW memory barrier (__sync_synchronize()) after tree update, but it will slow down writer). If only one core is writing data, another cores can safely (well, almost) read it. Warning in docs was about this lockless case.

Comments URL: https://news.ycombinator.com/item?id=17839935

Points: 105

# Comments: 20