Are you talking about the 2008 financial crisis? or do you mean something else?

because of this separation, the compute (e.q SQL parsing, etc) can be scaled independently and the storage can also do the same, which for example use AWS S3

so if your SQL query is CPU heavy, then Neon can just add more "compute" nodes while the "storage" cluster remain the same

to me, this is similar to what the usual microservice where you have a API service and DB. the difference is Neon is purposely running DB on top of that structure

also by relying on the whatsapp metadata (the one where you can export by urself), it should be already good enough for feds agent to locate u.

for the E2E itself, as non crypto guy, seems the open whisper paper implemented on whatsapp alr good enough?

1. Make sure the redirect url is a valid harvestapp.com (more checks on state)

2. Encrypt the state since the start of the request, so then they can double check the state hasn't been forged by decrypt and compare

Is there any option beside those?

Comments URL: https://news.ycombinator.com/item?id=34892279

Points: 5

# Comments: 4

Or are you saying that we can just create another similar table with only avg_user_session and then join by the dimension key?

In our benchmark, we tried aggregating around 1 billion rows of raw data (2 months data) using count exact distinct -> could achieve around 50-60 seconds. If we use the HLL, the query can be finished around 20-30 seconds.

For the retention part, we're planning to keep it 1 year of data, so around 6 billion rows.

Regarding the 1st point, let say I already have the process implemented in place. If I want to add new common column, is there a better way to easily add new common metrics/dimension without doing backfill?

I've experimented with OLAP engine such as Clickhouse and its view, so far looks good but need a lot of investment upfront to maintain it

I guess the way you mentioned is the only way we can do and we'll try to optimize from that

Cheers!

So imagine, you have this table called A with this structure

+----------+--------------+---------+----------+

| location | order_count | gmv | net_gmv |

+----------+--------------+---------+----------+

| TX | 1000 | 9000.0 | 8000.0 |

| FL | 1000 | 9000.0 | 8000.0 |

+----------+--------------+---------+----------+

then you want to have another table called B with this structure

+-------+--------------+---------+----------+

| age | order_count | gmv | net_gmv |

+-------+--------------+---------+----------+

| 20-30 | 1000 | 9000.0 | 8000.0 |

| 30-40 | 1000 | 9000.0 | 8000.0 |

| 40-50 | 1000 | 9000.0 | 8000.0 |

+-------+--------------+---------+----------+

The location and age are the dimension needed for the report, eventually we'll be having different dimension needed for our report. What we're doing now is we develop a Spark-SQL job for each table. But we think this is not gonna scale because every time we want to add new dimension, we need to develop the Spark-SQL job again (same logic but different group by dimension)

So I'm wondering whether there's a better way to do this. Anyone has any experience with this kind of problem before? Any pointer how to do this efficiently (I'm thinking someone could just specify the dimension they need and there'll be a script where it'll automatically generate the new table based on the specified dimension)

Thanks

Comments URL: https://news.ycombinator.com/item?id=31662713

Points: 2

# Comments: 9