- Lazy loading - Virtual scrolling (allows millions of rows) - "Infinite Pixel Technique" (allows billions of rows)

Hyperparam sponsored Sylvain’s work as part of our broader effort to invest in open-source infrastructure and get ahead of the data-scale problems that are emerging with LLMs. With a regular table, you can view thousands of rows, but the browser breaks pretty quickly. We created HighTable with virtual scroll so you can see millions of rows, but that still wasn’t enough for massive unstructured datasets. What Sylvain has built virtualizes the virtual scroll so you can literally view billions of rows—all inside the browser. His write-up goes deep into the mechanics of building a ridiculously large-scale table component in react.

Comments URL: https://news.ycombinator.com/item?id=46994944

Points: 3

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=46557332

Points: 1

# Comments: 4

More technical details in the post. Feedback welcome.

Comments URL: https://news.ycombinator.com/item?id=46435321

Points: 2

# Comments: 0

Here’s my problem: I have gigabytes of LLM conversation logs in parquet in S3. I want to add per-row annotations (llm-as-a-judge scores), ideally without touching the original text data.

So for a given dataset, I want to add a new column. This seemed like a perfect use case for Iceberg. Iceberg does let you evolve the table schema, including adding a column. BUT you can only add a column with a default value. If I want to fill in that column with annotations, ICEBERG MAKES ME REWRITE EVERY ROW. So despite being based on parquet, a column-oriented format, I need to re-write the entire source text data (gigabytes of data) just to add ~1mb of annotations. This feels wildly inefficient.

I considered just storing the column in its own table and then joining them. This does work but the joins are annoying to work with, and I suspect query engines do not optimize well a "join on row_number" operation.

I've been exploring using little-known features of parquet like the file_path field to store column data in external files. But literally zero parquet clients support this.

I'm running out of ideas for how to work with this data efficiently. It's bad enough that I am considering building my own table format if I can’t find a solution. Anyone have suggestions?

Comments URL: https://news.ycombinator.com/item?id=46358172

Points: 2

# Comments: 2

What have you found work well for this? I’m trying to fine-tune on a text dataset and be able to inspect the output from eval runs. I would prefer local and open source tools to a paid service.

Comments URL: https://news.ycombinator.com/item?id=46305665

Points: 2

# Comments: 0

With OPFS + Parquet + Wasm, the browser already has everything it needs to handle multi-GB LLM datasets client-side.

Is the world of data UIs evolving? Are there new data tools and best practices beyond notebooks and DuckDB?

Comments URL: https://news.ycombinator.com/item?id=46221195

Points: 1

# Comments: 3

No human has the patience to sift through all that text, so we need better tools to help us understand and analyze it. That's why I built Hyperparam to be the first tool specifically designed for working with LLM data at scale. No one else seemed to be solving this problem.

Today we’re launching Hyperparam, a browser-native app for exploring and transforming multi-gigabyte datasets in real time. It combines a fast UI that can stream huge unstructured datasets with an army of AI agents that can score, label, filter, and categorize them. Now you can actually make sense of AI-scale data instead of drowning in it.

Example: Using the chat, ask Hyperparam’s AI agent to score every conversation in a 100K-row dataset for sycophancy, filter out the worst responses, adjust prompts, regenerate, and export your dataset V2. It all runs in one browser tab with no waiting and no lag.

It’s free while it’s in beta if you want to try it on your own data.

Comments URL: https://news.ycombinator.com/item?id=45981930

Points: 16

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=45919418

Points: 1

# Comments: 1

What I've found is that moving data processing toward the browser has been for one, a refreshing developer experience because I don't need to build a pair of backend+frontend. From a user experience point of view, I think you can build MORE interactive data applications by pushing it toward the frontend.

I wanted to see what happens when we treat the browser like part of the data stack, using pure JavaScript to load, slice, and explore datasets interactively. That experiment led to a small set of open-source tools — Hyparquet and HighTable. They’re designed to test the limits of browser-native data processing to see where the browser stops being a thin client and starts acting like a real data engine.

Curious what others think about the future of browser-first data tools:

- Where do you see the practical limits for client-side data processing? - What would make browser-based architectures a viable alternative to traditional data stacks?

Comments URL: https://news.ycombinator.com/item?id=45837156

Points: 1

# Comments: 2

- Make a faster, more interactive viewer for AI datasets (which are mostly parquet format)

- Simplify the stack by doing everything from the browser (no backend)

TLDR: My open-source library Hyparquet can load data in 155ms, which would take 3466ms in duckdb-wasm for the same file.

Comments URL: https://news.ycombinator.com/item?id=44672363

Points: 16

# Comments: 1

But whats important is that with Hyperparam you can do it in the browser, where the bottleneck will always be network-bound not cpu-bound.

I was trying to look at, filter, and transform large AI datasets, and I was frustrated with how bad the existing tool was for working with datasets with huge amounts of text (web scrapes, github dumps, reasoning tokens, agent chat logs). Jupyter notebook is woefully bad at helping you to look at your data.

So I wanted to build better browser tools for working with AI datasets. But to do that I had to build these tools (there was no working parquet implementation in JS when I started).

Anyway I'm still working on building an app for data processing using LLM chat assistant to help a single user curate entire datasets singlehandedly. But for now I'm releasing these components to the community as open source. And having them "do a single task each" was very much intentional. Thanks for the comment!