Comments URL: https://news.ycombinator.com/item?id=47662833

Points: 44

# Comments: 7

Comments URL: https://news.ycombinator.com/item?id=46151778

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45363901

Points: 4

# Comments: 0

A high level summary is that while this is an impressive model, it underperforms even current SOTA VLMs on document parsing and has a tendency to hallucinate with OCR, table structure, and drop content.

Comments URL: https://news.ycombinator.com/item?id=43287210

Points: 15

# Comments: 0

We’ve generally found that Gemini 2.0 is a great model and have tested this (and nearly every VLM) very extensively.

A big part of our research focus is incorporating the best of what new VLMs offer without losing the benefits and reliability of traditional CV models. A simple example of this is we’ve found bounding box based attribution to be a non-negotiable for many of our current customers. Citing the specific region in a document where an answer came from becomes (in our opinion) even MORE important when using large vision models in the loop, as there is a continued risk of hallucination.

Whether that matters in your product is ultimately use case dependent, but the more important challenge for us has been reliability in outputs. RD-TableBench currently uses a single table image on a page, but when testing with real world dense pages we find that VLMs deviate more. Sometimes that involves minor edits (summarizing a sentence but preserving meaning), but sometimes it’s a more serious case such as hallucinating large sets of content.

The more extreme case is that internally we fine tuned a version of Gemini 1.5 along with base Gemini 2.0, specifically for checkbox extraction. We found that even with a broad distribution of checkbox data we couldn’t prevent frequent checkbox hallucination on both the flash (+17% error rate) and pro model (+8% error rate). Our customers in industries like healthcare expect us to get it right, out of the box, deterministically, and our team’s directive is to get as close as we can to that ideal state.

We think that the ideal state involves a combination of the two. The flexibility that VLMs provide, for example with cases like handwriting, is what I think will make it possible to go from 80 or 90 percent accuracy to some number very close 99%. I should note that the Reducto performance for table extraction is with our pre-VLM table parsing pipeline, and we’ll have more to share in terms of updates there soon. For now, our focus is entirely on the performance frontier (though we do scale costs down with volume). In the longer term as inference becomes more efficient we want to move the needle on cost as well.

Overall though, I’m very excited about the progress here.

--- One small comment on your footnote, the evaluation script with Needlemen-Wunsch algorithm doesn’t actually consider the headers outputted by the models and looks only at the table structure itself.

you'll find that most of the errors here are structural issues with the table or inability to parse some special characters. tables can get crazy!

I'd encourage you to take a look at some of our data points to compare for yourself! Link: huggingface.co/spaces/reducto/rd_table_bench

In terms of the overall importance of table extraction, we've found it to be a key bottleneck for folks looking to do document parsing. It's up there amongst the hardest problems in the space alongside complex form region parsing. I don't have the exact statistics handy, but I'd estimate that ~25% of the pages we parse have some hairy tables in them!

A ton of document parsing solutions have been coming out lately, each claiming SOTA with little evidence. A lot of these turned out to be LLM or LVM wrappers that hallucinate frequently on complex tables.

We just released RD-TableBench, an open benchmark to help teams evaluate extraction performance for complex tables. The benchmark includes a variety of challenging scenarios including scanned tables, handwriting, language detection, merged cells, and more.

We employed an independent team of PhD-level human labelers who manually annotated 1000 complex table images from a diverse set of publicly available documents.

Alongside this, we also release a new bioinformatics inspired algorithm for grading table similarity. Would love to hear any feedback!

-Raunak

Comments URL: https://news.ycombinator.com/item?id=42054144

Points: 29

# Comments: 6

Please give our demo a try with some of your own PDFs or reach out at founders@reducto.ai if you’d like to start using Reducto in production.

Comments URL: https://news.ycombinator.com/item?id=39523565

Points: 3

# Comments: 0

A few weeks ago I shared a beta of Remembrall here and got a lot of great feedback from people in this community, with one of the most common requests being to open source the project.

Excited to share that we’re doing exactly that!

Remembrall is a proxy (integrates in two lines!) on top of your OpenAI queries that uses GPT to save/update important details from each user’s conversations into a vector db. When the user continues the conversation we query the db for relevant info and prepend it into the system prompt. We have a lot of improvements in the works (function calling, queryable user profiles, and more) and would love your feedback on features you’d like to see!