Comments URL: https://news.ycombinator.com/item?id=47472719

Points: 2

# Comments: 0

https://github.com/doctly/switchboard/releases/tag/v0.0.8

Comments URL: https://news.ycombinator.com/item?id=47351580

Points: 5

# Comments: 3

We're building Doctly.ai - PDF Extraction with AI.

We started out with document conversions to Markdown but quickly realized that most use cases were for JSON conversion. We recently launched our "Extractor Studio" where you can have AI analyze a few sample variations of your documents and come up with a schema for you and publish it to an API endpoint.

We've built a technique on top of AI models that dramatically improves run to run consistency of JSON output.

Checkout the blog post here: https://medium.com/@abasiri/introducing-doctlys-extractor-st...

Comments URL: https://news.ycombinator.com/item?id=45426585

Points: 10

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=43414002

Points: 11

# Comments: 3

If I do find anything, I'll share with you for comments before I publish the post.

One thing I’ve noticed in many benchmarks, though, is the potential for bias. I’m actually working on a post about this issue, so it’s top of mind for me. For example, in the omni benchmark, the ground truth expected a specific order for heading information—like logo, phone number, and customer details. While this data was all located near the top of the document, the exact ordering felt subjective. Should the model prioritize horizontal or vertical scanning? Since the ground truth was created by the company running the benchmark, their model naturally scored the highest for maintaining the same order as the ground-truth.

However, this approach penalized other LLMs for not adhering to the "correct" order, even though the order itself was arguably arbitrary. This kind of bias can skew results and make it harder to evaluate models fairly. I’d love to see benchmarks that account for subjectivity or allow for multiple valid interpretations of document structure.

Did you run into this when looking at the benchmarks?

On a side note, Doctly.ai leverages multiple LLMs to evaluate documents, and runs a tournament with a judge for each page to get the best data (this is only on the Precision Ultra selection).

Give it a try, no credit cards needed to try it. If you email me (ali@doctly.ai) i can give you extra free credits for testing.

https://docs.mistral.ai/capabilities/document/

I used base64 encoding of the image of the pdf page. The output was an object that has the markdown, and coordinates for the images:

[OCRPageObject(index=0, markdown='', images=[OCRImageObject(id='img-0.jpeg', top_left_x=140, top_left_y=65, bottom_right_x=2136, bottom_right_y=1635, image_base64=None)], dimensions=OCRPageDimensions(dpi=200, height=1778, width=2300))] model='mistral-ocr-2503-completion' usage_info=OCRUsageInfo(pages_processed=1, doc_size_bytes=634209)

Doctly runs a tournament style judge. It will run multiple generations across LLMs and pick the best one. Outperforming single generation and single model.