Comments URL: https://news.ycombinator.com/item?id=45863229

Points: 1

# Comments: 0

On Gemini and other VLMs - we excluded these models because they don't do visual grounding - aka they don't provide page layouts, bounding boxes of elements on the pages. This is a table stakes feature for use-cases customers are building with Tensorlake. It wouldn't be possible to build citations without bounding boxes.

On pricing - we are probably the only company offer a pure on-demand pricing without any tiers. With Tensorlake, you can get back markdown from every page, summaries of figures, tables and charts, structured data, page classification, etc - in ONE api call. This means we are running a bunch of different models under the hood. If you add up the token count, and complexity of infrastructure to build a complex pipeline around Gemini, and other OCR/Layout detection model I bet the price you would end up with won't be any cheaper than what we provide :) Plus doing this at scale is very very complex - it requires building a lot of sophisticated infrastructure - another source of cost behind modern Document Ingestion services.

Comments URL: https://news.ycombinator.com/item?id=45559572

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=44956119

Points: 2

# Comments: 0

The other thing is segmenting a document and linearizing it so that an LLM can understand the content better. Layout understanding helps with figuring out the natural reading order of various blocks of the page.

This is exactly the reason why Computer Vision approaches for parsing PDFs works so well in the real world. Relying on metadata in files just doesn't scale across different source of PDFs.

We convert PDFs to images, run a layout understanding model on them first, and then apply specialized models like text recognition and table recognition models on them, stitch them back together to get acceptable results for domains where accuracy is table stakes.

Comments URL: https://news.ycombinator.com/item?id=43997563

Points: 4

# Comments: 1

On information dense pages, LLMs often hallucinate half of the times, they have trouble understanding empty cells in tables, doesn't understand checkboxes, etc.

We had to invest heavily into building a state of the art layout understanding model and finally a table structure understanding for reliability. LLMs will get there, but there are some ways to go there.

Where they do well is in VQA type use cases, ask a question, very narrowly scoped, they will work much better than OCR+Layout models, because they are much more generalizable and flexible to use.

Bringing in orchestration logic in the app layer means there is more code being bundled with the app, which has its own set of tradeoffs - like bringing in a different set of code dependencies which might conflict with application code.

In 2025, I would be surprised if a good workflow engine didn’t have a completely server-less development mode :)

Even with a centralized scheduler it should be possible to create a DevEx that makes use of decorators to author workflows easily.

We are doing that with Indexify(https://github.com/tensorlakeai/indexify) for authoring data intensive workflows to process unstructured data(documents, videos, etc) - it’s like Spark but uses Python instead of Scala/SQL/UDFs. Indexify’s scheduler is centralized and it uses RocksDB under the hood for persistence, and long term we are moving to a hybrid storage system - S3 for less frequently updated data, and SSD for read cache and frequently updated data(on going tasks).

The scheduler’s latency for scheduling new tasks is consistently under < 800 microseconds on SSDs.

This is how schedulers have been designed traditionally that have a proven track record of working in production - Borg, Hashicorp Nomad, etc. There are many ways a centralized scheduler can scale out beyond a single machine - parallel scheduling across different by sharding jobs, node pools, and then linearizing and deduplicating conflicts during writes is one such approach.

Love DBOs and Hatchet! cheering for you @jedberg and @abelanger :-)

Disclaimer - I am the founder of Tensorlake, and worked on Nomad and Apache Mesos in the past.

Why did we build this?

We couldn't find a self contained framework independent, re-ranking library for Typescript. We also wanted to swap models and algorithms easily, publish and track latency metrics closely. We implemented two different re-ranking algorithms - 1. LLM based re-ranking: It uses the algorithm presented in the paper - "Is ChatGPT Good at Search?" https://arxiv.org/abs/2304.09542 - they implement a sliding window based algorithm to re-rank search results which could be potentially larger than the context length of an LLM. We added support for LLama3 and GPT-4. For Llama3, we are using Groq, but other model providers can be added easily. 2. Reciprocal Rank Fusion - A lightweight algorithm to merge search results from more than one index while preserving their relative importance.

We recently built a consumer application which indexes 100s of 1000s of images using Indexfiy (https://getindexfiy.ai), which indexes various aspects of the image - caption(using a text embedding model), visual descriptions(using a VLM), CLIP embeddings. During the retrieval process we lookup 40 images from each of the index based on user queries, and then use this library to re-rank the results. The results are frankly amazing, compared to not re-ranking at all.

Latency - Latency is a big deal for applications which humans interact with, and LLama3 8B on Groq is the fastest LLM re-ranker based on our experience. They are able to process ~1000 tokens/s. We are able to re-rank 100 images in roughly 1.4 seconds. We haven't tried using GPT4 in production to be able to share any latency related numbers.

Choosing a model - Pick the model which has the best tradeoff for latency vs accuracy. There are many smaller re-ranking models available - Jina AI, BGE, Sentence Transformers, etc. We have heard good things about Cohere's re-ranker as well. We hope to add support for more models in the future. The library has a clean Model Provider interface, so contributions are welcome!

We are hoping this helps developers who are building RAG/any search based LLM applications in React or any other JavaScript framework. Love to hear your thoughts, and feedback to improve the library!

Comments URL: https://news.ycombinator.com/item?id=40646164

Points: 1

# Comments: 0