Comments URL: https://news.ycombinator.com/item?id=47616741

Points: 2

# Comments: 0

We just completed our most extensive internal evaluation of cognee to date, using HotPotQA as a baseline. While the results showed strong gains, they also reinforced a growing realization: we need better ways to evaluate how AI memory systems actually perform.

We ran Cognee through 45 evaluation cycles on 24 questions from HotPotQA, using ChatGPT 4o for the analysis. Each part of the evaluation process is affected by the inherent variance in GPT’s output: cognification, answer generation, and answer evaluation. We especially noticed significant variance across different metrics on small runs, which is why we chose the repeated, end-to-end approach.

We compared results using the same questions and setup with:

Mem0 Lightrag Graphiti

While they are standard in QA, EM and F1 scores reward surface-level overlap and miss the core value proposition of AI memory systems. For example, a syntactically perfect answer can be factually wrong, and a fuzzy-but-correct response can be penalized for missing the reference phrasing.

LLMs are inconsistent, that is another issue.

Even HotPotQA assumes all relevant information sits neatly in two paragraphs. That’s not how memory works. Real-world AI memory systems need to link information across documents, conversations, and knowledge domains that traditional QA benchmarks just can’t capture.

Consider the difference:

Traditional QA:

“What year was the company that acquired X founded?”

Memory Challenge:

“How do the concerns raised in last month’s security review relate to the authentication changes discussed in the architecture meeting three weeks ago?”

Only one of these tests long-term knowledge, reasoning across sources, and organizational memory—care to guess which one?

We are working on a new dataset and benchmarks to measure memory, and would love feedback!

Comments URL: https://news.ycombinator.com/item?id=44839899

Points: 1

# Comments: 1

At cognee we are working on building graph/vector based memory on top of vector and graph stores. Our pipelines had 116k runs last month and we are projecting north of a million in a few months. We need help on the infra side.

Check our OSS tool here: https://github.com/topoteretes/cognee

Open Roles: Platform Engineer - https://apply.workable.com/topoteretes-ug-haftungsbeschrankt...

We have temporal resolution mechanisms we are building, and framework is generalizable enough to build any custom logic. We have a few ideas and some things will be posted there soon.

cognee has notion of nodesets which work similar to tags: https://docs.cognee.ai/core-concepts/node-sets

And also we have graphs per user now available. So, user permisions + graph filtering

LLMs don't remember context well—they can’t ingest your data and keep it in memory. This limitation leads to lacking interactions, a lack of accuracy, and the inability to connect your data sources cheaply because developers must include long, unmanaged context in every prompt.

When we were building RAGs we saw that we can’t find the data we need and that there are too many knobs to turn in RAG frameworks. We had to tweak many parameters and also could not specify the rules we wanted the data to follow. No ontologies, no rulesets, no state or good data engineering practices and a lot of manual work. That’s why cognee

cognee builds memory that combines graph, vector, and relational stores. Here is how it works:

Adding data: When you use cognee with your AI App, it can take in any message, string, S3 bucket or even a relational database and automatically ingest it

Managing information: cognee sorts this information into semantic graph: - It extracts entities and connections between things. - It embeds the data in the vector store, It enriches data with custom ontologies you provide, that help ground the graphs and make them more reliable. - The overall information is stored in many layers of a graph and vector store that allows for finding similar information later using a variety of types of searches.

Retrieving data: When given an input query, cognee searches for and retrieves related stored information by leveraging a combination of graph traversal techniques, vector similarity, and COT techniques. It can use the internal benchmarking system to make sure that your pipelines are returning only accurate data when you need it.

---

cognee introduces a self-improving group of memory layers that cover various topics, data sources and can be customized. This reduces the need to build everything from scratch, and you can use our primitives to get started and move more quickly. On the other hand, you can do everything yourself, from start to end. We’ve designed the system to be modular and extensible.

We’ve open-sourced cognee —specifically the framework and various vector and graph database adapters, as well as our default data pipeline, cognify—under the Apache 2.0 license. This includes the ability to add, cognify, and retrieve data within your AI applications and also extend it with custom components that are just pure Python.

However, many keep features that are optimized for production use, as a part of our paid platform. We release these functionalities too, including in the last few weeks permission management + distributed pipelines(dev). These are a part of our open-source package and are available to those who in production environments need things like rate limiting and credential management. We will release a paid offering for developers to get easy API access to our platform.

Our automation tooling allows us to optimize the pipelines and find the best combination of parameters that answer questions our stakeholders have.

We’d love to hear what you think! Please feel free to try our demo, check out the code, read the research paper, and share thoughts or suggestions with us. Your feedback will help shape where we take cognee from here!

Comments URL: https://news.ycombinator.com/item?id=44169594

Points: 9

# Comments: 2

Before we built cognee, Vasilije(B Economics and Clinical Psychology) worked at a few unicorns (Omio, Zalando, Taxfix), while Boris managed large-scale applications in production at Pera and StuDocu. Laszlo joined after getting his PhD in Graph Theory at the University of Szeged.

Using LLMs to connect to large datasets (RAG) has been popularized and has shown great promise. Unfortunately, this approach doesn’t live up to the hype.

Let’s assume we want to load a large repository from GitHub to a vector store. Connectingfiles in larger systems with RAG would fail because a fixed RAG limit is too constraining in longer dependency chains. While we need results that are aware of the context of the whole repository, RAG’s similarity-based retrieval does not capture the full context of interdependent files spread across the repository.

This approach allows cognee to retrieve all relevant and correct context at inference time. For example, if `function A` in one file calls `function B` in another file, which calls `function C` in a third file, all code and summaries that further explain their position and purpose in that chain are served as context. As a result, the system has complete visibility into how different code parts work together within the repo.

Last year, Microsoft took a leap published GraphRAG - i.e. RAG with Knowledge Graphs. We think it is the right direction. Our initial ideas were similar to this paper and this got some attention on Twitter (https://x.com/tricalt/status/1722216426709365024)

Over time we understood we needed tooling to create dynamically evolving groups of graphs, cross-connected and evaluated together. Our tool is named after a process called cognification. We prefer the definition that Vakalo (1978) uses to explain that cognify represents "building a fitting (mental) picture"

We believe that agents of tomorrow will require a correct dynamic “mental picture” or context to operate in a rapidly evolving landscape.

To address this, we built ECL pipelines, where we do the following: - Extract data from various sources using dlt and existing frameworks - Cognify - create a graph/vector representation of the data - Load - store the data in the vector (in this case our partner FalkorDB), graph, and relational stores

We can also continuously feed the graph with new information, and when testing this approach we found that on HotpotQA, with human labeling, we achieved 87% answer accuracy (https://docs.cognee.ai/evaluations).

To show how the approach works we did an integration with continue.dev and built a codegraph

Here is how codegraph was implemented: We're explicitly including repository structure details and integrating custom dependency graph versions. Think of it as a more insightful way to understand your codebase's architecture. By transforming dependency graphs into knowledge graphs, we're creating a quick, graph-based version of tools like tree-sitter. This means faster and more accurate code analysis. We worked on modeling causal relationships within code and enriching them with LLMs. This helps you understand how different parts of your code influence each other. We created graph skeletons in memory which allows us to perform various operations on graphs and power custom retrievers.

If you want to integrate cognee into your systems or have a look at codegraph, our GitHub repository is (https://github.com/topoteretes/cognee)

Thank you for reading! We’re definitely early and welcome your ideas and experiences as it relates to agents, graphs, evals, and human+LLM memory.

Comments URL: https://news.ycombinator.com/item?id=43031915

Points: 6

# Comments: 1

We built the tool with four main concepts in mind:

1. Loader -> uses dlt in the backend to load and structure the data

2. Cognify step -> creates a graph with summaries, labels and factoids that are interconnected across the documents and stored as a representation in the vector store

3. Optimizer -> Uses DSPy to optimize LLM queries, and we plan to extend it to most of the knobs we can turn, like chunking etc.

4. Search -> allows for searching using search types supported in graph stores (ex. Neo4j) or hybrid, BM25, or other search types available in vector stores.

We are quite early with the product but we would love to hear feedback on what we can improve.

Comments URL: https://news.ycombinator.com/item?id=40117217

Points: 7

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=36480598

Points: 3

# Comments: 1