To do this, I created a Python CLI tool that analyzes semantic structure using vector embeddings. It parses markdown files, generates embeddings (all-mpnet-base-v2 or OpenAI), computes cosine similarity, runs k-means clustering, detects redundancy and semantic gaps, and produces visualizations like heatmaps, dendrograms, and UMAP projections. The stack includes Python 3.12, sentence-transformers, scikit-learn, UMAP, and Plotly, with embedding caching for speed.

Analysis Overview: The site contains 25 pages (~12.9k words) across features, concepts, use cases, and resources. No stub pages were found.

Topic coherence (measured via average similarity between sections) ranged from 0.73 to 0.93, with most pages between 0.78–0.88. Lower coherence wasn’t necessarily bad — the Proof Engine page scored lower because it intentionally covers many subtopics.

Semantic redundancy showed only one pair above 0.85 similarity, both intentional cross-link sections. Earlier, I removed two index pages with 85%+ similarity to parent pages, flattening navigation from three layers to two.

No semantic gaps were detected; all pages were well connected. Hub analysis confirmed that Home, Learn, and the AEO Playbook act as central nodes, matching the intended architecture of concepts → applications → tools.

Heatmap clustering revealed:

* Concept pages: 0.65–0.80 similarity * Feature pages: 0.45–0.65 similarity * Use cases: 0.70–0.79 similarity

Embeddings were chosen over keyword analysis because they capture meaning rather than wording, detecting paraphrased overlap and relationships relevant to AI retrieval systems.

Limitations include model sensitivity, arbitrary cluster counts, and coherence scores that don’t fully account for intentional structure. Planned improvements include entity coverage analysis, competitor comparisons, and query-simulation testing.

The entire process took under a minute but prevented structural issues that could cause discoverability problems later. Running semantic analysis pre-launch helped validate architecture, reduce duplication, and ensure content works for both humans and AI retrieval systems.

Comments URL: https://news.ycombinator.com/item?id=46976895

Points: 1

# Comments: 1

Who’s on the team?

Find ceramic dinner sets with 4+ star ratings, available for delivery this week.

and get instant answers from any part of the site, including CMS content.

It doesn’t rely just on schema like traditional solutions and integrates easily—similar to Finsweet filters, but more dynamic.

It's useful in my own projects. I want to explore more use cases from others so I can make it even more useful.

If you are interested let me know would love to connect, get ideas and test with you.

Here is a basic demo: https://neue.blr1.cdn.digitaloceanspaces.com/LLMquery.mp4

Comments URL: https://news.ycombinator.com/item?id=44445468

Points: 1

# Comments: 0