I don't use StackOverflow at all anymore.

Unfortunately, it seems like they are neglected other than being an afterthought in building semantic searches, and don't get the attention they deserve as tools to build things with.

I went ahead and embedded the entire The Online Plain Text English Dictionary (OPTED), using OpenAI's `text-embedding-3-large` and uploaded it to kaggle, but it seems to me it has no traction. Reenforcing my view that embeddings don't get the love they deserve.

Comments URL: https://news.ycombinator.com/item?id=44720860

Points: 2

# Comments: 1

Ultimately, I want to make a tool where you can plug in any arbitrary document store/scraper to an LLM with RAG. But I think we are still not there yet in terms of all purpose scrapers.

- TalkToYoutuber[1]: Download the transcripts from a youtube channel and hook it up to gtp4 with RAG to let me "talk" to youtubers. There's a bunch of youtubers who have useful knowledge to share, but no blog or wiki, so semantic searching their video transcripts is the next best thing.

- YoutubeThumbnailSearch[2] - Embded all of a youtube channels thumbnails using CLIP and search them using text. I often need to search through news channels with 10k+ videos, often in foreign languages, so not having to rely on the title or transcript but the video thumbnail helps. This is a much more niche usecase tbf.

I am thinking of making a scaled down version of [1] so I can "talk" to long videos, like conference speeches or university lectures. Should take an hour or two to cook it up since it will reuse most of the code from [1].

---

[1] https://github.com/FardinAhsan146/TalkToYoutuber

[2] https://github.com/FardinAhsan146/YoutubeThumbnailSearch

This also asks for a search for better social technology, as opposed to asserting that we must slow down the search for better physical technology because the social technology isn't keeping up.

Let's say that some unspecified statistical technique found two distinct clusters from some given raw data. These two clusters almost exactly line up with another set of clusters, and the reason for that another set of clustering is Z.

Implying that the first set isn't because of Z but it's probably all a confounder is... technically true, but extremely misleading for just about any and all sensemaking that we do.

If we applied this level of epistemic caution to anything at all, we would never pull conclusions about anything ever.

---------------

I'll say the quite part out loud. Blank slatists and their political ilk are not this careful about statistical inferences when it aligns with their preconceptions.

This is a problem that is very easy to solve if you were allowed to use proxies for intelligence and grit. Just test for IQ and trait conscientiousness. [1]

----

[1] IQ is a stronger prediction of on the job performance than anything else. https://www.steveloh.org/news/2020/5/27/best-predictor-of-jo...

What does the second part of the sentence have anything to do with creating tech companies?