Overall I'm really happy to see Typesense mentioned here.

A lot of the smaller scale RAG projects, etc you see around would be well served by Typesense but it seems to be relatively unknown for whatever reasons. It's probably one of the easiest solutions to deploy, has reasonable defaults, good docs, easy clustering, etc while still be very capable, performant, and powerful if you need to dig in further.

How are you involved in the hiring process?

> Our engineers are fucking morons. And this guy was the dumbest of the bunch.

Very indicative of a toxic culture you seem to have been pulled in to and likely have contributed to by this point given your language and broad generalizations.

Describing a wide group of people you're also responsible for as "fucking morons" says more about you than them.

- It's a cornerstone of their brand.

- R2 users are at least paying for something (storage).

- Their network is massively overbuilt to be able to absorb DDoS attacks.

- They offer free bandwidth with their CDN - including completely free users. These resources need to get fetched from origin and/or cached. R2 doesn't have to fetch from origin which eliminates the bandwidth required for the fetch.

Large providers typically pay very little/nothing for bandwidth other than equipment costs (ports, etc). As a large provider they have free peering to most of the "last mile" ISP/eyeball networks in the world. This benefits all parties because these ISPs don't have to pay transit providers and neither does Cloudflare and it's faster. Same goes for all of the big clouds.

People who think AWS/GCP/Azure/etc bandwidth pricing of $0.12/GB (or whatever) is fair/reasonable have no idea what bandwidth actually costs these operators. As noted it's effectively nothing and the big clouds capitalize on this ignorance by charging insane markups for bandwidth.

In AI world they have OpenVINO, Intel Neural Compressor, and a slew of other implementations that typically offer dramatic performance improvements.

Like we see with AMD trying to compete with Nvidia software matters - a lot.

The US election system is very distributed and fragmented - there is virtually no standardization.

Even in the tightest margins for something like President you'd need to have seriously good data to figure out which random municipality voting system(s) you'd need to target to actually affect the outcome.

I can provide what I can provide publicly. The first thing we ever do is develop benchmarks given the uniqueness of the nuclear energy space and our application. In this case it's FermiBench[0].

When working with operating nuclear power plants there are some fairly unique challenges:

1. Document collections tend to be in the billions of pages. When you have regulatory requirements to extensively document EVERYTHING and plants that have been operating for several decades you end up with a lot of data...

2. There are very strict security requirements - generally speaking everything is on-prem and hard air-gapped. We don't have the luxury of cloud elasticity. Sparse embeddings are very efficient especially in terms of RAM and storage. Especially important when factoring in budgetary requirements. We're already dropping in eight H100s (minimum) so it starts to creep up fast...

3. Existing document/record management systems in the nuclear space are keyword search based if they have search at all. This has led to substantial user conditioning - they're not exactly used to what we'd call "semantic search". Sparse embeddings in combination with other techniques bridge that well.

4. Interpretability. It's nice to be able to peek at the embedding and be able to get something out of it at a glance.

So it's basically a combination of efficiency, performance, and meeting users where they are. Our Fermi model series is still v1 but we've found performance (in every sense of the word) to be very good based on benchmarking and initial user testing.

I should also add that some aspects of this (like pretrained BERT) are fairly compute-intense to train. Fortunately we work with the Department of Energy Oak Ridge National Laboratory and developed all of this on Frontier[1] (for free).

[0] - https://huggingface.co/datasets/atomic-canyon/FermiBench

[1] - https://en.wikipedia.org/wiki/Frontier_(supercomputer)

When faced with a similar challenge we developed a custom tokenizer, pretrained BERT base model[0], and finally a SPLADE-esque sparse embedding model[1] on top of that.

[0] - https://huggingface.co/atomic-canyon/fermi-bert-1024

[1] - https://huggingface.co/atomic-canyon/fermi-1024

Nice!

Fortunately MTEB lets you sort by model parameter size because using 7B parameter LLMs for embeddings is just... Yuck.

I'm shocked at the number of startups, etc you see trying to do RAG, etc that basically have no idea what they are, how they actually work, etc.

The "R" in RAG stands for retrieval - as in the entire field of information retrieval. But let's ignore that and skip right to the "G" (generative)...

Garbage in, garbage out people!

Let's just say that if you think off-the-shelf embedding models are going to work well with this kind of highly specialized content you're going to have a rough time.

[0] - https://huggingface.co/atomic-canyon/fermi-1024

There are embedding approaches that balance "semantic understanding" with BM25-ish.

They're still pretty obscure outside of the information retrieval space but sparse embeddings[0] are the "most" widely used.

[0] - https://zilliz.com/learn/sparse-and-dense-embeddings

Turns out people don’t actually want GPUs, they want solutions that happen to run best on GPUs. Nvidia understands that, AMD doesn’t.

Lisa Su keeps talking about “chips chips chips” and MAYBE “Oh btw here’s a minor ROCm update”. Meanwhile, Nvidia continues to masterfully execute deeper and wider into overall solutions and ecosystems - a substantial portion of which is software.

Nvidia is at the point where they’re eating the entire stack. They do a lot of work on their own models and then package them up nice and tight for you with NIM and Nvidia AI Enterprise. On top of stuff like Metropolis, RIVA, countless things. They even have a ton of frameworks to ingest/handle data, finetune/train, and then deploy via NIM.

Enterprise customers can be 100% Nvidia for a solution. When Nvidia is the #1-#2 most valuable company in the world “no one ever got fired for buying Nvidia” hits hard.

The people who say “AMD and Nvidia are equal - it’s all PyTorch anyway” have no view of the larger picture.

With x86_64, day one you could take a drive out of an Intel system, put it in an AMD system, and it would boot and run perfectly. You can still do that today unless you build something for REALLY specific/obscure CPU instructions.

Needless to say that’s not the case with GPUs and a lot of people that make the AMD vs Intel comparison don’t seem to understand that.

https://www.theguardian.com/world/2018/jan/28/fitness-tracki...

Yeah, that's um, wild.

It’s a budgeting thing.

To your point on the P40, it's an eight year old card but fortunately Nvidia has a history of long term support (especially for "datacenter" GPUs). The Pascal series is still fully supported by the latest Nvidia driver and CUDA releases, and projects like llama.cpp are still fairly regularly adding performance optimizations for even Maxwell series GPUs!

Current V100/A100/H100/etc hardware families are not going to end up as e-waste anytime soon. In fact, compare used pricing (and demand) of GPUs to CPUs, RAM, disk, motherboards, etc from eight years ago... That hardware ends up in the trash/at e-waste recyclers much, much sooner (even with /r/homelab).

[0] - https://old.reddit.com/r/LocalLLaMA/

Anecdotally this seems to be the key impact. With social media almost everyone now has a "brand" and that brand is typically not supported with a post of you out of it, sloppy, etc.

Along those lines, there also seems to be MUCH more emphasis on health - granted superficial health (looking good) but health nonetheless. Fortunately the standards for "ideal beauty" for women especially have shifted from the 90s/2000s no-such-thing-as-too-thin dangerous and extremely unhealthy to a physique that is well-muscled and actually healthy (while being inclusive of different body types).

When I'm at the gym and the high school/college kids show up I just can't believe their level of physical fitness and development. Self-selecting given it's the gym but when I was in high school (class of 2002) the most fit kid on the football, basketball, track, volleyball, etc teams would look out of shape next to what appears to be the "average" gym-goer of this generation. The numbers also seem to be quite a bit higher - there are A LOT of these kids hitting it really hard in the gym.

Needless to say this clearly obsessive-level focus and work is not supported by using drugs like marijuana and alcohol. If nothing else having a lot of followers is much more important and "cool".

If anything I'm more interested in usage statistics of steroids and other performance-enhancing drugs. Some of the physiques, performance, etc I see just don't seem possible to achieve naturally at 16-25.

I like to remind myself that AWS is 20 years old. That's an entire generation of people from devs to C-Suite that likely don't know anything else. For many of these people all they know about hardware is their laptop. All they know about bandwidth is what they pay their local ISP. All they know about storage is (maybe) USB flash drives and what Apple charges for 256GB vs 512GB.

This is not a criticism. More of a reality check to myself and others that at this point not a lot of people outside of bigger cloud and ISPs know what an Autonomous System is or what buying transit and peering costs. Nor have they bought a cabinet of bare metal or talked to a co-lo provider.

> Not a criticism, just an observation from my experiences.

Exactly!

Your machine -> Local router -> Configured upstream DNS Server (ISP/CF/Quad8/etc) -> ? -> Authoritative DNS Server

Any one of those layers can override/mess with/cache in a variety of ways including TTL. This is why Cloudflare and a variety of other providers use IP anycast. They accepted DNS for what it is and worked around it.

Not only is the IP always the IP, the "global" BGP routing table actually universally and consistently updates much faster than DNS. Then whatever routers, machines, etc downstream from that don't matter.

Anecdotal (as is my position). I can theoretically understand this happening but not only have I never seen it, such an issue would need to be escalated. That's a "this is unacceptable" high-level phone call. A call you more than likely have a chance of someone in actual authority answering because IME unless you have SERIOUS spend with big cloud you'll be lucky to make it a rung or two up sales/support.

Plus backups and redundancies that should prevent even the failure of a chassis/storage/etc from being a significant critical issue.

> their failures tend to be you twiddling your thumbs vs hair on fire on phone with the vendor trying to get it resolved

As a Founder/CTO I have the opposite take - put me and my team in a position to /do something/ vs sitting around waiting for AWS to come back whenever it decides to and while they obscure comms, don't update the fake status dashboards, etc. Meanwhile you're telling your customer "Ummm, we don't know - Amazon has a problem. When it comes back I guess it's back".

Coming from a background of telecom, healthcare, and nuclear energy I can't believe that even flies.