However, I don't want your bloated or minimum effort dog-shit app just to watch a movie on a plane, browse a site like Reddit, order a pizza, read a news article/blog, or shop at your specific online store. I will begrudgingly download it if I must, but I'll hate you for it.

(Until they release a new human hand sized phone at least)

...The engineer then rolls his eyes like "yeah no duh". If we could just magically do stuff like that, we would have done it already. Classic management thinking they have an original idea with no understanding of the engineering beneath it all. I thought they would just tell him off and that would be it. I really felt seen in that moment.

The frustrating thing is, they then take pointy haired boss's idea seriously. The rest of the season is spent actually pursuing that dumb, dumb idea... This felt disrespectful, and I stopped watching.

That does not seem at all to be what citation 2 is saying.

Imagine the input text as though it were the whole internet and each page is just 1 token. Your job is to build a neural-network Google results page for that mini internet of tokens.

In traditional search, we are given a search query, and we want to find web pages via an intermediate search results page with 10 blue links. Basically, when we're Googling something, we want to know "What web pages are relevant to this given search query?", and then given those links we ask "what do those web pages actually say?" and click on the links to answer our question. In this case, the "Query" is obviously the user search query, the "Key" is one of the ten blue links (usually the title of the page), and the "Value" is the content of the web page that link goes to.

In the attention mechanism, we are given a token and we want to find its meaning when contextualized with other tokens. Basically, we are first trying to answer the question "which other tokens are relevant to this token?", and then given the answer to that we ask "what is the meaning of the original token given these other relevant tokens?" The "Query" is a given token in the input text, the "Key" is another token in the input text, and the "Value" is the final meaning of the original token with that other token in context (in the form of an embedding). For a given token, you can imagine it is as though the attention mechanism "clicked the 10 blue links" of the other most relevant tokens in the input and combined them in some way to figure out the meaning of the original query token (and also you might imagine we ran such a query in parallel for every token in the input text at the same time).

So the self attention mechanism is basically google search but instead of a user query, it's a token in the input, instead of a blue link, it's another token, and instead of a web page, it's meaning.

We went in for liposuction and buccal fat removal surgery and came out the other side severely disfigured with Maralago face and a hunchback.

(Also to those who say not enough people wanted a mini phone to be worth producing: I submit the case of Prego chunky pasta sauce. Not many people want a chunky pasta sauce, but you sell a whole lot more pasta sauce in total if you sell both regular and chunky pasta sauce. Malcolm Gladwell has a TED talk about this.)

2.5 was quite good. Not stupidly good like the jump from GPT 2 to 3 or 3.5 to 4, but really good. It was a big jump in ELO and benchmarks. People like it, and I think it's just psychologically satisfying that the player everybody would have expected to win the AI race is currently in the lead. Gemini finally gets a day in the sun.

I'm sure this will change with whenever somebody comes up with the next big idea though. It probably won't take much to beat Gemini in the long run. There is literally zero moat.