Buyers should have all the facts

After using it for a couple hours (on Elixir code) with Qwen2.5-Coder-3B and no attempts to customize it, this checks a lot of boxes for me:

  - I pretty much want fancy autocomplete: filling in obvious things and saving my fingers the work, and these suggestions are pretty good
  - the default keybindings work for me, I like that I can keep current line or multi-line suggestions
  - no concerns around sending code off to a third-party
  - works offline when I'm traveling
  - it's fast!

    VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8

Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model.

Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU.

[0] https://blog.runpod.io/understanding-vram-and-how-much-your-...

[0] https://www.kff.org/report-section/ehbs-2024-section-1-cost-...

This analysis[1] from the Cato Institute found the opposite: “immigrants pay more in taxes than they consume in benefits”.

[1] https://www.cato.org/blog/fiscal-impact-immigration-united-s...

  - code generation makes for an extremely productive experience that makes standing up an actually-useful application very fast
  - Elixir is a great language, especially for the web, and using it to render the frontend feels like having the full power of the language plus the simplicity of HTML (with little/no writing JavaScript)
  - it's extremely efficient since only tiny changes are sent over the WebSocket when data is updated on the server
  - you're already using WebSockets, so adding any kind of real-time functionality is very easy (chat, notifications, game state)

We've hit some bugs and gotchas over the years leading up to this 1.0 release, but it has long felt like a stable, well-built library that keeps our codebases simple and maintainable, which lets you move fast.

Congratulations to Chris, Jose, and all the other wonderful contributors!

1. https://www.irs.gov/businesses/small-businesses-self-employe...

The world's oldest extant wooden structure is the Kondō (main hall) of the temple Hōryū-ji 法隆寺 in Ikaruga, in the Nara Prefecture of Japan. It was initially built in 607 but completely burned down due to lightning. It was rebuilt in 670, but again nearly burned down by accident in 1949 [1].

It's interesting to contemplate how across these timescales war, disasters, and accidents make it so difficult for structures to survive.

[1] https://en.wikipedia.org/wiki/H%C5%8Dry%C5%AB-ji

Comments URL: https://news.ycombinator.com/item?id=40846656

Points: 2

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=40324323

Points: 2

# Comments: 0

I've found LASER[1] which originally had the idea to embed all languages in the same vector space, though it's a bit harder to use than models available through SentenceTransformers. LASER2 stuck with this approach, but LASER3 switched to language-specific models. However, I haven't found benchmarks for these models, and they were released about 2 years ago.

Another alternative would be to translate everything before embedding, which would introduce some amount of error, though maybe it wouldn't be significant.

1. https://github.com/facebookresearch/LASER

https://huggingface.co/spaces/mteb/leaderboard

Bitcoin: 145B = 145,000M kWh/year

ChatGPT: 0.5M * 365 = 182M kWh/year

Based on the numbers from the article, ChatGPT is using 3 orders of magnitude less electricity for something that provides high utility for vastly more people.

There are of course many other uses of AI aside from ChatGPT and more cryptocurrencies aside from BTC, but these are very different power consumptions.

Comments URL: https://news.ycombinator.com/item?id=39197917

Points: 34

# Comments: 15

> embedding-based approach will be cheaper and faster, but worse result than full text

I’m not sure results would be worse, I think it depends on the extent to which the models are able to ignore irrelevant context, which is a problem [2]. Using retrieval can come closer to providing only relevant context.

1. https://huggingface.co/spaces/mteb/leaderboard

2. https://arxiv.org/abs/2302.00093

Comments URL: https://news.ycombinator.com/item?id=36973090

Points: 18

# Comments: 0

It works for 7B/13B/30B/65B LLaMA and Alpaca (fine-tuned LLaMA which definitely works better). The smaller models at least should run on pretty much any computer.