I believe most can do this with the right software.

It's not a fix, but it should extend the life considerably.

With the right software, they can be a surprisingly powerful AI host or transcoding server.

It does prefer a narrow RPM band, which is fine.

Reliability is the biggest concern TBH, but maybe that's not a huge bummer if its more of a backup/assistant engine.

I'd say the majority of instruct tunes, for instance, use OpenAI output (which is against their TOS).

But its all just research! So who cares! Or at least, that seems to be the mood.

And I don't think I was using it wrong. I know, for instance, the Chinese language models are funny about sampling since I run Yi all the time.

- Miqu 70B (General Chat)

- Deepseed 33B (Coding)

- Yi 34B (for chat over 32K context)

And of course, there are finetunes of all these.

And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with).

Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call SOTA (though mixtral is excellent for the batch size 1 speed)

Another rationale is that the Apple ecosystems tends to not use JPEG by default anyway, right? It uses HEIC or something.

Sometimes it works, but sometimes the project is just too big, I think.

tinygrad isn't in the realm of production ready though, AFAIK.

It's technically way more advanced. Not as outrageously priced as an H100 either.

AMD would be selling 48GB 7900s or AI-only W7900s if they really wanted a consumer card ramp.

They don't. Not because they can't (they literally prevent OEMs from doing so, who would double up VRAM in a heartbeat without AMD lifting a finger), but because AMD doesn't want that.

But note that SDXL is really awful in automatic1111 or vanilla HF diffusers for me. You have to use something with proper augmentations (like ComfyUI or Fooocus(which runs on ComfyUI)).

And the 4090 MSRP was outrageous to begin with.

There are some boutique hosts like Hot Aisle serving MI300s (who I really should reach out to), but for the immediate future our little startup is stuck with the big cloud providers. No MI300s for us mere mortals, not even to rent.

I can certainly appreciate frustration with the AMD stack, but be blunt, I was not impressed with Hotz's YouTube rant from before.[1] It didn't give the impression of a stable framework, and this doesn't either.

Also (at least from the end user llm inference side of things) ROCm is not nearly as unusable as it used to be. We would certainly be renting MI300s over A100s (or even H100s) if we could get any, and we use a number of different inference backends.

1: https://news.ycombinator.com/item?id=36193625

The community is entrechend in 1.5 because that's what everyone is now familiar with, IMO

And the store is price fixing with nearby stores.

I think theres a difference between opportunistic crime, and reasonably upstanding people being utterly frustrated with a market status quo.

If course there is a ton of piracy that is just straight up theft from perfectly convenient platforms, but the unacceptable, anticompetitive commercial platforms are the core that keeps the community going, I think.

https://abadguide.files.wordpress.com/2012/01/jh66.jpg?w=640

Not an OCR program or anything specialized, just some generic (vision) llm that can run on my desktop with a dumb prompt...