Comments URL: https://news.ycombinator.com/item?id=47272174

Points: 1

# Comments: 0

For dense LLMs, like llama-3.1-8B, you profit a lot from having all the weights available close to the actual multiply-accumulate hardware.

With MoE, it is rather like a memory lookup. Instead of a 1:1 pairing of MACs to stored weights, you suddenly are forced to have a large memory block next to a small MAC block. And once this mismatch becomes large enough, there is a huge gain by using a highly optimized memory process for the memory instead of mask ROM.

At that point we are back to a chiplet approach...

Comments URL: https://news.ycombinator.com/item?id=46751967

Points: 2

# Comments: 0

I noted that LAuReL is cited in the mHC paper, but they refer to it as "expanding the width of the residual stream", which is rather odd.

Gemma 3n is also using a low-rank projection of the residual stream called LAuReL. Google did not publicize this too much, I noted it when poking around in the model file.

https://arxiv.org/pdf/2411.07501v3

https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3...

Seems to be what they call LAuReL-LR in the paper, with D=2048 and R=64

They are, of course, a bit slower in EUV adoption. But its already there:

https://www.tomshardware.com/pc-components/dram/micron-sampl...

https://www.techinsights.com/blog/samsung-d1z-lpddr5-dram-eu...

https://patents.google.com/?q=(H03F3%2f16)&sort=old

The Matare/Welker Patent is missing though

https://patents.google.com/patent/US2673948A/en.541

The entire debate is tiring. It would be better if these reviews would put the actual device physics of the different concepts into context.

Is there any report of a reproduction of the device proposed by Lilienfeld in his patents? If he managed to make functional devices back then, it should be possible today? (Note: Cu2S is not a very well controllable semiconductor...)

Edit:

Gemini Deep Research summary here, its quite informative: https://docs.google.com/document/d/1jE0wQVeWP9Eiybh_C6zMKeZ5...

Also specifically on Cu based TFT: https://docs.google.com/document/d/1_B2x2gBPKgGFVgJyQ0qzPdI4...

From the second document: "The primary obstacle for $Cu_2S$ TFTs is degeneracy. Spontaneous copper vacancies form with negligible energy cost in the sulfur lattice. As a result, stoichiometric $Cu_2S$ is thermodynamically unstable in air, rapidly oxidizing or losing copper to form substoichiometric phases ($Cu_{2-x}S$) with hole concentrations exceeding $10^{20}-10^{21} \text{ cm}^{-3}$."

This explains why there are zero reproductions of Lilienfelds devices. It should be noted that Lilienfeld is one of the inventors of electrolytic capacitors and did therefore know where well how to create extremely thin insulating layers as needed for TFTs. It is not impossible to assume that he could have used other semiconductors (e.g. CdS) with his concept. However, the patents seems to specifically mention Cu2S, which does not yield functional TFTs.

This eliminates the need for more specialized models and the associated engineering and optimizations for their infrastructure needs.

The public discourse about LLM assisted coding is often driven by front end developers or rather non-professionals trying to build web apps, but the value it brings to prototyping system concepts across hardware/software domains can hardly be understated.

Instead of trying to find suitable simulation environments and trying to couple them, I can simply whip up a gui based tool to play around with whatever signal chain/optimization problem/control I want to investigate. Usually I would have to find/hire people to do this, but using LLMs I can iterate ideas at a crazy cadence.

Later, implementation does of course require proper engineering.

That said, it is often confusing how different models are hyped. As mentioned, there is an overt focus on front end design etc. For the work I am doing, I found Claude 4.5 (both models) to be absolutely unchallenged. Gemini 3 Pro is also getting there, but long term agentic capability still needs to catch up. GPT 5.1/codex is excellent for brainstorming in the UX, but I found it too unresponsive and intransparent as a code assistant. It does not even matter if it can solve bugs other llms cannot find, because you should not put yourself into a situation where you don't understand the system you are building.

It seems Grok 4.1 uses more emojis than 4.

Also GPT5.1 thinking is now using emojis, even in math reasoning. 5 didn't do that.

Comments URL: https://news.ycombinator.com/item?id=45758896

Points: 5

# Comments: 0

(where x is a number dependent on architectural features like MLHA, QGA...)

There is this thing called KV cache which holds an enormous latent state.

SNNs are more similar to pulse density modulation (PDM), if you are looking for an electronic equivalent.

If we just look at spikes as a different numerical representation, then they are clearly inferior. For example, consider that encoding the number 7 will require seven consecutive pulses on a single spiking line. Encoding the number in binary will require one pulse on three parallel lines.

Binary encoding wins 7x in speed and 7/3=2.333x in power efficiency...

On the other hand, if we assume that we are able to encode information in the gaps between pulses, then things quickly change.