On the other hand that means they are weaker if competition comes along as it's expected that consumers and business would demand significantly more due to comparisons.

An M2 is according to a reddit post around 27 tflops

So < 1/10 the performance of just computation. let alone the memory.

What workflow would use something like this?

What I’m advocating is a substantial increase in this aspect—keeping model size the same while expanding dimensionality. The "curse of dimensionality" illustrates how a modest increase in dimensions leads to a significantly larger volume.

While I agree that backpropagation isn’t a complete solution, it’s ultimately just a stochastic search method. The key point here is that expanding the dimensionality of a model’s space is likely the only viable long-term direction. To achieve this, backpropagation needs to work within an increasingly multidimensional space.

A useful analogy is training a small model on random versus structured data. With structured data, we can learn an extensive amount, but with random data, we hit a hard limit imposed by the network. Why is that?

I'm honestly a bit envious of future engineers who will be tackling these kinds of problems with a 100-line Jupyter notebook on a laptop years from now. If we discovered the right method or algorithm for these long-horizon problems, a 2B-parameter model might even outperform current models on everything except short, extreme reasoning problems.

The only solution I've ever considered for this is expanding a model's dimensionality over time, rather than focusing on perfect weights. The higher dimensionality you can provide to a model, the greater its theoretical storage capacity. This could resemble a two-layer model—one layer acting as a superposition of multiple ideal points, and the other layer knowing how to use them.

When you think about the loss landscape, imagine it with many minima for a given task. If we could create a method that navigates these minima by reconfiguring the model when needed, we could theoretically develop a single model with near-infinite local minima—and therefore, higher-dimensional memory. This may sound wild, but consider the fact that the human brain potentially creates and disconnects thousands of new connections in a single day. Could it be that these connections steer our internal loss landscape between different minima we need throughout the day?

However the real challenge would be:

1. bring this mapping into a AI framework for inferencing 2. We don't know the "OS" on how it runs. Just randomly triggering a neuron probably wouldn't work as there is a lot of other factors that trigger neurons.

Based on the top page today I may even be able to make the argument we can't even simulate the abilities of a fruit fly.

The absolute frontier models can perform only a fraction of a fraction of what a typical work day looks like for a human. I calculated the chance that a frontier model today has about a 1^-29 chance of performing a single day of connected tasks based on GAIA benchmarks.

So it may generate 10 Billion answers to fusion and only 1-10 are correct.

There would be no way to know which one is correct without first knowing the answer to the question.

This is my main issue with these methods. They assume the future via RL then when it gets it right they mark that.

We should really be looking at methods of percentage it was wrong rather then it was right a single time.

Impressive but the problem with RL is that it requires knowledge of the future.

When I do interviews (probably limited compared to you but some) I do it like I wish someone would interview me.

I focused purely on curiosity. how many things disparate things are they interested in, the things that overlap with my knowledge I probe deep. I believe in Einsteins quote.

"I have no special talents. I am only passionately curious."

If someone knows about how RDMA and GPU MIGS work they are probably pretty damn interested in how HPC clusters function. More importantly can they compress this information and explain it so that a non technical person could understand?

There are so many endless number of questions I could ask someone to prob their knowledge of a technical field it kind of upsets me that most of the time people ask the most shallow of questions.

I believe this is because most people actually study their fields a very limited amount, because most people are honestly not truly interested in what they do.

The biggest implication of this is that I may be able to tell if someone has this trait but I understand that the majority of people could not as they literally don't know the things they could ask.

Asking system designs of me if you aren't knowledgable of the field would probably be the easiest to see the complexity of systems I can build.

Was originally building firmware and hardware for previous company while testing DS on their systems. They liked my work and I switched to DS and ended up a team lead.

Honestly have never had a take home exercise but would love it if I could. I basically make my own work, if I don't get work I will build other projects for them and try to sell it to the company.

I normally make good value projects and can sell it. It's how I went from hardware to DS lead of a team in a few years.

I do these things because I see a place in the company to grow value and do it.

I can write almost all basic coding and my true skill set is in custom complex DS pipelines at scale.

I study hours every day for many years now. I know many complex systems however studying algorithms bore me to tears.

I've built HPC clusters, k8s clusters, Custom DL method, custom high performance file system, low level complex image analysis algorithms, firmware, UIs, custom OS work.

I've done a lot of stuff because I can't help wanting to learn it. But I fail even basic leetcode questions.

Am I a bad engineer?

There seems to be no way for me to show my abilities to companies other than passing a leetcode but at the same time stopping learning DL methods to learn leetcode feels painful. I only want to learn the systems that create the most value for a company.

I imagine if you interviewed me you would think I wrote a fraudulent resume. Not sure how I am supposed to convince someone otherwise though. Perhaps I've been dumb in not working on code that can be seen outside of a company.

have a human created story and text as a guideline.

With that have genAI make the text per stage, you would get different statements every time and would stay on track.

Would be interesting to play a game where all players say the same information in slightly different ways every single playthrough.

I think hallucinations are a major unsearched gateway to AGI.

It's actually not very easy to achieve this. I could give a very long winded answer (don't tempt me) but suffice to say it's a resolution problem.

All AI have a fixed resolution on creation. Long running tasks focus on a very particular narrowing space per step, the resolution required for an infinite task is infinite resolution.

No 9s of error will ever fix this.

Funny enough, small animals do this with ease so I strongly disagree the idea that our AI outcompete even small mammals in every way.

Comments URL: https://news.ycombinator.com/item?id=40242261

Points: 2

# Comments: 0