Hacker News: mirker

New comment by mirker in "Dynamic Branch Prediction with Perceptrons (2000) [pdf]"

mirker — Wed, 10 May 2023 15:54:48 +0000

One thing to point out is that the threshold of predictor complexity is dependent on the execution pipeline. A very speculative and deep architecture has a bigger need for better predictors, since it has a massive penalty when there is a misprediction.

New comment by mirker in "Releasing 3B and 7B RedPajama"

mirker — Sat, 06 May 2023 01:17:36 +0000

Does anyone have experience using these open source models in production?

New comment by mirker in "MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks"

mirker — Fri, 05 May 2023 11:26:24 +0000

There is a ton of value. OpenAI having proprietary LLMs single handedly pivoted the entire field to LLMs. A random GitHub repository doesn’t come close to impact.

New comment by mirker in "MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks"

mirker — Fri, 05 May 2023 11:20:20 +0000

Because the authors don’t get a large reward for open sourcing the work and they stand to lose future value by lowering the gate to competition. You may want the code, but Google will not care (or it might dislike it).

Look at GPT-3+, OpenAI gets fame and fortune while people struggle to reproduce their last-gen models.

New comment by mirker in "IBM to pause hiring in plan to replace 7,800 jobs with AI News"

mirker — Tue, 02 May 2023 04:06:56 +0000

They asked Watson of course.

New comment by mirker in "Geoffrey Hinton leaves Google and warns of danger ahead"

mirker — Tue, 02 May 2023 00:48:36 +0000

Agreed.

Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making.

Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical?

Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:55:47 +0000

The APIs were messed up early on, which is a reason TF2 happened. Every team started making their own random implementations of stuff. You had the TF Slim API, you had Keras, etc. The API just got fatter and fatter and then libraries would make cross dependencies to bake in the API mistakes.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:44:17 +0000

Nah, TF has had dynamic execution since TF2 and it’s still losing users, it seems. The execution model and API is simply more complicated. What’s a session, placeholder, constant, tensor, …? PyTorch was sold as numpy with GPU support and it is pretty close to that. JAX is an attempt to approach language simplicity and purity.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:33:50 +0000

PyTorch examples were also cleaner. torchvision had ResNet training batteries included while TF had role your own or clone some weird Keras repository.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:27:37 +0000

I don’t understand what you mean. Here’s how many applied ML papers work: create a new dataset for a novel problem, download a PyTorch model, point model at dataset directory. Is it novel? By construction. Is the ML technique novel? No.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:13:11 +0000

The story I’ve heard is the economics undergrads can’t get into economics grad school. This is just a rumor but the sentiment is that undergrads get taught a watered down version of economics theory. Economics theory is potentially very technical and includes game theory and proofs. Even in CS, undergrads take intro theory courses and “bottom out” in their math skills, even though grad-level CS gets much more difficult. Therefore, I’d imagine the primary determinant of this rigor phenomenon is the GPA inflation of the major.

New comment by mirker in "Why did Google Brain exist?"

mirker — Thu, 27 Apr 2023 04:02:49 +0000

I agree with you but does anyone even recognize the last category outside blue-sky research? People have a tendency to bin other people into buckets. Being a master at 2 things means you can’t be easily placed in a typical team structure.

New comment by mirker in "Starbucks claims $181M in revenue from unused gift cards (2022)"

mirker — Thu, 27 Apr 2023 01:32:45 +0000

But it’s more thoughtful. The purchaser thought long and hard about what place the purchasee can use it in.

New comment by mirker in "Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug"

mirker — Sun, 23 Apr 2023 19:44:36 +0000

When people say you need the third hardware revision to get functional performance, and it still doesn’t work, you should conclude it’s poorly designed. The fact that these chips were being sold with old hardware revisions also seems anti-consumer.

New comment by mirker in "Will we run out of ML data? Evidence from projecting dataset size trends (2022)"

mirker — Sat, 22 Apr 2023 20:13:31 +0000

Yeah the issue is you can generate data, but it won’t be good data. Training over random strings won’t make you learn language, but it’s technically data.

New comment by mirker in "SafeGPT: New tool to detect LLMs' hallucinations, biases and privacy issues"

mirker — Fri, 21 Apr 2023 13:49:25 +0000

The OPs point is that it’s likely impossible to do what is claimed here in general. Imagine the LLM says something like Fermat’s Last Theorem. To verify it, you’d have to either 1) have a proof assistant powerful enough to construct a proof 2) use a second ML model to guess truthfulness. The former is technically challenging and the latter is another model, with its own biases and factual inconsistencies.

New comment by mirker in "Google DeepMind"

mirker — Fri, 21 Apr 2023 00:35:49 +0000

Many of those people left though.

New comment by mirker in "Ask HN: AI to study my DSL and then output it?"

mirker — Wed, 19 Apr 2023 19:31:44 +0000

There’s actually a few papers already on constrained decoding. I won’t link them but if you go on arxiv and really look you will find a couple in the past year.

New comment by mirker in "Samsung considers moving to Bing as default search engine"

mirker — Mon, 17 Apr 2023 20:32:06 +0000

Sure, I agree they are useful. My objection is it’s more in the tool category than science, while Alphafold is both. There isn’t convincing evidence that GPTs are pushing what we know; rather, they make it easier to process/search what we already know. You could hire an ML expert to be your tutor without GPTs and you’d get equal or better tutoring, though at a higher price. You can’t hire people to predict protein folding better than Alphafold. It’s very convenient that GPTs exist and they can provide tons of value, but they’re essentially the next version of mechanical turk or a domain expert you’d hire for contract work except more scalable. The net impact of GPTs may also be higher due to how often we use text, but I’d rather see a society curing disease, etc. than one generating fake books, etc.

New comment by mirker in "Samsung considers moving to Bing as default search engine"

mirker — Mon, 17 Apr 2023 18:42:26 +0000

Alphafold is open and seems fundamentally transformative in the science space. GPT is nice but it’s a smart meme-generator at the moment. I don’t disagree with the impact on G’s bottom line, though.