Hacker News: beckhamc

New comment by beckhamc in "Questionable Practices in Machine Learning"

beckhamc — Sun, 06 Oct 2024 21:16:54 +0000

I'm not sure if this is directly mentioned in the paper, but I didn't see any mention specifically about the conflation between a validation set and test set. When people actually make a distinction between the two (which is seemingly not all that common nowadays), you're meant to perform model selection on the validation set, i.e. find the best HPs such that you minimise `loss(model,valid_set)`. Once you've found your most performant model according to that, you then evaluate it on the test set once, and that's your unbiased measure of generalisation error. Since the ML community (and reviewers) are obsessed with "SOTA", "novelty", and bold numbers, a table of results purely composed of test set numbers is not easily controllable (when you're trying to be ethical) from the point of view of actually "passing" the peer review process. Conversely, what's easily controllable is a table full of validation set numbers: just perform extremely aggressive model validation on your model until your model gets higher numbers than everything else. Even simpler solution, why not just ditch the distinction between the valid and test set to begin with? (I'm joking, btw.) Now you see the problem.

Questionable Practices in Machine Learning

beckhamc — Sun, 06 Oct 2024 20:33:59 +0000

Article URL: https://arxiv.org/abs/2407.12220

Comments URL: https://news.ycombinator.com/item?id=41760069

Points: 6

# Comments: 1

New comment by beckhamc in "Were RNNs all we needed?"

beckhamc — Fri, 04 Oct 2024 17:58:22 +0000

Your description of tanh isn't even correct, it squashes a real number to `(-1, 1)`, not "less than one".

You're curious about whether there is gain in parameterising activation functions and learning them instead, or rather, why it's not used much in practice. That's an interesting and curious academic question, and it seems like you're already experimenting with trying out your own kinds of activation functions. However, people in this thread (including myself) wanted to clarify some perceived misunderstandings you had about nonlinearities and "why" they are used in DNNs. Or how "squashing functions" is a misnomer because `g(x) = x/1000` doesn't introduce any nonlinearities. Yet you continue to fixate and double down on your knowledge of "what" a tanh is, and even that is incorrect.

New comment by beckhamc in "Were RNNs all we needed?"

beckhamc — Fri, 04 Oct 2024 03:38:55 +0000

Learnable parameters on activations do exist, look up parametric activation functions.

New comment by beckhamc in "Were RNNs all we needed?"

beckhamc — Fri, 04 Oct 2024 01:16:24 +0000

How was that person derailing the convo? Nothing says an activation function has to "squash" a number to be in some range. Leaky ReLUs for instance do `f(x) = x if x > 0 else ax` (for some coefficient `a != 0`), that doesn't squash `x` to be in any range (unless you want to be peculiar about your precise definition of what it means to squash a number). The function takes a real in `[-inf, inf]` and produces a number in `[-inf, inf]`.

> Sure there's a squashing function on the output to keep it in a range from 0 to 1 but that's done BECAUSE we're just adding up stuff.

It's not because you're "adding up stuff", there is specific mathematical or statistical reason why it is used. For neural networks it's there to stop your multi layer network collapsing to a single layer one (i.e. a linear algebra reason). You can choose whatever function you want, for hidden layers tanh generally isn't used anymore, it's usually some variant of a ReLU. In fact Leaky ReLUs are very commonly used so OP isn't changing the subject.

If you define a "perceptron" (`g(Wx+b)` and `W` is a `Px1` matrix) and train it as a logistic regression model then you want `g` to be sigmoid. Its purpose is to ensure that the output can be interpreted as a probability (given that use the correct statistical loss), which means squashing the number. The inverse isn't true, if I take random numbers from the internet and squash them to `[0,1]` I don't go call them probabilities.

> and not only is it's PRIMARY function to squash a number, that's it's ONLY function.

Squashing the number isn't the reason, it's the side effect. And even then, I just said that not all activation functions squash numbers.

> All the training does is adjust linear weights tho, like I said.

Not sure what your point is. What is a "linear weight"?

We call layers of the form `g(Wx+b)` "linear" layers but that's an abused term, if g() is non-linear then the output is not linear. Who cares if the inner term `Wx + b` is linear? With enough of these layers you can approximate fairly complicated functions. If you're arguing as to whether there is a better fundamental building block then that is another discussion.

New comment by beckhamc in "Ask HN: Former gifted children with hard lives, how did you turn out?"

beckhamc — Mon, 16 Sep 2024 02:40:13 +0000

ACE 7, recently got my PhD in ML and am doing reasonably well, at least from an 'objective' lens (I have a well paying job and bought my first condo recently). My mind is still chaotic as ever (I have a mix of OCD / Tourettes / PTS) and it's still often hard for me to concentrate without overthinking some portion of my life or something completely unrelated to the task at hand.

If I can plug a Youtuber who really 'gets it' (mental health and depression), it's Dr Scott Eilers. Everything else is just clichéd garbage.

(Disclaimer: I am not 'gifted' but did very academically well during high school and uni)

New comment by beckhamc in "Ask HN: Machine learning engineers, what do you do at work?"

beckhamc — Sat, 08 Jun 2024 17:32:46 +0000

no we just replaced feature engineering with architectural engineering

New comment by beckhamc in "Why is machine learning 'hard'? (2016)"

beckhamc — Tue, 23 Jan 2024 23:31:56 +0000

The issue is the obsession with benchmark datasets and their flaky evaluation

New comment by beckhamc in "John Carmack and John Romero reunited to talk DOOM on its 30th Anniversary"

beckhamc — Mon, 11 Dec 2023 06:37:27 +0000

sophisticated AI != aimbot accuracy

New comment by beckhamc in "Yabai – A tiling window manager for macOS"

beckhamc — Thu, 30 Nov 2023 23:07:27 +0000

I wish I had the ability to toggle PiP for any open window, while I am in full screen mode. For instance, I have both Chrome and Emacs side by side full screen and I can use a hotkey to drop down my iTerm window over both of them. (Basically like a Quake terminal but that feature is specific to iTerm).

New comment by beckhamc in "Ask HN: Could you share your personal blog here?"

beckhamc — Tue, 04 Jul 2023 20:15:09 +0000

http://beckham.nz/blog

Mostly theoretical-ish deep learning stuff as of late (I'm a PhD candidate in that field). But I want to expand it into really anything: psychology, dating, video game reviews, etc.

New comment by beckhamc in "P < 0.05 Considered Harmful"

beckhamc — Tue, 11 Apr 2023 17:26:48 +0000

A relevant quote here seems to be: "if a measure becomes a target, it ceases to become a good measure". Also true of academia in general because of the obsession with citations.

Techniques for label conditioning in Gaussian denoising diffusion models

beckhamc — Sat, 25 Mar 2023 21:44:01 +0000

Article URL: https://beckham.nz/2023/01/27/ddpms_guidance.html

Comments URL: https://news.ycombinator.com/item?id=35307751

Points: 1

# Comments: 0

New comment by beckhamc in "Weka Violates MinIO's Open Source Licenses"

beckhamc — Sat, 25 Mar 2023 16:11:44 +0000

Not to be confused with open source ML toolkit WEKA: https://en.m.wikipedia.org/wiki/Weka_(machine_learning)

Huh, that's funny, I guess which one came first?

New comment by beckhamc in "TabFS – a browser extension that mounts the browser tabs as a filesystem"

beckhamc — Sat, 18 Feb 2023 18:22:29 +0000

Nice work! Is it possible to somehow switch to a tab somehow using this? (Use case: switching browser tabs within Emacs). Thanks