Hacker News: bjenik

New comment by bjenik in "The FFT Strikes Back: An Efficient Alternative to Self-Attention"

bjenik — Wed, 26 Feb 2025 15:53:12 +0000

Working in the Fourier domain has been a staple of scientific and engineering applications. Learning those interactions rather than just hardcoding them has been fairly widely explored as well - the term to look for is Fourier Neural Operators [1][2]. It turns out you can prove universality even if the nonlinearity remains in the real domain [3].

The concept is fairly mainstream nowadays, to the degree that Jensen talked about it in his GTC keynote in 2021 [4] and there’s even a mainstage TED talk about its applications [5].

A nice property of doing things this way is that your model ends up being resolution-invariant which is particularly interesting for engineering domains. Scaling these methods has sparked the "let’s do a fully deep-learning-based weather model"-race [6][7].

As for using this on text data: my intuition would be that is going to not work as well because of a fairly unique property of text: for image, video and scientific data each individual element is of approximately equal importance, whereas in text you can have discrete tokens like a "not" somewhere in there that change the meaning of everything around it fairly significantly and you’d want that all to all interaction to capture that. Any kind of mixing that smoothes things out is going to inherently be at a disadvantage - probably true to some degree for most of those efficiency saving methods and why we’re seeing more limited adoption on text.

[1] https://arxiv.org/abs/2010.08895

[2] https://www.nature.com/articles/s42254-024-00712-5

[3] https://jmlr.org/papers/v22/21-0806.html

[4] https://www.youtube.com/watch?v=jhDiaUL_RaM&t=2472s

[5] https://www.ted.com/talks/anima_anandkumar_ai_that_connects_...

[6] https://arxiv.org/abs/2202.11214 (Feb 2022)

[7] https://www.wired.com/story/ai-hurricane-predictions-are-sto...

How AI based programming could work (2016)

bjenik — Fri, 17 Mar 2023 21:32:28 +0000

Article URL: https://bjenik.com/AIBasedProgramming/

Comments URL: https://news.ycombinator.com/item?id=35203199

Points: 4

# Comments: 0

Building a hackable editor in the browser

bjenik — Sun, 14 Jul 2019 17:07:27 +0000

Article URL: http://bjenik.com/hackableeditor/

Comments URL: https://news.ycombinator.com/item?id=20434474

Points: 112

# Comments: 72

Mounting python objects as file system

bjenik — Sun, 06 Jan 2019 17:46:23 +0000

Article URL: https://bjenik.com/objectfs/

Comments URL: https://news.ycombinator.com/item?id=18839469

Points: 2

# Comments: 1

New comment by bjenik in "How AI based programming could work"

bjenik — Sun, 31 Jul 2016 19:19:26 +0000

Actually the examples in the text are quite concrete - the few lines would be everything you'd have to write to get such a site working (modulo a bit of http plumbing). Obviously we are nowhere near such a thing. The rest of the text acts as a bit of a pointer what could help to get there.

From my other comment (in this case the example was an amazon like shopping site):

> The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there".

The way of telling a neural network these things would be these few lines of code which would hook up different parts to get it working.

New comment by bjenik in "How AI based programming could work"

bjenik — Sun, 31 Jul 2016 18:44:29 +0000

The forth black box paper seems to be quite close actually. The big difference is that I would like to go for higher level tasks. The low level variant - having code missing a few lines/functions and filling those is indeed quite well researched. More interesting would be the opposite: Having blocks of code (called components in my text) and letting the net find a good way to use them. Normally the way a programmer would build something like let's say the amazon website is sketching up a few ideas how it could look - cutting it into small parts and then writing (or reusing) the code for the small parts.

A lot of the "using AI for programming" papers (AI in this case being anything from nets to logic stuff) focus on building these small parts - I'm more interested in the levels above that. The "generating code from natural language/dialogs" also kind of misses my idea as it still assumes a "programmer" to tell it what to do (in the amazon example "display the products on a grid" or "make the buy button bigger"). The programmer would in this case either guess the right thing ("everybody has their products in a grid") or run something like A/B testing against some metric ("items sold" or "$ revenue") and then go back to the conversational programming tool and tell it "change the button". Why not skip the programmer and give the metric directly to the net? The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there".

Regarding the vague ideas: I believe (also totally unsubstantiated - though research in this area would be quite interesting) that writing down ideas first and then looking into them (either by reading other papers or doing own research) is far better than the other way round, because it has a higher probability of leading to something new, because you don't follow assumptions and errors others potentially made which could lead to a dead end. Of course there is a certain trade off, because if you try to reinvent everything you will not get to the point of something new.

How AI based programming could work

bjenik — Sun, 31 Jul 2016 13:48:42 +0000

Article URL: http://bjenik.com/AIBasedProgramming/

Comments URL: https://news.ycombinator.com/item?id=12196948

Points: 89

# Comments: 59

The Future of Programming

bjenik — Fri, 29 Jul 2016 18:54:09 +0000

Article URL: http://bjenik.com/TheFutureofProgramming/

Comments URL: https://news.ycombinator.com/item?id=12189022

Points: 3

# Comments: 0

New comment by bjenik in "How Gawker Brings in Millions in Affiliate Sales"

bjenik — Sat, 02 Jan 2016 14:35:06 +0000

I think this is what the future of media will/should look like: building your reader's/consumer's trust in your publication by quality [1] content to make them come back to you when they need advice on a buying decision. The beauty of this model is that this advice can (and will [2]) be honest, because amazon and other retailers/appstores etc. do not care about what the customer actually buys as long as it is from them, which allows the publication to lead the reader to the best product while still making money. Of course there are still imperfections [3], for example sending them to a specific retailer like amazon when another would be better for them, but this is a lot better than advertising for stuff you do not need or one-sided sponsored content.

[1] "quality" is not meant as an absolute value here, but relative to the publication's target - so a "quality" article on Gawker will of course (and rightfully so) be different from one you will find in the New Yorker

[2] as long as they make the same money from two options it is in the interest of the publication to choose the honest one, because they want you to come back

[3] these imperfections are unfortunately the only thing the advertising market lives on, because if everyone could figure out what they need they would buy exactly this from the best/cheapest retailer which would kill every incentive for any kind of advertising/referral money. This will also be a problem google will face some time in the future: if their search engine gets too perfect there is no need to advertise anymore, because if the user would actually want it they would find it anyways and if not the money is wasted