Hacker News: simonbyrne

New comment by simonbyrne in "State of Machine Learning in Julia"

simonbyrne — Wed, 12 Jan 2022 16:57:53 +0000

If that's seriously what you're looking for, you should really consider Fortran. There is a reason it is still very widely used in scientific domains.

New comment by simonbyrne in "Floating point visually explained (2017)"

simonbyrne — Sun, 28 Nov 2021 20:05:28 +0000

You can just wrap the literal in a conversion function, eg Float32(0x1p52), which should get constant propagated at compile time.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Sat, 13 Nov 2021 21:22:18 +0000

That's fascinating thread, thanks: https://github.com/WebAssembly/design/issues/148

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 22:10:09 +0000

Do you know what happens when you have ops with different flags? e.g. if you have (a + b) + c, where one + allows reassoc but one doesn't?

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 20:33:02 +0000

From personal experience, yes: I've seen multiple cases of scientists finding the ultimate cause of their bugs was some fast-math-related optimization.

The problem isn't necessarily the code they wrote themselves: it is often that they've compiled someone else's code or an open source library with fast-math, which broke some internal piece.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 20:25:49 +0000

I tried to lay out a reasonable path: incrementally test accuracy and performance, and only enable the necessary optimizations to get the desired performance. Good tests will catch the obvious catastrophic cases, but some will inevitably be weird edge cases.

As always, the devil is in the details: you typically can't check exact equality, as e.g. reassociating arithmetic can give slightly different (but not necessarily worse) results. So the challenge is coming up with appropriate measure of determining whether something is wrong.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 20:21:04 +0000

I'm not exactly sure what you're asking here, but the point is that "to machine precision" is relative: if f(x) and g(x) are O(1e200), then the absolute error of each is still O(1e185). However f(x)/g(x) will still be very accurate (with absolute error O(1e-15)).

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 19:22:54 +0000

The key thing about floating point is that it maintains relative accuracy: in your case, if you have say f(x) and g(x) are both O(1e200), and are correct to some small relative tolerance, say 1e-10 (that is, the absolute error is 1e190). Then the relative for f(x)/g(x) stays nicely bounded to about 2e-10.

However if you do f(x) - g(x), the absolute error is on the order of 2e190: if f(x) - g(x) is small, then now the relative error can be huge (this is known as catastrophic cancellation).

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 19:01:58 +0000

Herbie is a great tool, especially for teaching.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 18:14:41 +0000

In theory, every function should do that to check things like rounding mode etc. But that would be pretty slow, especially for low-latency operations (modifying mxcsr will disrupt pipelining for example).

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 18:01:37 +0000

-ffp-contract=fast will enable FMA contraction, i.e. replacing a * b + c with fma(a,b,c). This is generally okay, but there are a few cases where it can cause problems: the canonical example is computing an expression of the form:

a * d - b * c

If a == b and c == d (and all are finite), then this should give 0 (which is true for strict IEEE 754 math), but if you replace it with an fma then you can get either a positive or negative value, depending on the order in which it was contracted. Issues like this pop up in complex multiplication, or applying the quadratic formula.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 17:53:10 +0000

My point isn't that fast-math isn't useful: it very much is. The problem is that it is a whole grab bag of things that can do very dangerous things. Rather than using a sledgehammer, you should try to be selective and enable only the useful optimizations, e.g. you could just enable -ffp-contract=fast and -fno-math-errno.

New comment by simonbyrne in "Beware of fast-math"

simonbyrne — Fri, 12 Nov 2021 17:47:14 +0000

Not necessarily: if your cospi(x) function is always returning 1.0 (https://github.com/JuliaLang/julia/issues/30073#issuecomment...), but you wrote your code assuming the result was in a different interval, then you could quite easily invoke undefined behavior.

Beware of fast-math

simonbyrne — Fri, 12 Nov 2021 16:53:06 +0000

Article URL: https://simonbyrne.github.io/notes/fastmath/

Comments URL: https://news.ycombinator.com/item?id=29201473

Points: 189

# Comments: 107

New comment by simonbyrne in "U.S. Officials in Germany Hit by Havana Syndrome"

simonbyrne — Wed, 18 Aug 2021 15:17:41 +0000

> But then people latched on to the idea that it was caused by some kind of sonic device or hitherto unknown weapon targeting embassies.

Possibly prompted by historical precedent: https://en.wikipedia.org/wiki/Moscow_Signal

New comment by simonbyrne in "The Floppy Disk of Floating Point"

simonbyrne — Tue, 14 Jul 2020 18:13:51 +0000

One of the interesting things about the 80-bit format was that allowed the use of 64-bit integers on a 16 or 32-bit machine. This was what Thomas Nicely was using them for when he found the Pentium FDIV bug: https://faculty.lynchburg.edu/~nicely/pentbug/pentbug.html

New comment by simonbyrne in "JuliaLang: The Ingredients for a Composable Programming Language"

simonbyrne — Mon, 10 Feb 2020 20:21:19 +0000

> or wait for optimizations to be done, e.g have pandas, numpy etc handle multi-core processors etc ?

All those exist already. Indeed, other than DataFrames.jl (the pandas equivalent) they are part of the language itself.

New comment by simonbyrne in "The Floating-Point Guide (2010)"

simonbyrne — Thu, 21 Nov 2019 17:52:02 +0000

I disagree about practical and accessible:

- its exposition is complicated (trying to prove everything in a general base makes it difficult to understand)

- it's woefully out of date (lack of guard digits haven't been an issue for at least 25 years, extended precision hasn't been an issue for the past 10 or so, and most languages now default to having fairly strict floating point semantics)

- it gets bogged down in irrelevant minutiae (rounding modes and exception flags, while available in modern hardware, aren't really supported by any modern languages/compilers)

- it doesn't really provide any practical advice (it barely mentions binary-decimal conversion, it jumps to doubling precision and Kahan summation without suggesting any intermediate steps such as sorted or pairwise summation).

But my biggest complaint is the frequency with which users are referred to it on StackOverflow as if (1) it is a good way to learn about floating point concepts, and (2) anyone using floating point numbers should be expected to understand it all.

New comment by simonbyrne in "My years working on black programs"

simonbyrne — Wed, 20 Nov 2019 05:58:47 +0000

Deep learning is the obvious answer: 10 years ago it was largely confined to a small number of research groups and niche conferences.

New comment by simonbyrne in "The Floating-Point Guide (2010)"

simonbyrne — Tue, 19 Nov 2019 17:46:35 +0000

Thank you! The Goldberg article is a terrible way to learn about floating point, and the frequency with which it is referred to on StackOverflow is really disheartening.