But my point is, I was surprised at how hard day 5, part 2 was. I didn't give up and solved it, but went away wondering whey I'd missed something obvious and overcomplicated it. So it brings some relief to know it was 'supposed" to be a bit challenging!

But perhaps pilots would be less likely to get rated for Max aircraft if (hypothetically) there's a perceived broad issue with their safety? This could cause those who have a choice (stronger pilots) to go airbus.

Not saying it's the case, just pondering here.

- If I query for an identifier which doesn't exist - Server replies 404, this should fall into my error handling as there's a data inconsistency

- If I query say like ?category=automotive&price=1 and I get a 200 containing a json body in which there's an empty list of matches, then my client code can handle 0 matches as well as 1 or 10 with no special handling.

- 404 if resource requested by id

- 200 with empty list of results if it was a 'search' type request with params (not referring directly to an id)

The only inertia vim adds to my workflow is escaping into command mode. I have a 'jk' shortcut combo rather than escape, but if I'm hammering away I often mistime it and need to backspace out my jjkk or whatever.

To trace the problem, I added a breakpoint on ByteBuffer.allocateDirect() - this is how DM is normally allocated. But rather than suspending the process, we set it to print the stacktrace and continue.

This gave us handy output of possible causes. Focusing on the cases which were called from our own code (likely to be the root cause), we found a call to Files.readAllBytes() - which immediately sounds suspicious to cause a memory leak. Indeed this was the culprit, and explained why the problem was sporadic and hard to reproduce (there is some kind of per-thread caching behaviour of direct memory I never did understand properly).

Incidentally, Files.readAllBytes() uses direct memory because it uses an area of memory which both the OS can access (onto which to read the file contents) and which also Java can access (for the application code, rather than copying to the heap, which could be inefficient for large files).

Fortunately, this "by inspection" approach got us to the answer.

Others have stated profiling would work well. We'd have needed a local reproduction of the issue, which wasn't available at the time (could not reproduce without realistic workloads). Otherwise we could have used our SaaS APM software which runs in deployed envs, but that takes a sampling approach and doesn't really dig deep enough for this kind of stuff.

Now imagine terabytes worth of datapoints, and thousands of dimensions rather than two.

I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language.

Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case.

Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.

The filter was the weirdest, least intuitive part. I ended up porting some open source code to java. This eventually worked, but was essentially impossible to debug without some sort of software oscilloscope.

That was about 18 years ago (yikes), and I still think about DSP from time to time.

The best intuition I have of low pass filters is to imagine it like an averaging, smoothing function which operates on the sample values as a sliding window. But that's not really true - it's not a straight rolling average, instead the sinc function (sin x / x) is used to scale the sample values with respect to time.

The way FIR filters work assumes a finite amount of samples (otherwise it would be technically impossible to know all past and future samples), so instead you pick a time period over which to calculate the output. Since there are always past and future samples included, this leads to a delayed output. Eg for a 100 sample filter window, you'd need to first have 100 samples in order to calculate the first output sample. This is intuitive, since a filter is sort of a smoothing, averaging function.

https://cybersecurityventures.com/hackerpocalypse-cybercrime...

The language is unfamiliar to me, and quite intriguing. I find the pronunciation of Zuid very counter-intuitive.

I only just recently learned that national debt is sourced from bonds, purchased by individuals and companies as a safer place than cash, so take what I say with a pinch or two of salt.

You'd definitely want to check out the second hand market if getting started in woodwind.

It's really common in news about the economy. I guess the outcome is really important, but we only have these instantaneous signals. So something like superstition kicks in (when we're not carefully considering what is stated).

Sometimes novelty comes from outside the team. For example, cross-department re-platforming efforts, vendor changes, and long running migration projects. This can all lead to inconsistency, which needs to be managed so it doesn't get out of hand.

This should be factored in to a 'novelty budget' too in my opinion. A large department may have a slice of applications considered legacy-legacy, others just legacy, and some starting to move to the new stuff.

I think this is more likely to happen in companies which have been around longer - there's more accrued tech debt to deal with, there are more applications to migrate, and projects to add new functionality don't go away.

Another drawback to novelty is when keeping up to date with security vulnerabilities. If you have a common stack, then the problem is less granular.

Google have their monorepo, which (I believe) means their applications are all built against the same dependency tree, so it's simpler to keep track of upgrades.

Novelty is essential though to keep up with the innovation around us. The question is how to find the right blend of novelty (high benefit, low risk) and stability (easier to manage, maintain, and onboard engineers).