[1]: https://www.boggsfiles.com/x-files-scripts-by-season/season-... [2]: https://drive.google.com/file/d/1BQbfRmeq3FsajjYcEuqDHv0kf2B...

IMO, with the current tech, it entirely depends on what data you're talking about.

For metrics and traces, I would use the OTel collector personally. You will have much more flexibility and it's pretty easy to write custom processors in Go. Support for traces is quite mature and metrics isn't far off. We've been running collectors for production scale of metric and trace ingest for the past couple of years, on the order of 1m events/sec (metric datapoints or spans). You mentioned low volume so that's less important, but I just wanted to mention in case others find this comment.

Logs are a bit different. We looked in to this in the past year. Vector has emerging support for OTLP but it's pretty early. Still, I bet it's pretty straightforward if your backend can ingest via OTLP. Our main concern with running the otel-collector as the log ingest agent was around throughput/performance. Vector is battle-tested, otel is still a bit early in this space. I imagine over time the gap will be closed but I would probably still reach for Vector for this use-case for higher scale. That said, YMMV and as with any technical decision, empirical data and benchmarking on your workloads will be the best way to determine the tradeoffs.

For your scale you could probably get away with an OTel collector daemonset and maybe a deployment with the Target Allocator (to allocate Prometheus scrapes) and call it a day :)

Comments URL: https://news.ycombinator.com/item?id=43515877

Points: 66

# Comments: 18

Eventually I think there was some work to move to GraphQL, which can solve some of the same problems. But GraphQL is a technology, and BFF is more of a pattern. There is a later reflection on that blog that makes this distinction, which I only read today.[1] It makes another observation that I kind of forgot about, because it was hiding right in front of my face as a worker there:

"The defining characteristic of a BFF is that the API used by a client application is part of said application, owned by the same team that owns it, and it is not meant to be used by any other applications or clients."

The ownership model is indeed a big deal. In practice, it helped in many ways to have a sort of intermediate layer between the client applications and the rest of the architecture. For example, in the SoundCloud web application, when you load a page of playlists, only the first 5 tracks in the playlist are visible to the end-user. So the web BFF application had special logic to only partially load all the track metadata past track 5, which had significant impact on scalability and latency, especially when rendering a lot of playlists that had lots of tracks!

[0]: https://philcalcado.com/2015/09/18/the_back_end_for_front_en... [1]: https://philcalcado.com/2019/07/12/some_thoughts_graphql_bff...

[0]: https://github.com/opentracing/specification/issues/142

It is built on a plugin architecture, so plugins are certainly a viable option and this is documented in more detail here[0]. In general I have found the Keycloak docs thorough and well-written. When I operated Keycloak I built a few plugins to solve specific needs/assumptions we had around IdP when migrating to Keycloak from a bespoke solution.

Re: your second point, the docs also describe this in detail[1]. Having the realm data exist in a simple form that can be exported/imported was very useful. However, I would have liked if they thought more about how to do live backup/restore; perhaps that is easier now than it was at the time.

[0]: https://www.keycloak.org/docs/latest/server_development/inde... [1]: https://www.keycloak.org/server/importExport#_importing_a_re...

I find it interesting how this ties in to the "productivity paradox."[0] The idea the author seems to be getting at--that software accelerates the creation of ever more elaborate solutions (often to problems created by prior iterations of software), and in the process leaves a wake of complexity that frustrates and baffles the society it was supposed to serve--is something I'd like to read more about.

[0]: https://en.wikipedia.org/wiki/Productivity_paradox

Comments URL: https://news.ycombinator.com/item?id=36171478

Points: 2

# Comments: 0

"What happened? Well, I think part of it is a hypertrophy of this drive to validate work as a thing in itself. It used to be that Americans mostly subscribed to a rough-and-ready version of the labor theory of value. Everything we see around us that we consider beautiful, useful, or important was made that way by people who sank their physical and mental efforts into creating and maintaining it. Work is valuable insofar as it creates these things that people like and need. Since the beginning of the 20th century, there has been an enormous effort on the part of the people running this country to turn that around: to convince everyone that value really comes from the minds and visions of entrepreneurs, and that ordinary working people are just mindless robots who bring those visions to reality.

But at the same time, they’ve had to validate work on some level, so they’ve simultaneously been telling us: work is a value in itself. It creates discipline, maturity, or some such, and anyone who doesn’t work most of the time at something they don’t enjoy is a bad person, lazy, dangerous, parasitical. So work is valuable whether or not it produces anything of value."

[1]: https://theanarchistlibrary.org/library/david-graeber-and-th...

Comments URL: https://news.ycombinator.com/item?id=34596411

Points: 12

# Comments: 0

I looked at the "who's hiring?" post and there were not many positions posted in science/research. I wanted to share this relatively new resource, which includes a job board advertising open positions in research software. You do not need a PhD for most/any of these positions. I worked as a research software engineer for several years and it was very rewarding, and a very different kind of work than I was doing in industry. I don't think this opportunity is well-known still, and suspect others might be interested in making the move from industry to research, even if just to try it out.

Comments URL: https://news.ycombinator.com/item?id=33095666

Points: 3

# Comments: 1

Disclosure: I left my position on this research team this summer after a pleasant 4 years. It's quite a different environment than one normally might find on these listings.

The Nimbus team[1] is funded by multiple grants from the National Science Foundation and concentrates on building accessible, powerful cloud infrastructure for the comp sci research community. Our users are folks who are doing work that cannot be done on commercial clouds for a variety of reasons (lack of low-level access, performance variability too high, cost $$, etc.) We work with stuff like OpenStack, Kubernetes, Liqid, and are always exploring new things, like how to build a distributed edge/IoT testbed, and how to enable practical reproducibility of CS experiments.

I think more development talent should consider working in science/research. There are lots of opportunities and I don't think they are often advertised well.

The job is in Chicago but we work largely remote. Still, in-person has been preferred especially as sometimes it's helpful to work with collaborators at the Argonne National Laboratory west of Chicago. Happy to answer more in pm.

[1]: https://www.nimbusproject.org/jobs

I spent time writing this utility after being surprised that there didn't seem to be many resources in this space. I wanted to write a weekly report that would send mail to my boss (and boss' boss) about the state of some key metrics. The main design goals were to (1) support a simple on-premises deployment use-case and (2) leverage the existing Python ecosystem around visualizations. I hope others find it useful and am interested in feedback/suggestions. It is open source but is released under the Prosperity license, so if it's used commercially you should purchase an annual license after 30 days of trial.

Comments URL: https://news.ycombinator.com/item?id=28513371

Points: 2

# Comments: 1

The "shrinking" capability of the test library highlighted is brilliant.

I'm inspired to think of how to start to leverage something like this on some upcoming work.