Comments URL: https://news.ycombinator.com/item?id=47366898

Points: 5

# Comments: 0

Can you share any good writing on this? It seems like many of the features within chatgpt/claude are undocumented + would benefit from better examples.

There are so many features that are really useful when being used in combination: audio mode to collect top-of-mind ideas while going for a walk, prompt deep research to compile report that goes deep on multiple topics including comparison tables, then use reasoning model to compile summary within a canvas, rewrite with a large non-reasoning model to improve the writing style (4.5 sounds way nicer than o3 imo).

Anecdotally, many of my friends want to "encourage the use of AI across their teams". Thus, there's probably even a good financial incentive and market to create actionable monetized courses on this. Social feeds also do not seem to be saturated with this kind of content yet, so probably it is fairly doable to gain distribution when starting out.

Comments URL: https://news.ycombinator.com/item?id=43745267

Points: 1

# Comments: 0

(LiteLLM is a Python library and proxy/gateway that handles cost management, virtual keys, caching, and rate-limiting for OpenAI or other LLM APIs. Langfuse manages LLM tracing, evaluation, prompt management, and experiments.)

We’ve each been building our open-source projects since early 2023 and learned that many devs and especially platform teams use the two together, so we created an integrated “OSS LLMOps stack.”

This is a fully self-hostable, technology-agnostic setup that lets you (1) Use LLMs via a standardized interface without adding complexity to the application; (2) Keep LLM Tracing, Evaluation, Prompt Management in-house for compliance; (3) Track cost and usage via a single interface, create virtual API keys for attribution of costs

It also enables direct transfer of LLM traces from the LiteLLM proxy to Langfuse. This simplifies the rollout of LLMOps practices (observability and evaluations) across multiple projects—you don't need to instrument all applications.

Additionally, the LiteLLM proxy can fetch and cache prompts from Langfuse's prompt management system, using them as templates for requests made through the proxy.

Both of these workflows can function without the integration, but are easier to manage with it!

We’d love your feedback!

Comments URL: https://news.ycombinator.com/item?id=43182241

Points: 64

# Comments: 7

It’s interesting to read how many individuals contributed in all sorts of important ways in the history of computing.

Context: When adding Clickhouse to Langfuse [1] to store all observability data, the migration DX was a big challenge. Some migrations need to copy/reorder data between tables which can take a long time and thus need be performed in the background/async. Many teams run Langfuse self-hosted and reducing downtime of an upgrade is important, thus, we added some custom logic to run these “background migrations” for now [2]. For regular migrations we use golang-migrate, works decently although DX isn’t as good as eg Prisma for Postgres.

[1] OSS LLM Observability, https://github.com/langfuse/langfuse

[2] https://langfuse.com/self-hosting/background-migrations

We are tracking this here including potential tradeoffs: https://github.com/orgs/langfuse/discussions/2509

We also offer some optional commercial add-on features that can help iterate faster or support very large teams using Langfuse. However, these features are entirely optional and we do our best to be transparent about this across our docs.

(1) Stick with a single Docker container and Postgres. This option is simple to self-host, operate, and iterate on, but it suffers from poor performance at scale, especially for analytical queries that become crucial as the project grows. Additionally, as more features emerged, we needed a queue and benefited from caching and asynchronous processing, which required splitting into a second container and adding Redis. These features would have been blocked when going for this setup.

(2) Switch to a scalable setup with a robust infrastructure that enables us to develop features that interest the majority of our community. We have chosen this path and prioritized templates and Helm charts to simplify self-hosting. Please let us know if you have any questions or feedback as we transition to v3. We aim to make this process as easy as possible.

Regarding OTel, we are considering adding a collector to Langfuse as the OTel semantics are currently developing well. The needs of the Langfuse community are evolving rapidly, and starting with our own instrumentation has allowed us to move quickly while the semantic conventions were not developed. We are tracking this here and would greatly appreciate your feedback, upvotes, or any comments you have on this thread: https://github.com/orgs/langfuse/discussions/2509

Overview of community integrations: https://langfuse.com/docs/integrations/overview

Packages that depend on Langfuse: https://langfuse.com/faq/all/packages-depending-on-langfuse

Can you create an issue with more details on the API performance problems? We monitor strict SLOs on the public API for Langfuse Cloud and are not aware of any ongoing issues, would love to learn more.

> i wonder if there are new ops solutions for the realtime apis popping up

This is something we have spent quite some time on already, both on designs internally and talking to teams using Langfuse with realtime applications. IMO the usage patterns are still developing and the data capturing/visualization needs across teams is not aligned. What matters: (1) capture streams, (2) for non-text provide timestamped transcript/labels, (3) capture the difference between user-time and api-level-time (e.g. when catching up on a stream after having categorized the input first).

We are excited to build support for this, if you or others have ideas or a wishlist, please add them to this thread: https://github.com/orgs/langfuse/discussions/4757

> retries for instructor like structured outputs mess up the traces, i wonder if they can be tracked and collapsible

Great feedback. Being able to retroactively downrank llm calls to be `debug` level in order to collapse/hide them by default would be interesting. Added thread for this here: https://github.com/orgs/langfuse/discussions/4758

> chatgpt canvas like "drafting" workflows are on the rise (https://www.latent.space/p/inference-fast-and-slow) and again its noisy to see in a chat flow

Can you share an example trace for this or open a thread on github? Would love to understand this in more detail as I have seen different trace-representations of it -- the best yet was a _git diff_ on a wrapper span for every iteration.

> how often do people actually use the feedback tagging and then subsequently finetuning? i always feel guilty that i dont do it yet and wonder when and where i should.

Have not seen finetuning based on user-feedback a lot as the feedback can be noisy and low in frequency (unless there is a very clear feedback loop built into the product). More common workflow that I have seen: identify new problems via user feedback -> review them manually -> create llm-as-a-judge or other automated evals for this problem -> select "good" examples for fine-tuning based on a mix of different evals that currently run on production data -> sanitize the dataset (e.g. remove PII).

Finetuning has been more popular for structured output, sql generation (clear feedback loop / retries at run-time if the output does not work). More teams fine-tune on all the output that has passed this initial run-time gate for model distillation without further quality controls on the training dataset. They usually then run evals on a test dataset in order to verify whether the fine-tuned hits their quality bar.

Being unopinionated and API-first has been a core design decision. We want to build the building blocks that everyone needs while acknowledging that most Langfuse users are very sophisticated teams that have a clear idea of what they want to achieve. Over time we will build more abstractions for common workflows to make it easier to get started but new features will always start API-first.

More on this here: https://langfuse.com/why