I'll give a shoutout as well to Glimpse: https://github.com/LegendarySpy/Glimpse

The latest open source local STT models people are running on devices are significantly more robust (e.g. whisper models, parakeet models, etc.). So background noise, mumbling, and/or just not having a perfect audio environment doesn't trip up the SoTA models as much (all of them still do get tripped up).

I work in voice AI and am using these models (both proprietary and local open source) every day. Night and day different for me.

Comments URL: https://news.ycombinator.com/item?id=47478985

Points: 2

# Comments: 0

They could've released something like that years ago (the discovery service it's built on has existed for over a decade) but creating a simple, accessible, unified CLI for general integration apparently wasn't worth it until agents became the hot thing.

I wonder when / if there will be a rug pull on all of this. Because I really don't see what the long-term incentives are for incumbent tech platforms to make it easy for automated systems to essentially pull users away from the actual platform. I guess they're focused on the short term incentives. And once they decide the party's over, promising upstarts and competition can get absorbed and it'll be business as usual. Idk, we'll see.

> Chunks the audio when the model believes based on the words said by the user that they have completed their utterance.

Source: https://developers.openai.com/api/docs/guides/realtime-vad

OpenAI's Semantic mode is looking at the semantic meaning of the transcribed text to make an educated guess about where the user's end of utterance is.

According to Deepgram, Flux's end-of-turn detection is not just a semantic VAD (which inherently is a separate model from the STT model that's doing the transcribing). Deepgram describes Flux as:

> the same model that produces transcripts is also responsible for modeling conversational flow and turn detection.

[...]

> With complete semantic, acoustic, and full-turn context in a fused model, Flux is able to very accurately detect turn ends and avoid the premature interruptions common with traditional approaches.

Source: https://deepgram.com/learn/introducing-flux-conversational-s...

So according to them, end-of-turn detection isn't just based on semantic content of the transcript (which makes sense given the latency), but rather the the characteristics of the actual audio waveform itself as well.

Which Pipecat (open source voice AI orchestration platform) actually does as well seemingly with their smart-turn native turn detection model as well: https://github.com/pipecat-ai/smart-turn (minus the built-in transcription)

The cascading model (STT -> LLM -> TTS), is unlikely to go away anytime soon for a whole lot of reasons. A big one is observability. The people paying for voice agents are enterprises. Enterprises care about reliability and liability. The cascading model approach is much more amenable to specialization (rather than raw flexibility / generality) and auditability.

Organizations in regulated industries (e.g. healthcare, finance, education) need to be able to see what a voice agent "heard" before it tries to "act" on transcribed text, and same goes for seeing what LLM output text is going to be "said" before it's actually synthesized and played back.

Speech-to-Speech (end-to-end) models definitely have a place for more "narrative" use cases (think interviewing, conducting surveys / polls, etc.).

But from my experience from working with clients, they are clamoring for systems and orchestration that actually use some good ol' fashioned engineering and that don't solely rely on the latest-and-greatest SoTA ML models.

For anyone curious: https://flux.deepgram.com/

I think it's so easy for people to discount "mental energy" since culturally we don't often acknowledge it as a finite resource the same way we do physical energy. Well maybe the problem is we view them as separate things in the first place.

When I was younger I just didn't have to worry about so much stuff.

They have their Granite family of models, but they're small language models so surely significantly less resources are going into them.

I don't think it's going to be anything like 50% or even 30% of users using non-flagship hardware or software products. But it could still be significant. And I think the more important thing isn't going to be market share as much as proof of viability. More successful examples will beget more.

It's about planting seeds from which future digital ecosystems can grow -- that have interoperability, functionality, and openness built into their foundations.

I believe that what drove you to make this post and the way I feel is not unique and are part of a larger swell in similar sentiments.

You throw in other factors too like the mass tech layoffs and the continued doubling down of tech barons on their cravings to intermingle with the surveillance state and military industrial complex... I just can't see how the future doesn't have more people disillusioned with the current state of the tech industry.

I think big tech will continue to overplay their hand and the mess that comes after will be an opportunity to give people what they want and show alternatives to what's already been done that we know won't work out.