I assume the responses from that point onwards didn't take the video input into account, and the model just assumes the user has completed the first step based on the conversation history. I don't know how these 'live' ai sessions things work but based on the existing openai/gemini live ai chat products it seems to me most of the time the model will immediately comment on the video when the 'live' chat starts but for the rest of the conversation it works using TTS+STT unless the user asks the AI to consider the visual input.

I guess if you have enough experience with these live AI sessions you can probably see why it's going wrong and steer it back in the right direction with more explicit instructions but that wouldn't look very slick in a developer keynote. I think in reality this feature could still be pretty useful as long as you aren't expecting it to be as smooth as talking to a real person

I guess I could also follow suit and return your weird toxic/patronising insult here too since you clearly didn't understand my original comment, but perhaps it would be nicer if we didn't do that?

Clearly there's a disadvantage to using text in that situation, and I think it's that it almost always takes longer to express thoughts/intents using text. ISTM a sufficiently advanced computer voice interface would have the same advantage.