The alternative explanation is that the inference engine, which batches several unrelated requests for parallel processing, messed up the unpacking and returned an unrelated user’s query. This one would be very scary as it will leak arbitrary content, but it seems much less likely here.

- There is no double taxation if you just pay yourself a salary (since it’s a normal business expense). If you want to take money out of the company flexibly, a GmbH is the wrong structure.

- I’ve never heard of anybody doing an UG/GmbH + KG to get started. This is highly unusual. Most people either do just a simple UG or maybe they set up a holding structure with two separate GmbH / UG entities.

- Related to the above: if you go with a simple, standard structure you will incur minimal legal fees. You don’t need a lawyer, you just directly task a notary and tell them you want a standard setup.

- If you don’t want the complexity of a limited liability company, the standard way to reduce liability risk is to get liability insurance. Many, many people do this instead of having a GmbH.

The valid criticism is the a) lack of digital processes and b) sequential processing of steps that could happen in parallel. For example, I sped up my own GmbH process by driving to the register court and paying in cash on-site. For whatever reason that’s much faster and saves about a week.

I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.

So I really like the idea. But being an OSS maintainer these days seems... intense.

And hardware is very fun! It’s also very frustrating. But to me worth it.

For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate RGB vs RGB-D) and the VLA + fine-tuning one.

Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

And yes, having a nice robot makes life so much easier. Very happy with my choice so far!

I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)

That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.

On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.

The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.

And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.

I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.

Have you seen the recent nvidia thing? They do this at scale for robotics manipulation: https://research.nvidia.com/labs/gear/enpire/

Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.

On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.

At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.

Re your questions:

- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.

- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.

- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that