Another is that when working with manifolds, you usually don't get a set of global coordinates. Manifolds are defined by various local coordinate charts. A smooth manifold just means that you can change coordinates in a smooth (differentiable) way, but that doesn't mean two people on opposite sides of the manifold will agree on their coordinate system. On a sphere or circle, you can get an "almost global" coordinate system by removing the line or point where the coordinates would be ambiguous.

I'm not very well versed in the history, but the study of cartography certainly predates the modern idea of an abstract manifold. In fact, the modern view was born in an effort to unify a lot of classical ideas from the study of calculus on spheres etc.

I agree it's especially frustrating when they don't even get it right. That crosses the line for me, and I will admonish them to let me finish what I am saying.

It's just a trick that helps me pay attention in lectures, which a lot of people struggle with. Certainly you have to put the work outside of the classroom as well.

As you follow along with the speaker, try to predict what they will say next. These can be either local or global predictions. Guess what they will write next, or what will be on the next slide. With some practice (and exposure to the subject area) you can usually get it right. Also try to keep track of how things fit into the big picture. For example in a math class, there may be a big theorem that they're working towards using lots of smaller lemmas. How will it all come together?

When you get it right, it will feel like you are figuring out the material on your own, rather than having it explained to you. This is the most important part.

If you can manage to stay one step ahead of the lecturer, it will keep you way more engaged than trying to write everything down. Writing puts you one step behind what the speaker is saying. Because of this, I usually don't take any notes at all. It obviously works better when lecture notes are made available, but you can always look at the textbook.

People often assume that I have read the material or otherwise prepared for lectures, seminars, etc., because of how closely I follow what the speaker is saying. But really most talks are quite logical, and if you stay engaged it's easy to follow along. The key is to not zone out or break your concentration, and I find this method helps me immensely.

The reality is that there are some great, mature solvers out there that work well enough for most cases. And while it might be possible to eke out more performance in specific problems, it would be very hard to beat existing solvers in general.

Theoretical developments like this, while interesting on their own, don't really contribute much to day-to-day users of linear programming. A lot of smart people have worked very hard to "optimize the optimizers" from a practical standpoint.

[0] https://arxiv.org/abs/1412.6980

However, from what I have seen, this isn't really a useful way of reframing the problem. The optimal control problem is at least as hard, if not harder, than the original problem of training the neural network, and the latter has mature and performant software for doing it efficiently. That's not to say there isn't good software for optimal control, but it's a more general problem and therefore off-the-shelf solvers can't leverage the network structure very well.

Some researchers have made interesting theoretical connections like in neural ODEs, but even there the practicality is limited.

> Some functions have axes arguments. Some have different versions with different names. Some have Conventions. Some have Conventions and axes arguments. And some don’t provide any vectorized version at all.

> But the biggest flaw of NumPy is this: Say you create a function that solves some problem with arrays of some given shape. Now, how do you apply it to particular dimensions of some larger arrays? The answer is: You re-write your function from scratch in a much more complex way. The basic principle of programming is abstraction—solving simple problems and then using the solutions as building blocks for more complex problems. NumPy doesn’t let you do that.

Usually when I write Matlab code, the vectorized version just works, and if there are any changes needed, they're pretty minor and intuitive. With numpy I feel like I have to look up the documentation for every single function, transposing and reshaping the array into whatever shape that particular function expects. It's not very consistent.

More generally speaking, you're asking when the iteration `x_+ = Ax` converges to a fixed point which is an eigenvector of A. This can happen a few different ways. The obvious way is that A has an eigenvector `v` with eigenvalue 1, and all other eigenvalues with magnitude < 1. Then those other components will die out with repeated application of A, leaving only `v` in the limit.

For Markov chains, we can get this exact property from the Perron-Frobenius theorem, which applies to non-negative irreducible matrices. Irreducible means that the transition graph of the Markov chain is strongly connected. If that's the case, then there is a unique eigenvector called the stationary distribution (with eigenvalue 1), and all initial conditions will converge to it.

In case A is not irreducible, you may have different connected components, and the stationary distribution may depend on which component your initial condition is in. Going back to the n x n identity matrix, it has n connected components (it's a completely disconnected graph with all the self-transition probabilities = 1). So every initial condition is stationary, because you can't change anything after the initial step.

The works that I've read train an NN on numerical solutions for different geometries and boundary conditions. Then they try to infer the solutions for configurations outside the training set, which should be much faster than recomputing the numerical solution.

It could be interesting to do a comparison with finite volume methods to see when/how those approximations break down.

The statement that the Kalman filter is mean-square optimal because it generates a correct estimate in expectation is false. In fact, any gain L will generate an estimate whose expected value is x_k, as long as w_k and v_k are zero mean. The Kalman gain is a specific choice of L that is optimal in the mean square sense only when the disturbances are Gaussian. The Kalman gain is also time-varying and depends on the evolution of the estimate covariance, although it will converge to a steady-state value.

What's being described here is more properly called a Luenberger observer, but I guess that name doesn't get the same recognition outside the control community.

I'm also wondering why they chose to include H past estimates and measurements in the transformer. They're already embedding the Kalman gain into the weights of the transformer, so taking just one past estimate/measurement should exactly recover the Kalman filter. Going further into the past just makes the estimate worse, because of the softmax.

But I'm happy to say that after the first two weeks or so, it's been rock solid. The issues I had with wifi were patched pretty quickly, and everything else pretty much just works with the default configuration. The only thing is the battery life is still bad, partly because I have way too much RAM...

I've never seen an MILP used this way, to characterize the entire feasible set (or "solution pool"). Is this one of the fastest ways to do so? The usual branch-and-bound type methods won't apply, since the solver has to enumerate every feasible solution.

The CPLEX docs (https://www.ibm.com/docs/en/icos/22.1.1?topic=solutions-how-...) mention the potential slowness and also the numerical issues the author faces in the article.