Yoshida Kenko, Essays in Idleness

I'm in the early stages of evaluating pgvector myself. but having used pinecone I currently am liking pgvector better because of it being open source. The indexing algorithm is clear, one can understand and modify the parameters. Furthermore the database is postgresql, not a proprietary document store. When the other data in the problem is stored relationally, it is very convenient to have the vectors stored like this as well. And postgresql has good observability and metrics. I think when it comes to flexibility for specialized applications, pgvector seems like the clear winner. But I can definitely see pinecone's appeal if vector search is not a core component of the problem/business, as it is very easy to use and scales very easily

I agree with you for the NLP domain, but I wonder if there will also be a bitter lesson learned about the perceived generality of language for universal applications.

retrieve != query

this project is extremely simplistic in regards to its vector search tech. pgvector is an open source implementation of an _index_ (multiple algos actually), this uses Cloudflare's completely proprietary index with a single call.

> not to make huge (also architectural) mistakes

I'm noticing it's the first time I'm even having to make significant architectural decisions, which is difficult because I don't have much experience to draw from, so even the smallest decisions often require a lot of research.

It has been 3 years now and I feel that I have become decent at coding, yet my education in math and computer science is sorely lacking. For example, I am working on a semantic search project and I am reading about cross-encoders, and while I understand the problem statement and the solution and could operate the model based on the documentation, I have no idea how it works. Once I understand how to apply the tool, the programs become easy to write and I can resolve errors by directly referencing documentation, either of the language or the library. The last time I wrote python was major version 2 as a 16 year old, but I'm learning python 3 10 years later from the reference manual. This is what I mean by 'decent at coding' but lacking in math/cs background. Put frankly:

- I have gotten to the point where the foundations of the techniques to solve the problems I am working on are beyond my educational level, but I am a decent enough engineer to identify where the technique could be applied and how to implement it.

- I want to learn the math and cs to work on scientific computing. I want to investigate dynamical systems with computers. I think my potential lies not in specializing in a specific natural science (such as geophysics), but rather becoming a skilled implementer.

Is the goal of transitioning to working on scientific or research software as an implementer for someone with little formal education a reasonable one? If it's not, this will still be my hobby. How can I remediate/fill in the gaps of my math education?

- Book list for math: currently reading "A first course in mathematical analysis" by Burkhill for calculus (to refresh and extend high school), and then after that: my college linear algebra book (done wrong), Spivak's "Calculus on Manifolds", and "Differential Equations with Applications and Historical Notes".

- CS/Software Engineering: I gravitate towards functional programming and mathematical logic, so resources like the 'Software Foundations'[2] series are appealing. However my interest in numerical methods and scientific computing tells me that reading a book such as 'Computer Systems: A Programmer's Perspective' and then moving on to an applied numerical analysis book and OpenGL book would be a greater 'return on value' in the short term. There is also the need to continue my professional development as an engineer, and for this I am looking at books like 'Designing Data-Intensive Applications' and 'Software Design for Flexibility'.

How do I balance my interest in foundations with my interest in scientific/numerical computing? How do I balance my math/cs study with my professional development regarding engineering?

TL;DR;

I feel like I have become an 'intermediate' autodidact in the sense that I feel that I have overcome the dunning-kruger effect, I feel like an idiot[3] yet am a working programmer being paid an industry standard wage to work on designing software systems based on mathematics and computer science I do not fully understand, and I am looking for advice on how to actually start mastering the craft with an eye towards both foundations and scientific computing/numerical analysis implementation. I want to continue working part-time and study part-time on my own rather than go to school.

[1] https://www.gutenberg.org/ebooks/18267

[2] https://softwarefoundations.cis.upenn.edu/

[3] https://grugbrain.dev/

Comments URL: https://news.ycombinator.com/item?id=38926869

Points: 2

# Comments: 1

i'd wager nearly every development environment is integrated to some extent, piping output in the unix shell is integrating.

i find SRS very useful. but i personally think it is not well suited to reference information. i'm skeptical of putting all my appointments in SRS for example (maybe birthdays would be worth it). or say I had collected a bunch of papers related to a topic I was very interested in. I don't necessarily think I'd want to memorize the list of them via SRS, but having the titles written somewhere for reference would be great. I think the point of this paper is that if that 'somewhere' is an SRS card, it is almost completely devoid of context (other than 'studying flashcards on computer' context), but if that somewhere is a notebook that contains lists of papers related to all the topics i'm interested in, it's much easier to find. (though computers are good at searching fast)

wikipedia claims there is not much recent research when it references that link.

it is being shown that meditation can affect brain waves, I wonder if it is in a way that is beneficial for preventing seizures.