Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.

For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.

Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...

Boosted trees over embeddings work surprisingly well too.

When a new model is released they have been able to have a working implementation days later, not months when the model was redundant. And often it's much more usable than the one-off, unmaintained, research code written in a very fragile environment.

BLOOM was a decent fully open and very large model for its time, and a huge coordination effort. It was around the same time as another "joke", OPT, but that gave Meta the knowledge that let them build Llama. Huggingface didn't have the money to compete in that race, but I'd bet it helped them stay at the frontier of AI software.

Google could have owned this space; they had TensorFlow hub, but it was too hard to use and too locked down. Huggingface made it easy to run and train transformers, and got the timing just right.

I'm glad someone is trying to build truly open models (along with the great work at Allen AI), because the likes of Meta and Alibaba are only going to release weights as long as it suits their business.

I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.

LLMs, along with image and video generation models, are generators of very dynamic, engaging and personalised content. If Open AI or anyone else wins a monopoly there it could be terrible for Meta's business. Commoditizing it with Llama, and at the same time building internal capability and a community for their LLMs, was solid strategy from Meta.

Each time you embed a document you search for approximate nearest neighbours before adding it, so it is O(N) like MinHash. Vector indexes like HNSW and PQ have better performance/quality tradeoffs than SimHash LSH which is the analogue of MinHash for cosine distance.

The quality depends on what you mean by near duplicate and the embedding model you use. Current models work well, and if you have labelled data you can fine tune them to be better.

The main drawback is the additional cost of embedding all the documents, especially for longer documents. But this cost has dropped really quickly with smaller models, better optimisations, and faster hardware.

What might be missing is in-context learning, prompt engineering, novel forms of attention, RLHF, and LoRA (though it covers adaptors), but this is still changing rapidly and the details may be irrelevant in another year. If you have a look at a recent course like Stanford CS224N 2023 there's a lot of overlap.

In general it won't solve all your problems, but it's helpful for automating simple things like this (but you still need to test edge cases). With this prompt in Python (which I'm more familiar with):

  from dataclasses import dataclass
  from typing import TypeVar
  
  T = TypeVar('T')
  
  @dataclass
  class MaxDict:
      items: dict[T, float]
      max_value_item: T

      def add_item(

    def add_item(self, item: T, value: float):
        if value > self.items[self.max_value_item]:
            self.max_value_item = item
        self.items[item] = value

  // Print the fibonnaci sequence, except the number 5.
  int main() {

    int a = 0;
    int b = 1;
    int c = 0;
    while (c < 100) {
        c = a + b;
        a = b;
        b = c;
        if (c != 5) {
            cout << c << endl;
        }
    }
    return 0;

In both cases I think this is a real opportunity; we can let students get more quickly to bigger problems and systems thinking by leveraging these tools. It requires professors to start thinking innovatively about how to teach and assess these subjects.

String theory tries to do the same thing with other forces; so it's not clear whether there's such a transformation. Forces are technically simpler than embedding extra dimensions that there are no evidence for.

I think it would be interesting to see how quickly you could get from there to a useful system, say one where you could download the USB driver from github and compile it. But I definitely don't know enough to do that.

[1] https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/

For tabular data (which is probably most relevant in Pharma, and probably the best place to start) Introduction to Statistical Learning by Hastie et al and Max Kuhn's Applied Predictive modelling cover a lot of the classical techniques.

For univariate time series forecasting "Forecasting Principles and Practice" is great.

For natural language processing foundations Jurafsky's Speech and Language Processing is broadly recommended; for cutting edge natural language processing Stanford's CS224n is great: http://web.stanford.edu/class/cs224n/

But it's ok to start using libraries and fitting models without understanding how they work deeply, and coming back to these books later (just make sure you come back; there's lots of useful ideas in them!) In which case I'd recommend some of the resources the parent doesn't recommend

If you want to go an applied route I'd suggest starting somewhere like Kaggle and looking through the competitions for ones vaguely similar to yours. They've done all the hard work of choosing a challenging but solvable problem, sourcing and splitting the data, and choosing a metric. You then can see what techniques actually work really well, and benchmark different approaches. Academic challenges like Imagenet or Coco are also good for this, but you'll have to work harder to find relevant resources.

Once you've done this a couple of times, you can start framing your own problems, collecting and annotating your own datasets, deploying and maintaining models.

They claim they're making it more accessible with pedagogical examples (and innovations in representing differential vector calculus).

I still prefer index notation. Their notation looks pretty intricate and I think it would be really hard to do long calculations in it. It's also not clear to me how you'd generalise it to non-commutative vector algebras like in quantum mechanics.

[1] https://arxiv.org/pdf/1911.00892.pdf

[2] http://birdtracks.eu/