https://keras.io/examples/keras_recipes/trainer_pattern/

> - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible.

The Tensorflow backend has the excellent tf.data.Dataset API, which allows for out of core data and processing in a streaming way.

Why not do the data augmentation during preprocessing (so that the transformations don't have to be done by differentiable transforms)? I.e., map over a tf.Dataset with the transformation (and append to the original dataset).

    $ nim c -o:base64_test_nim -d:danger --cc:gcc --verbosity:0 base64_test.nim

    $ nim c -o:json_test_nim -d:danger --cc:gcc --verbosity:0 json_test.nim

Edit: It appears I'm incorrect, -d:danger does imply -d:release in newer Nim versions.

For example, the JSON test was compiled with:

    $ nim c -o:json_test_nim -d:danger --cc:gcc --verbosity:0 json_test.nim

Also, as others have mentioned, some of the most important skills for DS are data munging, data "presentation", and soft skills like managing expectations / relationships / etc.

I would not recommend this book if you want to get into DS with the idea that, "I'll read this and then I'll know everything I need to." It's too dense and academically-focused, and it would probably be discouraging if you try to read this all without getting your feet wet.

I've been following (and evangelizing) Nim for a while, this will make it easier to do so.

A̶F̶A̶I̶K̶,̶ ̶a̶ ̶R̶O̶C̶ ̶c̶u̶r̶v̶e̶ ̶c̶a̶n̶ ̶b̶e̶ ̶m̶i̶s̶l̶e̶a̶d̶i̶n̶g̶ ̶f̶o̶r̶ ̶a̶n̶ ̶i̶m̶b̶a̶l̶a̶n̶c̶e̶d̶ ̶d̶a̶t̶a̶s̶e̶t̶,̶ ̶b̶u̶t̶ ̶t̶h̶e̶ ̶A̶U̶C̶ ̶i̶s̶ ̶s̶t̶i̶l̶l̶ ̶o̶k̶a̶y̶ ̶f̶o̶r̶ ̶s̶e̶l̶e̶c̶t̶i̶n̶g̶ ̶m̶o̶d̶e̶l̶s̶.̶ Edit: This is incorrect, a PR curve + PR AUC should be used for model selection if imbalanced. I agree it would be really misleading if they (say) just reported accuracy (since the null classifier of always guess negative would give 80% overall accuracy). I̶ ̶t̶h̶o̶u̶g̶h̶t̶ ̶t̶h̶a̶t̶ ̶t̶h̶e̶ ̶A̶U̶C̶ ̶f̶o̶r̶ ̶R̶O̶C̶ ̶c̶u̶r̶v̶e̶ ̶s̶h̶o̶u̶l̶d̶ ̶s̶t̶i̶l̶l̶ ̶b̶e̶ ̶a̶ ̶v̶a̶l̶i̶d̶ ̶m̶e̶a̶s̶u̶r̶e̶ ̶s̶i̶n̶c̶e̶ ̶i̶t̶'̶s̶ ̶s̶h̶o̶w̶i̶n̶g̶ ̶h̶o̶w̶ ̶m̶u̶c̶h̶ ̶b̶e̶t̶t̶e̶r̶ ̶t̶h̶e̶ ̶m̶o̶d̶e̶l̶ ̶p̶e̶r̶f̶o̶r̶m̶s̶ ̶t̶h̶a̶n̶ ̶r̶a̶n̶d̶o̶m̶ ̶g̶u̶e̶s̶s̶i̶n̶g̶.̶

How do you usually handle imbalanced data? I've had some success with SMOTE or weighted loss for imbalanced datasets, but I'm embarrassed to say I've been using AUC with ROC curves as the default - if this gives inferior model selection than AUC with PR curve I'll have to start doing that instead.

    Ruby 2.3 and later ships with this gem and it will automatically be required when a Ruby process starts up. No special setup is required.

If I'm understanding your questions correctly, the main problems I see with this are:

- Using raw data instead of feature engineering (less of a problem given feature synthesis libraries like https://www.featuretools.com/ and other heuristic methods). I'd expect Google to do a good job of basic things like normalization of raw input features before training.

- Using features that it really shouldn't (if you just throw ML at your database for say, loan applications, then sensitive / personally identifying information can/will be used as features)

- Lack of insight / understanding as to what is driving the model. This can be partially overcome with post-training methods like LIME, Shapley values, etc.

I wouldn't expect predictions to be from a set of discrete values - if (say) predicting housing values and training a NN, the output should be continuous and based on the input features.

I like that this will make multithreading easier with shared memory, but I worry this will make the language more complex and delay 1.0. I'll have to read the linked list examples a few times before the owned pointer model sinks in.

Remote: Yes

Technologies: Keras, R, Python, and Spark are what I use ever day. Jupyter, scikit-learn, spaCy, NLTK, H2O, mlr, caret, prophet, tsfresh, ggplot, and tidyverse packages are libraries that I use commonly. I'm interested in exploring more with Pytorch but haven't used it much in production. I'm comfortable with Go, Flask, and Docker (mainly for productionizing models as a microservice) but don't use much in my current role.

Willing to relocate: No

Resume / CV: https://docs.google.com/document/d/1bx9MtbzhzMQDsRiX80Sh4GO5...

Email: sloanes dot k at gmail dot com

---

Looking for a data scientist position.

I think this does everything MLFlow does and more (besides maybe helping with deployment?)