In the programming language context, dialect can be applied on varying levels but usually signifies the former, where each individual or group has a persistent preference for some language or style. Within a single programming language, dialects are usually a bad thing because they risk splintering the community into mutually incompatible subgroups (e.g. scala fp styles, c++ boost).

Part of the reason why evolving a language is hard is because every time you introduce a new way to do something which could be done before, users have to choose. If that choice divides users into groups that persistently pick one over the other based on style or community affiliation, you’ve introduced a new dialect. If the choice flows more naturally from the situation the user finds themselves in, you’ve introduced a new register.

What worries me the most is the effect the public backlash towards these big companies can have on smaller actors that could enter this space in the near future. In the past we’ve seen open source projects like GPT-J come together to fund and reproduce closed models, and if we’re not careful to be nuanced in our criticism of big-tech frontrunners we might end up poisoning the waters enough to deter small actors without a dedicated legal team.

Copyright law is ultimately designed around humans as the only kind of actor. In an ideal world we would sit down and think about the way non-human learners should fit into this system and the balance of tradeoffs we want those laws to aim for. I hope that happens someday, but until then I hope we can cultivate a world where small actors are able to experiment with these technologies without fear of legal action.

That’s why it bothers me to see people arguing that language models should be thought of like human programmers making derivative works, even suggesting that we should require attribution for all generated outputs (i.e. the entire training set, always). That helps nobody, except of course big companies with infinite manpower.

On the practical level I agree with the part advising caution to those that might end up embedding an identifiably licensed snippet in their codebase via copilot. I also agree that copilot users plagiarizing significant chunks of GPL code for profit is immoral. This needs to be prevented.

I also share the frustration stemming from big companies leveraging their disproportional access to data and resources for profit given that the greatest value of these models is precisely the open source code it is trained on.

Ultimately though, what I care about is the potential for building better tools. LLMs potentially offer paths towards genuinely new forms of human-machine interaction, and I don’t want that exploration to be suffocated by legalism.

Today, progress in machine learning is almost entirely empirical, with limited theoretical advances following later to make sense of the empirical findings. Our theoretical underpinnings are so weak we don’t even have a rough estimate of how much harder it is to make a self-improving AGI than e.g. GPT-3. Maybe it’s many orders of magnitude harder, and not even remotely solvable by taking iterative steps from where we are now. Maybe it’s just one unifying theoretical advance away.

As for waiting and hoping it happens on its own, that’s not a great description of what’s happening. A huge number of people (possibly too many, leading to short-term incentives) are trying to make improvements in any way they can think of. Progress is happening at a staggering pace —- we just don’t have any idea how much progress is needed to reach the goal of AGI.

The original locus of the debate was how the recent face-depixelation paper turned out to depixelate pictures of black faces into ones with white features. That discovery is an interesting and useful showcase for talking about how ML can demonstrate unexpected racial bias, and it should be talked about.

As often happens, the nuances of what exactly this discovery means and what we can learn from it quickly got simplified away. Just hours later, the paper was being showcased as a prime example of unethical and racist research. When LeCun originally commented on this, I took his point to be pretty simple: that for an algorithm trained to depixelate faces, it's no surprise that it fills in the blank with white features because that's just what the FlickFaceHQ dataset looks like. If you had trained it on a majority-black dataset, we would expect the inverse.

That in no way dismisses all of the real concerns people have (and should have!) about bias in ML. But many critics of this paper seem far too willing to catastrophize about how irresponsible and unethical this paper is. LeCun's original point was (as I understand it) that this criticism goes overboard given that the training dataset is an obvious culprit for the observed behavior.

Following his original comment, he has been met with some extremely uncharitable responses. The most circulated example is this tweet (https://twitter.com/timnitGebru/status/1274809417653866496?s...) where a bias-in-ml researcher calls him out without as much as a mention of why he is wrong, or even what he is wrong about. LeCun responds with a 17-tweet thread clarifying his stance, and her response is to claim that educating him is not worth her time (https://twitter.com/timnitGebru/status/1275191341455048704?s...).

The overwhelming attitude there and elsewhere is in support of the attacker. Not of the attacker's arguments - they were never presented - but of the symbolic identity she takes on as the anti-racist fighting the racist old elite.

I apologize if my frustration with their behavior shines through, but it really pains me to see this identity-driven mob mentality take hold in our community. Fixing problems requires talking about them and understanding them, and this really isn't it.

Psychedelics amplify the perceived significance of many experiences, but what experience are you likely to be getting at a hospital or psychiatric facility? Every encounter I have had with modern medicine has been uncomfortable or even slightly demeaning.

The way we practice medicine today has bought us repeatability and accountability of treatment, but with it comes an overwhelming air of impersonality. That's not an issue if you needed surgery, but if you're prescribed a transformative experience I'd say that might be a show stopper.

Technological discovery doesn't just happen in a vacuum. There needs to be a societal production of potential new inventors and scientists. Where their interests lie is to some extent influenced by the culture they grow up in, and which research interests will receive funding (or societal status) is also dependent on culture, etc. Worse, if a problem has been widely solved using a bad solution, there is no real desire for a good one.

Take electrical engineering and circuit design as an example. I would make the case that despite the immense and obvious success of computers, our society is in some significant way worse off than it was 50 years ago if you were to measure our collective ability to solve new problems using electrical engineering. Fewer and fewer electrical engineers are being educated, while the competency threshold of the field is rising. As that threshold rises, our education system shifts from teaching people to invent things to teaching them about practical skills like how to work all the menus in [some modeling tool], and how to generate report templates for MS Word.

Software is so new that we've barely exceeded a single human lifespan (hardly enough time for information to get lost is it?), but how many times have you seen companies with a code base that was written by some greybeard in the 80's, which is critical to the success of the company but nobody understands? When the company realised that, did they try to address the root of the problem, or did they decide to keep piling shit on top of what they already had?

It may be that software itself is the problem. Software allows you to snapshot your current problem solving capacity and continue delivering it long after the problem solvers themselves are gone. Maybe if we're lucky, the current generation of AI researchers will get us to something that approaches human level intelligence before these problems become intractable.

- The article focuses on the properties of codebases, not individual code snippets. While it would be very valuable to do compare the properties of entire codebases, that is demanding on both the reader and the author.

- Opening the article with a statement like “Choosing a programming language is never easy...” is an acknowledgement of the fact that he is not claiming Go is unequivocally the best in all scenarios. The author is signaling that he is a reasonable person, and while that may seem like fluff it is a necessary component of communicating with a wide audience.

Recently there has been a lot of discussion in Sweden (maybe elsewhere too) about anonymized mobile phone location data that is sold online. In that case "data anonymization" usually meant swapping out personal identifiers for some token. If that was the only information you had you'd be more or less fine, but what if you have access to some correlated side-channel information that IS personally linked? In the location data example, just combining with publicly available home address data is enough to de-anonymize nearly every person in the dataset (i.e. where does anonymous token X go every night and leave every morning?).

This problem emerges very quickly as soon as you start linking together multiple pieces of anonymized data (or just sampling the data with high enough resolution). The only real virtue of data anonymization is that it prevents casual snooping by the people that work with the data.

One of the things I'm considering as a possible explanation for my issues is that all of the changes I've made have made my usage patterns smaller and more repetitive, and that going back to more inefficient methods slow me down and force me to move my arms in bigger movements might actually be less painful.

With small tweaks like a non-symlinked file that contains only machine local variables (I call mine .bashrc.local and source it from the main bashrc), you can evem make the configs vary across machines on the subtler things.

This works wonders for me getting custom configs everywhere, with the exception of windows machines that are always a huge headache to work with.

I am now about 7 years into an RSI issue that nothing seems to work on, but I haven't committed to any training routine for more than a couple of weeks (with no effect).