For example, Claude has a lot of trouble reading HN's front page. HN itself is fine, but the moment you ask it to pick out an article, it often chokes. The website has put up a verification captcha, or it's a paywall, etc. Paywalls can be bypassed by reading HN comments and looking for archive links. But those archives often block bots too, so you're back to square one.

Whether it's unethical is an interesting question. I believe I should have the right to do what I want with internet content, as long as I'm not abusive. Merely having a bot isn't abusive. It would be one thing if the bot is hammering a server or vacuuming up training data, but having a bot at all is presently very hard.

This service caught my attention because it could potentially solve the problem I'm running into. Simply taking snapshots of articles that hit HN shouldn't be so hard, but it is. HN sends millions of views to websites; one bot taking a snapshot isn't going to make a difference. I don't think it counts as "unethical" just because we're going against the website owner's wishes. When you post content to the internet, you sign up to share that content with everyone, other than what's denied by robots.txt. If it's not blacklisted by robots.txt, it should be possible for well-behaved bots to access.

I don't expect very many people here to care about the poor bot creators. Most of the bot creators are malicious anyway. But I personally lament the loss of being able to write a program that can process information from the browser in arbitrary ways. You should be able to, yet we're buying into the notion that it's okay for website owners to say "this content is only accessible by approved bots like Google, and everyone else can sod off."

HN proves it doesn't need to be like that. It gets dozens of millions of page views a day, a lot of which is bot traffic. HN only uses captchas for creating accounts or logging in. You're free to scrape any content as long as you respect the crawl delay of 30 seconds specified in robots.txt, and don't try to visit links that perform actions a human would take (like adding things to favorites or voting). That's how the internet should work: just deliver content.

You can do it by including your “user” cookie. (You can get that by examining the cookie store via the browser.)

I’ll slow down.

Comments URL: https://news.ycombinator.com/item?id=48564168

Points: 3

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=48564163

Points: 2

# Comments: 0

For example, to see all comments containing links to x.com:

https://news.ycombinator.com/from?site=x.com&kind=comment

Or all GitHub links:

https://news.ycombinator.com/from?site=github.com&kind=comment

Remarkably it matches subdomains too. Here’s all gists posted to HN:

https://news.ycombinator.com/from?site=gist.github.com&kind=comment

The submitted url shows all HN links posted to HN (i.e. comments containing news.ycombinator.com):

https://news.ycombinator.com/from?site=news.ycombinator.com&kind=comment

Note that this is distinct from showing comments with links to ycombinator.com, which doesn’t show links to HN:

https://news.ycombinator.com/from?site=ycombinator.com&kind=comment

I find this impressive, since it’s not merely matching text (otherwise site=ycombinator.com would also show HN links).

You can even set site=github.com/antirez to find comments with links to antirez’s repos:

https://news.ycombinator.com/from?site=github.com/antirez&kind=comment

site=x.com/paulg doesn’t work though, so apparently site=domain.com/path only works for certain domains.

I just discovered this, and embarrassingly emailed Dan asking whether kind=comment was broken since I didn’t understand what I was looking at. Was this announced somewhere and I missed it?

Comments URL: https://news.ycombinator.com/item?id=48563775

Points: 5

# Comments: 0

For example, to see all comments containing links to x.com:

https://news.ycombinator.com/from?site=x.com&kind=comment

Or all GitHub links:

https://news.ycombinator.com/from?site=github.com&kind=comme...

Remarkably it matches subdomains too. Here’s all gists posted to HN:

https://news.ycombinator.com/from?site=gist.github.com&kind=...

The submitted url shows all HN links posted to HN (i.e. comments containing news.ycombinator.com):

https://news.ycombinator.com/from?site=news.ycombinator.com&...

Note that this is distinct from showing comments with links to ycombinator.com, which doesn’t show links to HN:

https://news.ycombinator.com/from?site=ycombinator.com&kind=...

I find this impressive, since it’s not merely matching text (otherwise site=ycombinator.com would also show HN links).

You can even set site=github.com/antirez to find comments with links to antirez’s repos:

https://news.ycombinator.com/from?site=github.com/antirez&ki...

site=x.com/paulg doesn’t work though, so apparently site=domain.com/path only works for certain domains.

I just discovered this, and embarrassingly emailed Dan asking whether kind=comment was broken since I didn’t understand what I was looking at. Was this announced somewhere and I missed it?

Comments URL: https://news.ycombinator.com/item?id=48563638

Points: 4

# Comments: 5

Concretely: if you can do 90% of your work with AI, someone else can also do that same work, making you interchangeable unless that 10% is really important.

I think this is partly why it's so hard for people to find jobs right now. Everyone is interchangeable thanks to AI, so skill gives you less of an edge than it did in the past.

The trouble with economics is that if you get the incentives slightly wrong, people will relentlessly exploit the difference. So when I ask "but this, but that," I’m asking "have you thought of this exploit or that exploit?" Because unfortunately you can’t just ignore that they exist. Each of them needs a concrete answer, or the system will reward those who exploit it. (You can argue that’s true for the current system too, but at least it’s hard to exploit. The loopholes I listed above are all straightforward.)

Not the same thing, but I made a clone of pg’s website which can be used for exactly that: https://github.com/shawwn/pg

https://shawwn.github.io/pg/

If you want to read all essays, just clone the repo and open any of the .html files. Or any of the .page files which generated them.

It sounded like an easy way to lose $6k, but maybe the upside is worth the risk. I don’t have any experience to know whether this is a sufficiently good bet though.

The ultimate point of money is for someone to have it. And since corporations are owned by people, it’s not enough to say that corporations aren’t bound to the $100m cap. Otherwise people will just hoard money using corporations, and take loans against their value, a bit like how it already works today.

What counts as $100m? If you buy a million dollar house, then sell it a few years later for a million dollars, are you now limited to $99m since you already spent $1m? Does the money from selling the house go to taxes, or to you? If it goes to you, what’s to prevent someone from hoarding equity and only taking out $100m at a time, effectively living the life of a billionaire while keeping their balance under $100m?

If you own some stock, and it rises in value to $150m, you forfeit $50m to taxes, right? But then if the value of the stock goes down by half, would you have $75m or $50m?

And if the answer is “you have $150m, the cap only applies when selling stock,” then what stops people from putting the money under the custody of a business (which you said isn’t bound by the cap) and then taking loans against that extra $50m?

These aren’t contrived scenarios. Stock goes up and down all the time, and it’s important to be clear about how the proposed system will work.

I’m genuinely interested in answers. It’s easy to say “just cap someone’s wealth at $100m,” but people seem to shy away from proposing specifics.

The problem is that when you cap earnings to $100m, most investors lose motivation to invest in startups, because their investment isn’t likely to yield a reward. Unless you think all investments should be less than $100m, this kills large investment rounds. That would have killed OpenAI, for example, since their recent round was larger.

In other words, it’s fine to say “you can only earn a hundred million dollars.” The hard part is implementing it without killing the investment ecosystem. Every investor is chasing the big return that covers the 20 investments that didn’t pan out. If that big return is capped, there won’t be investment, and hence no startups.

To do that, I'm building sharc, a port of Hacker News that runs on Common Lisp, implementing all the latest features of HN. (That last bit is the hard part.) https://github.com/shawwn/sharc

I just implemented collapsing comments and root/next/prev/context nav links. Also (sitename "https://x.com/jsrailton") now returns "x.com/jsrailton" instead of "x.com". (About to implement HN's "from" endpoint, e.g. https://news.ycombinator.com/from?site=twitter.com/jsrailton)

You can read the entire changelog here: https://github.com/shawwn/sharc/tree/main/docs/agents/handof...