But it's moot, since Apache Lucene is part of the Apache Software Foundation and has much stronger promises about its licensing and governance. Which is not a small reason why Lucene is the de facto standard for search technology.

Comments URL: https://news.ycombinator.com/item?id=10976084

Points: 4

# Comments: 0

It's because Heroku's choice of vendors shouldn't matter to their customers. They see it as an implementation detail, and their responsibility to manage.

So I don't think that's an obfuscation. The people I know at Heroku all have an attitude of, "The buck stops here."

Addon provider here: haven't heard anything official from Heroku on this, so this is my own personal speculation based on the current public Provider API. It seems that Heroku Postgres and Redis are available, and while they're _technically_ addons, they naturally have access to somewhat privileged APIs and architectural information that other addons do not have.

Currently, when an addon is provisioned, we're given a region identifier for the US East and EU public regions. My uninformed guess is that Private Spaces amounts to "your dynos run on servers in a private VPC." IF the Postgres and Redis integrations were "quick 'n dirty," they could very well get provisioned within the same VPC. However, it also seems plausible that AWS VPC peering can be used for other addons to provide their own Private Spaces support.

So it seems to me the question comes down to whether Heroku can (and/or _wants_ to) support VPC pairing with addons via their Provider API, so that other providers can provide their own private spaces.

In the context of reasonable hobby/development costs, I our $10/mo plan fits well for that and is pretty popular. Pricing wise it compares well to a $7 dyno or $9 Postgres instance. And addons are prorated to the second, just like dynos.

Echoing mrkurt, ES is indeed hefty to run.

Elasticsearch is going to require a hundred+ megs of RAM just to run an empty node. And then you'll want enough heap and CPU that you're not GCing yourself to death while importing data, or while churning through filter and caches. And then enough memory to run those fancy aggregations in Kibana. Times the number of nodes you want for redundancy, and for your master-role quorum.

It's possible to do long-tail low-volume ES hosting for hobby development. But to do so you need to dig in and consider ES hosting as a holistic system. We've worked hard at Bonsai to build a hobbyist-friendly hosting system. Our approach is to fork ES itself to introduce native multi-tenancy. That means everyone in the cluster is amortizing the overhead of all that JVM memory (and redundant data nodes, and master-node quorum, a load balancer and authenticating reverse proxy, and analytics... etc...)

The problem remains that ES, while easy to spin up, is still a pretty heavy system to manage at scale. So it's generally a lot easier to chuck some extra resources at it, maximize your isolation, and eat the higher entry cost.

Indeed; more generally, I said CLI. TLDR of the previous link: it's an interesting general-purpose take on using ssh public/private keys (and methods) to sign and authenticate HTTP requests.

I've done a couple of HTTP auth schemes for my dayjob and am thinking of using something like this for my next.

1. https://github.com/joyent/node-http-signature/blob/master/ht...

It was an old API access key that got leaked, not our account credentials. We're still investigating how and where the key got leaked, but bottom line, it should have been revoked ages ago.

2FA is great, but it doesn't cover API keys. Rotate your API keys!

It's also possible to download and install ES locally and run any number of front-end interfaces, some of which include query builders. ElasticHQ seems like a decent option for that. The venerable Elasticsearch-head is another.

I think now that ES 1.0 has shipped, more experimental tools will start to emerge that help people learn and interact with ES itself. (If anyone out there is a front-end whiz and wants to help me build something like that, please email nz@bonsai.io!)

1. http://www.elasticsearch.org/guide/en/elasticsearch/referenc...

But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use cases. Maybe not even for most use cases. There are just too many tradeoffs to consider.

Believe me, I earn a living hosting Elasticsearch, so I'd love to see it become a robust primary data store. There are some use cases where it actually does make sense—just look at the amazing traction ES is experiencing for storing and indexing time-series data.

But as a general-purpose primary store, I'm not really holding my breath. Maybe I'm just becoming battle-worn and bitter. I would love to be proven otherwise over the next few years!

The main thing for good stability and performance is to be very good at batching your updates. You don't want to sling a ton of highly-parallel single-document updates at Lucene, lest you thrash the JVM and start garbage collecting like crazy.

From there, on the query side, you'll want to get a good working knowledge of the different tokenization and analysis options. There are a lot of subtle and interesting combinations to be had in there that influence performance and relevance of your search results.