https://deusu.org

Comments URL: https://news.ycombinator.com/item?id=13062278

Points: 5

# Comments: 0

And I started this software 20 years ago. Granted, a LOT of the software has changed since then. But I don't see a reason to throw away existing code unless it is in need of so much change that rewriting from scratch would be easier. And even then I might stick to what I know best, and what fits best with other parts of the software.

https://deusu.de/blog/2015-12-03-alle_jahre_wieder_wonach_de...

Warning! This is definitely NSFW! :)

What do you mean with postings?

The main index is split into 32 shards (there is also an additional news-index which is updated about every 5-10 minutes). Each shard is updated and queried seperately. The query actually runs 2/3 on a Windows server and 1/3 on a Linux server. The latter in Docker containers. I want to move everything to Linux over time.

Query has two phases. First only a rough - but fast - ranking is done. Then the top results of all shards are combined and completely re-ranked. This is basically a meta search engine hidden within.

First query phase is in src/searchservernew.dpr, and the second phase is in src/cgi/PostProcess.pas.

I'm gonna further improve this over the next days. Right now it's just a quick'n dirty hack. :)

Depending on who you are (there were 2 bitcoin donations today), you funded either about 18 or 28 hours of operations. :)

A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation.

And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format.

BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

The snippets are currently the first 255 characters of the page's text. For snippets to be customized to the search term, I would have to store all the text of the page. And that would require a lot more disk space. Space that I can't afford at the moment.

Block outgoing connects to local IP nets in your firewall. Otherwise your hosting provider might think you are trying to hack them. Apparently there are a lot of links out there that point to hosts which resolve to private IP ranges.

Another problem with following links is that you are bound to run across some that are malware command & control servers. Had several complaints to my ISP after authorities took over control of one and used the C&C server's domain as a honeypot. My crawler is on a whitelist now.

I had one person who vehemently complained that I was trying to hack him, because the software downloaded his robots.txt. I'm NOT kidding! :)

Make sure your robots.txt parsing is working correctly. I had an undiscovered bug in the software at some time which basically caused it to think everything is allowed. Luckily someone was nice enough to let me know. And he was really nice about it. And he would have had every right to be angry.

A major bottleneck is DNS queries. Run your own DNS server and even cache the hostname/IP pairs yourself. Do not even think about using your IPS's DNS server. If you bombard them with 100+ DNS requests/s then they WILL be angry. :)

2 are used for crawling, index-building and raw-data storage. Quadcore, 32gb RAM, 4tb HDD and 1gbit/s internet connection on each of these. They are rented and in a big data-center. Crawling uses "only" about 200-250mbit/s of bandwidth.

2 servers for webserver and queries. Quadcore, 32gb RAM. One with 2x512gb SSD, the other with only 1x512gb SSD. These servers are here at home. I have cable internet with 200mbit/s down, 20mbit/s up. Static IPs obviously.

A full crawl currently takes about 3 months.

There is also a free API in beta-test right now. Will probably be ready for official release next week.