Hacker News: lwthiker

New comment by lwthiker in "Ask HN: What are the best tools for web scraping in 2022?"

lwthiker — Wed, 10 Aug 2022 12:28:58 +0000

curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that.

[1] https://github.com/lwthiker/curl-impersonate

New comment by lwthiker in "Firefox appears to be flagged as suspicious by Cloudflare"

lwthiker — Sat, 21 May 2022 18:11:50 +0000

For a while now I can't even access WhatsApp Web [1] through Firefox. It gets stuck on the loading page and keeps refreshing itself forever. I have to resort to Chrome whenever I need WhatsApp on my laptop.

[1] https://web.whatsapp.com

New comment by lwthiker in "Firefox appears to be flagged as suspicious by Cloudflare"

lwthiker — Sat, 21 May 2022 17:58:31 +0000

I think they have different plans and configurations for their anti-bot service [1]. I'm not sure though because I'm not using their services.

[1] https://developers.cloudflare.com/bots/get-started/

Firefox appears to be flagged as suspicious by Cloudflare

lwthiker — Sat, 21 May 2022 16:26:29 +0000

Article URL: https://lwthiker.com/opensource/2022/05/21/firefox-flagged-suspicious.html

Comments URL: https://news.ycombinator.com/item?id=31459258

Points: 337

# Comments: 168

New comment by lwthiker in "Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake"

lwthiker — Fri, 18 Feb 2022 10:46:02 +0000

They don't block you completely, just present you with a JS challenge that delays your access to the site. A browser, even if behind a MITM proxy, would be able to solve this challenge.

New comment by lwthiker in "Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake"

lwthiker — Fri, 18 Feb 2022 06:35:53 +0000

I hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.

New comment by lwthiker in "Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake"

lwthiker — Fri, 18 Feb 2022 06:09:38 +0000

I will try to impersonate Chrome next, However, I suspect this is going to be more challenging. Chrome uses BoringSSL, which curl does not support. So it means either enforcing curl to compile with BoringSSL or modifying NSS to look like BoringSSL.

New comment by lwthiker in "Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake"

lwthiker — Fri, 18 Feb 2022 06:04:16 +0000

Thanks for the suggestion, I had no idea ESR was a thing. I've just added support for Firefox ESR 91 (it was pretty similar and required adding one cipher to the cipher list and changing the user agent).

Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

lwthiker — Thu, 17 Feb 2022 20:57:37 +0000

Article URL: https://github.com/lwthiker/curl-impersonate

Comments URL: https://news.ycombinator.com/item?id=30378562

Points: 249

# Comments: 57

Analyzing a Stock Exchange's API

lwthiker — Sat, 12 Feb 2022 10:39:28 +0000

Article URL: https://lwthiker.com/reversing/2022/02/12/analyzing-stock-exchange-api.html

Comments URL: https://news.ycombinator.com/item?id=30311686

Points: 1

# Comments: 0

Show HN: Pumbaa – A portfolio simulation tool for passive, index investments

lwthiker — Fri, 11 Feb 2022 13:15:39 +0000

Hi HN,

About a year ago I wrote a few scripts and spreadsheets to calculate the historical performance of various passive investment portfolios. There is portfoliovisualizer.com but if you are not from the US it's less relevant because everything is calculated with respect to the US dollar, and you cannot combine funds from other countries (say from the EU).

In the past month I managed to turn it into a web app with the goal of giving individuals better tools for financial education. It allows you to build a portfolio composed of a few index funds (ETFs or mutual funds) and simulate its historical performance. It currently works with 7 currencies and includes funds from the US, EU, UK, Canada and Israel. Intentionally, only broad-market, passive index funds are included.

If all of this doesn't mean much to you, you can visit the examples page to get a taste of some interesting portfolios. Here is an interesting example: For the past 22 years, US long-term treasuries had very similar returns to the S&P 500 (cherry-picked dates, but still interesting) [1].

On the technical side: The backend is a simple Flask app. A task runs once a day in the background to fetch the latest prices of ETFs and mutual funds from 20+ sources (central banks, stock exchanges). Frontend is Tailwind for CSS and a bit of native Javascript with d3. Everything runs on a $10/mo VPS.

This is my first web app and there is still a lot of room for improvement so any feedback is welcome.

Thanks in advance!

[1] https://pumbaa.app/backtest?b=1¤cy=USD&initialBalance=...

Comments URL: https://news.ycombinator.com/item?id=30301047

Points: 4

# Comments: 0