Hacker News: data_dan_

New comment by data_dan_ in "Ask HN: Could you share your personal blog here?"

data_dan_ — Thu, 06 Jul 2023 18:02:44 +0000

https://www.danliden.com/

At the moment, I post fairly infrequently. The whole site is written using emacs org mode. Most of the posts have to do with emacs and data stuff (often doing data stuff in emacs).

Pre-Classify Tasks for Better ChatGPT Completions

data_dan_ — Wed, 26 Apr 2023 12:57:01 +0000

Article URL: https://innerjoin.bit.io/pre-classify-tasks-for-better-chatgpt-completions-f197ad01618c?gi=448e4c0c1717

Comments URL: https://news.ycombinator.com/item?id=35712919

Points: 2

# Comments: 0

Vector Similarity Search in Postgres with bit.io and pgvector

data_dan_ — Mon, 10 Apr 2023 15:23:34 +0000

Article URL: https://innerjoin.bit.io/vector-similarity-search-in-postgres-with-bit-io-and-pgvector-c58ac34f408b

Comments URL: https://news.ycombinator.com/item?id=35513432

Points: 5

# Comments: 0

Comparing ChatGPT to Codex for text-to-SQL translation

data_dan_ — Fri, 17 Mar 2023 20:18:34 +0000

Article URL: https://innerjoin.bit.io/llm-sql-translation-prompt-testing-part-1-comparing-chatgpt-to-codex-78da57213ebe

Comments URL: https://news.ycombinator.com/item?id=35202332

Points: 4

# Comments: 0

Make ChatGPT Stop Chatting and Start Writing SQL

data_dan_ — Wed, 15 Mar 2023 16:58:38 +0000

Article URL: https://innerjoin.bit.io/make-chatgpt-stop-chatting-and-start-writing-sql-fd5560049ae4?gi=0dbaa203018e

Comments URL: https://news.ycombinator.com/item?id=35171230

Points: 4

# Comments: 0

Making a Production LLM Prompt for Text-to-SQL Translation

data_dan_ — Thu, 02 Mar 2023 20:46:57 +0000

Article URL: https://innerjoin.bit.io/making-a-production-llm-prompt-for-text-to-sql-translation-b798b6e94783

Comments URL: https://news.ycombinator.com/item?id=35001805

Points: 3

# Comments: 0

AI-Powered Text-to-SQL Translation in Bit.io

data_dan_ — Mon, 27 Feb 2023 20:56:17 +0000

Article URL: https://blog.bit.io/ai-powered-text-to-sql-translation-in-bit-io-1fbcf32fd586

Comments URL: https://news.ycombinator.com/item?id=34961741

Points: 5

# Comments: 0

Measuring and Mitigating Postgres Network Latency

data_dan_ — Tue, 13 Dec 2022 13:55:28 +0000

Article URL: https://innerjoin.bit.io/the-distance-dilemma-measuring-and-mitigating-postgres-network-latency-76f6cd1a6c57

Comments URL: https://news.ycombinator.com/item?id=33968840

Points: 3

# Comments: 0

Cities in the Southwest Decouple Growth from the Need for More Water

data_dan_ — Mon, 09 May 2022 15:49:54 +0000

Article URL: https://e360.yale.edu/features/a-quiet-revolution-southwest-cities-learn-to-thrive-amid-drought

Comments URL: https://news.ycombinator.com/item?id=31315675

Points: 2

# Comments: 1

Literate Emacs Config with imenu-list

data_dan_ — Sat, 07 May 2022 18:23:06 +0000

Article URL: https://github.com/rougier/dotemacs

Comments URL: https://news.ycombinator.com/item?id=31297112

Points: 1

# Comments: 0

New comment by data_dan_ in "Record Linkage on People's Names with Approximate String Matching"

data_dan_ — Thu, 05 May 2022 15:24:07 +0000

I wrote this article. For some background—

In March, we published an article on Stock Trades by members of congressional committees: https://innerjoin.bit.io/data-cant-tell-us-whether-congressi...

To conduct this research, we needed to know: (1) which members of congress made which stock trades, and (2) which members of congress belonged to which congressional committees. The data for (1) was available from the the senate/house stock watchers sites; the data for (2) came from the ProPublica Congress API. There was no primary key available for linking the two datasets: the best we had to work with were the names of the members of congress.

This would be fine, if the names were represented uniquely and consistently. This was not the case. You can't join "Mitch McConnell" to "A. Mitchell McConnell, Jr." without a bit of work.

Manually matching every single name from the first data source to every single name in the second would be tedious, time consuming, and error prone. Instead, we used the Levenshtein distance to compute a similarity metric between each name in the first dataset and each name in the second. Simply using the best match according to this metric correctly matched more than 95% of the names, and made it incredibly simple to review the list and manually fix the few incorrect matches.

There's also an accompanying Deepnote dashboard where you can compare string distances between pairs of strings of your choosing: https://deepnote.com/@dliden-bitdotio/Whats-in-a-Name-28418c...

Record Linkage on People's Names with Approximate String Matching

data_dan_ — Thu, 05 May 2022 14:54:59 +0000

Article URL: https://innerjoin.bit.io/simple-record-linkage-with-approximate-string-matching-a4a480566122?gi=57afada8ed63

Comments URL: https://news.ycombinator.com/item?id=31274156

Points: 2

# Comments: 1

Using Telemedicine to Address the Access Gap for Opioid Treatment

data_dan_ — Mon, 02 May 2022 23:38:15 +0000

Article URL: https://knowablemagazine.org/article/mind/2022/does-online-opioid-treatment-work

Comments URL: https://news.ycombinator.com/item?id=31242537

Points: 2

# Comments: 0

How Much Resource Use is Fair? National responsibility for ecological breakdown

data_dan_ — Mon, 18 Apr 2022 16:08:34 +0000

Article URL: https://blog.datawrapper.de/fair-share-resources/

Comments URL: https://news.ycombinator.com/item?id=31072610

Points: 3

# Comments: 0

Surveying SQL parser libraries in a few high-level languages

data_dan_ — Mon, 18 Apr 2022 07:54:34 +0000

Article URL: https://datastation.multiprocess.io/blog/2022-04-11-sql-parsers.html

Comments URL: https://news.ycombinator.com/item?id=31068375

Points: 2

# Comments: 0

Bayesian Data Analysis with Jags: A Dirichlet-Multinomial Model of Wordle Scores

data_dan_ — Fri, 15 Apr 2022 11:44:47 +0000

Article URL: https://innerjoin.bit.io/ask-a-bayesian-who-is-better-at-wordle-76a0e5199ed

Comments URL: https://news.ycombinator.com/item?id=31039061

Points: 2

# Comments: 0

New comment by data_dan_ in "How Fast Can We Write a 10M Row CSV to a Postgres Database with Pandas?"

data_dan_ — Wed, 02 Mar 2022 18:22:57 +0000

I wrote this article after a colleague pointed out that the Pandas DataFrame.to_sql() method uses row-by-row INSERTs. There are plenty of good reasons for this, and the to_sql method works great with many different SQL database flavors, but it's not fast.

This article compares the performance of different methods for writing a Pandas DataFrame to a PostgreSQL database using the to_sql method on DataFrames ranging from 100 rows to 10,000,000 rows.

New comment by data_dan_ in "How Great Is the Great Resignation? A Primer on Labor Turnover Data Sources"

data_dan_ — Wed, 09 Feb 2022 23:48:58 +0000

To me it just looks like it's too early to tell. Did quits go up very quickly in 2021? Sure! But that comes on the heels of a massive spike in layoffs that occurred in 2020. It is at least a possibility that the current situation is a response to that.

One point I didn't go into is the fact that the labor force participation rate also dropped steeply in 2020 and hasn't recovered to pre-pandemic levels yet. So that could create labor shortages that are not necessarily represented in the quits rate.

New comment by data_dan_ in "How Great Is the Great Resignation? A Primer on Labor Turnover Data Sources"

data_dan_ — Wed, 09 Feb 2022 15:14:28 +0000

(I wrote this) I wish I had clearer answers to those questions! The conclusion is just a little unsatisfying from a writing perspective—it's just too early to tell exactly what's going on. The quits data through most of 2021 don't look all that different from what we'd expect based on the pre-pandemic trend, though there were definitely more than anticipated, especially later in the year. But this is following a massively-disruptive pandemic that put a lot of people out of work and in general had a dramatic impact on the employment situation.

I think the most interesting part is the decrease in layoffs that coincided with the increase in quits. People aren't leaving that much more than before, but when they leave, they're doing so on their own terms.

New comment by data_dan_ in "Ask HN: At what point in your engineering career did you feel respected?"

data_dan_ — Mon, 03 Jan 2022 21:28:07 +0000

Maybe I've been especially fortunate, or I'm just not understanding the question right, but I've felt respected pretty much from the beginning of my career (aside from grad school, which was not great, in many respects). Or at least I've felt "treated with respect" -- not sure if that's exactly the same thing. But I've always been given a fair amount of independence at work, and I've generally been able to solve whatever I'm assigned to solve, or clearly articulate the challenges, allowing those with different/more expertise to help out. And I've never been made to feel ashamed or inadequate because of either the work I've completed or the work I've needed to seek out additional help to complete.