I like your breakdown, and I've observed similar things in my experience as an engineering focused data person! I've had many discussions with my colleagues about how to manage effectively these different blends of roles and skills.

I'm looking for someone for an engineering type of data role right now. Is there a way to get in touch with you about it?

Our product helps companies listen to their customers by unifying natural language feedback across various channels, applying signals using various natural language modeling techniques, then aggregating them to help teams deliver better outcomes using more relevant information.

Hope to hear from you (brandon at frame.ai) :)

edit: forgot to share agreement for your breakdown

The 1960s and 1970s crisis had a lot to do with the end of NYC's industrial epoch. Suburban development and globalization eliminated manufacturing and pulled workers and residents out of the city. The recovery of NYC was bringing high-value services, retail, and tourism back along with arts and culture.

In the time since, NYC has become increasingly a luxury experience, which is indeed part of its strength but also its weakness, since it accelerates decline when people can up and leave without having roots.

The feels correct to me.

I lived in New York City for 15 years. Until last year. I've thought about this theme all year. Decades of policy supporting foreign investment and developer speculation gutted the chance for even affluent upper middle class New Yorkers to afford housing and setup a home base, and so many left. The situation has been incomparably more challenging for low income residents.

I agree the urban collapse meme is much easier to spread than a thoughtful discussion about policy and priorities and how to balance the economic strength of a city's major players with the daily priorities of everyday citizens. I hope the New York remainders shift priorities and initiate a different kind of prosperous era than the one I got to enjoy.

Comments URL: https://news.ycombinator.com/item?id=22564280

Points: 2

# Comments: 0

The article discusses a low-dimensional KNN problem. The curse of dimensionality guides intuition that the methods here likely will not apply to extremely high-dimensional problems.

faiss actually comes with a lot of excellent documentation that describes the problems unique to KNN on embedding vectors. In particular, for extremely large datasets, most of the tractable methods are approximations that make use of clustering, quantization, and centriod-difference tricks to make computation efficient.

See https://github.com/facebookresearch/faiss/wiki/Faiss-indexes and related links for more information.

We've frequently had the same dream of adding more native support for nearest-neighbor type queries, since that is the workhorse of so many useful techniques in the modern NLP stack.

Right now, we have lots of dense vectors stored in massive toast tables in PG. It's faster to fetch them rather than recompute them, especially since there are a number of preprocessing steps that limit what we pay attention to.

The discussion here about full text search versus semantic search is interesting. In our experience, both are highly relevant. Sometimes it's most useful for our customers to segment their conversation data by exact text matches, and other times semantic clustering is most effective. I think there's plenty of reason to offer both kinds of capabilities.

I can't find a way to disable this feature. Does anybody know how?

The reason this bad practice is common is that it is allowed by the spec in https://tools.ietf.org/html/rfc6749#section-6 as an optional action to take on refresh grants. Please, do not do this.

They want to have a rapid response path (little to no delay using staging envs) to respond to emergencies. The old SOP allowed all releases to use the emergency path. By not using it in the SOP anymore, I'd be concerned that it would break silently from some other refactor or change.

Your notion is to maintain the emergency rollout as a relaxation of the new SOP such that the time in staging is reduced to almost nothing. That sounds like a good idea since it avoids maintaining two processes and having greater risk of breakage. So, same logic but using different thresholds versus two independent processes.

  5. Changing the SOP to do staged rollouts of rules in
     the same manner used for other software at Cloudflare
     while retaining the ability to do emergency global
     deployment for active attacks.

It's a fact of managing process that branches are liability and the hot path is the thing that will have the highest level of reliability. I wonder if anyone there has concerns about diluting the rapid response path (the one having the highest associated risk) by making this process change.

edit: fix verbatim formatting

Good riddance to password expiration.

Each shard allows at most 5 GetRecords operations per second. If you want to fan out to many consumers, you will reach those limits quickly and have to implement a significant latency/throughput tradeoff to make it work.

For API limits, see: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...

Consider a common scenario where you SELECT a set of rows and take a SUM over a column. Suppose your query and another query begin reading from the same committed state of the same table. Suppose that the other query uses an UPDATE command on a set of rows in the table, and that the other query commits before yours does. In order to be consistent, the database system must detect the situation where the other query updated a row that affects the filter WHERE you scanned the table, otherwise your sum could be incorrect if the other query's committed state would cause your query to compute a SUM over a different set of rows or over modified values in your SUM. The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause!

Serializability is a well studied concept. You can find lots of good resources about algorithms for implementing concurrency control and for detecting whether or not two transactions are serializable. The high-level summary is that it takes a lot of operational bookkeeping to guarantee that two queries have no conflicts, especially when you are using real-world examples having many filters and joins.

Logging request or response payloads without an explicit whitelist should raise flags for any developer. There are very few cases where you can assert that not only in the present but also for all future use cases of a system, the entirety of a payload will not contain sensitive user data.

Only a whitelist will suffice to maintain good security. It's common for developers to attach sensitive data for debugging and other use cases under arbitrary paths.

Systems can improve further by adding patterns and other heuristics to drop values from the whitelist that look like sensitive data.

Here's the basic recipe:

  1. Spin up a fresh Postgres instance on Docker using -P to claim an available ephemeral TCP port
  2. Use `docker inspect` to read the Postgres port
  3. Run DDL script on the fresh instance
  4. Run SchemaCrawler Docker container using --network host option so it can connect to Postgres
     and using -v so it can save a schema image to the host filesystem