Hacker News: dmoura

Open Dataset: Vehicle Accidents

dmoura — Thu, 13 Mar 2025 16:20:50 +0000

Article URL: https://huggingface.co/datasets/nexar-ai/nexar_collision_prediction

Comments URL: https://news.ycombinator.com/item?id=43354796

Points: 2

# Comments: 0

Nexar Dashcam Crash Prediction Challenge

dmoura — Fri, 07 Mar 2025 19:30:50 +0000

Article URL: https://www.kaggle.com/competitions/nexar-collision-prediction

Comments URL: https://news.ycombinator.com/item?id=43293515

Points: 10

# Comments: 4

Limitations of 'Understanding the Limitations of Mathematical Reasoning in LLMs'

dmoura — Thu, 07 Nov 2024 18:21:05 +0000

Article URL: https://desirivanova.com/post/gsm-symbolic/

Comments URL: https://news.ycombinator.com/item?id=42079309

Points: 1

# Comments: 0

New comment by dmoura in "My solopreneur story"

dmoura — Sat, 23 Sep 2023 21:23:58 +0000

Thank you for sharing your learnings and for your transparency! Congrats!

New comment by dmoura in "Fq: Jq for Binary Formats"

dmoura — Sat, 03 Jun 2023 15:31:59 +0000

I prefer a SQL-like format. It’s not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql (I am the author). Congrats on fq!

The Five Dimensions of Sustainable Software Engineering

dmoura — Fri, 25 Nov 2022 18:22:38 +0000

Article URL: http://luiscruz.github.io/2022/01/01/sustainable-se-intro.html

Comments URL: https://news.ycombinator.com/item?id=33745280

Points: 2

# Comments: 0

Kangas: Explore Multimedia Datasets at Scale

dmoura — Mon, 21 Nov 2022 14:51:50 +0000

Article URL: https://github.com/comet-ml/kangas

Comments URL: https://news.ycombinator.com/item?id=33693106

Points: 9

# Comments: 2

Neural Geometry and Rendering ECCV2022

dmoura — Mon, 14 Nov 2022 10:10:13 +0000

Article URL: https://ngr-co3d.github.io/

Comments URL: https://news.ycombinator.com/item?id=33592068

Points: 2

# Comments: 0

Reconstructing Training Data from Trained Neural Networks

dmoura — Mon, 14 Nov 2022 10:07:22 +0000

Article URL: https://giladude1.github.io/reconstruction/

Comments URL: https://news.ycombinator.com/item?id=33592049

Points: 11

# Comments: 0

New comment by dmoura in "Command-line data analytics"

dmoura — Thu, 03 Nov 2022 14:09:31 +0000

DuckDB is great! I love what you guys are building. The main gap for me is native support of JSON (lines), like you have for CSV and Parquet.

New comment by dmoura in "Command-line data analytics"

dmoura — Thu, 03 Nov 2022 14:05:55 +0000

updated, thank you

New comment by dmoura in "Command-line data analytics"

dmoura — Thu, 03 Nov 2022 13:02:22 +0000

Things you can do with SPyQL CLI that you can't with clickhouse local (AFAIK, top of my mind, not exhaustive):

- use python code in your queries

- import python libs (just install them with pip/conda)

- write your one UDFs in Python

- run OS commands from within the query (using os.system)

- have guaranty of row order (like in grep, sed, etc)

And there is more, please take a look at: https://spyql.readthedocs.io/en/latest/distinctive.html

New comment by dmoura in "Command-line data analytics"

dmoura — Thu, 03 Nov 2022 12:19:08 +0000

Author of the benchmark and of SPyQL here. ClickHouse is fantastic. Amazing performance. SPyQL is built on top of Python but still can be faster than jq and several other tools as shown in the benchmark. SPyQL can handle large datasets but Clickhouse local should always show better performance.

SPyQL CLI is more oriented to work in harmony with the shell (piping), to be very simple to use and to leverage the Python ecosystem (you can import Python libs and use them in your queries).

New comment by dmoura in "JC – JSONifies the output of many CLI tools"

dmoura — Thu, 03 Nov 2022 11:07:44 +0000

This is great!

I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.

I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.

[1] https://github.com/dcmoura/spyql [2] https://danielcmoura.com/blog/2022/spyql-cell-towers/

Command-line data analytics

dmoura — Tue, 01 Nov 2022 08:52:09 +0000

Article URL: https://danielcmoura.com/blog/2022/spyql-cell-towers/

Comments URL: https://news.ycombinator.com/item?id=33418573

Points: 103

# Comments: 25

SPyQL – SQL Powered by Python

dmoura — Sun, 30 Oct 2022 23:35:29 +0000

Article URL: https://spyql.readthedocs.io/en/latest/index.html

Comments URL: https://news.ycombinator.com/item?id=33400202

Points: 3

# Comments: 0

Make beautiful visualisations of large graphs online (2015)

dmoura — Wed, 04 May 2022 22:03:27 +0000

Article URL: https://github.com/dcmoura/3DHEB

Comments URL: https://news.ycombinator.com/item?id=31266438

Points: 2

# Comments: 0

New comment by dmoura in "The fastest tool for querying large JSON files is written in Python (benchmark)"

dmoura — Thu, 21 Apr 2022 16:48:09 +0000

Thank you all for your feedback. The benchmark was updated and the fastest tool is NOT written in Python. Here are the highlights:

* Added ClickHouse (written in C++) to the benchmark: I was unaware that the clickhouse-local tool would handle these tasks. ClickHouse is now the fastest (together with OctoSQL);

* OctoSQL (written in Go) was updated as a response to the benchmark: updates included switching to fastjson, short-circuiting LIMIT, and eagerly printing when outputting JSON and CSV. Now, OctoSQL is one of the fastest and memory is stable;

* SPyQL (written in Python) is now third: SPyQL leverages orjson (Rust) to parse JSONs, while the query engine is written in Python. When processing 1GB of input data, SPyQL takes 4x-5x more time than the best, while still achieving up to 2x higher performance than jq (written in C);

* I removed Pandas from the benchmark and focused on command-line tools. I am planning a separate benchmark on Python libs where Pandas, Polars and Modin (and eventually others) will be included.

This benchmark is a living document. If you are interested in receiving updates, please subscribe to the following issue: https://github.com/dcmoura/spyql/issues/72

Thank you!

New comment by dmoura in "The fastest tool for querying large JSON files is written in Python (benchmark)"

dmoura — Tue, 19 Apr 2022 11:30:19 +0000

Are you able to calculate aggregates, like an average?

New comment by dmoura in "The fastest tool for querying large JSON files is written in Python (benchmark)"

dmoura — Mon, 18 Apr 2022 09:13:42 +0000

The initial idea was to focus on cmd line tools... I added pandas for comparison, as it is one of the most used libs to work with datasets. I will either remove Pandas from the equation or add Polars. By the way, I run some benchmarks and polars seems a bit faster than spyql for the aggregation challenge, but does not scale (loads everything into memory)