jsontap builds a reactive node tree on top of ijson's streaming parser. You await scalar/object values and async for over arrays while the JSON is still being fed in, one chunk at a time. This makes it ideal for LLM token streams, chunked HTTP responses, or any scenario where JSON arrives incrementally.

Comments URL: https://news.ycombinator.com/item?id=47188343

Points: 1

# Comments: 1

The current version includes:

- DAG scheduling with stages, tasks, and dependencies (but perhaps, designing around "RDD" would've been the right call)

- Modeling the input, output, shuffle partition sizes as probability distributions.

- Automatic retries on executor or shuffle-fetch failures

- Single-job execution

- Simple CLI to tweak cluster configuration, simulate failures, and scaling up executors

This tool might be relevant to the following folks:

- Data & Infrastructure engineers running Apache Spark who want to experiment with cluster configurations

- Anyone curious about Spark internals

I'd appreciate feedback from anyone with experience in discrete event simulation, particularly regarding the planned features as well as from anyone who may find this useful to shape its development.

A walkthrough section in the README demonstrates how it can be used.

GH repo https://github.com/fhalde/fauxspark

Comments URL: https://news.ycombinator.com/item?id=45527270

Points: 1

# Comments: 1

I've built FauxSpark. A discrete event simulation of Apache Spark, built with SimPy.

It's designed to let users experiment with and understand the runtime characteristics of Apache Spark workloads under different cluster configurations, failures & job schedules without spawning a real cluster.

In this initial version, FauxSpark implements a simplified version of Apache Spark which includes:

- DAG scheduling with stages, tasks, and dependencies

- Automatic retries of tasks & stages on executor failure

- Stage resubmission on shuffle-fetch failures

- Basic shuffle read (like really simple)

- Runs a single job at a time

- A simple CLI with a few knobs to configure cluster, simulate failures, scale up etc,.

Repo → https://github.com/fhalde/fauxspark

I'd appreciate your feedback and tips from anyone into discrete event simulation (DES).

Comments URL: https://news.ycombinator.com/item?id=45454733

Points: 1

# Comments: 0

Customers: 10$

Jensen: 500$ :(

Customers: 20$

Jensen: 250$

Customer: 10$ + your jacket

Jensen: :'((