The copy-pasting of data from my markdowns to the tax software was the hardest path for me as well.

The intention of this is to reduce hallucination on information extraction, right?

Also, how do you convert your docs / information into JSON documents?

Comments URL: https://news.ycombinator.com/item?id=47557058

Points: 17

# Comments: 10

I have used the kk8s MCP directly inside Github Copilot Chat in VSCode and restricted the write tools in the Configure Tools prompt. With a pseudo protocol established via this MCP and the IDE integration, I find it much safer to prompt the LLM into debugging a live K8s cluster vs. without having any such primitives.

So I don't see why MCPs are or should be dead.

I built this because I was tired of the compute markup that products like AWS EMR and Databricks charge for the convenience of using Apache Spark via their platforms. One can argue that Databricks is a superior product with a lot of additional value in their offering but I don't see that with AWS EMR Apache Spark at all (given my personal experience working with it).

My motivation to build this was to be able to create your own Apache Spark cluster without needing any understanding of the underlying data infrastructure engineering and quickly get to the point of writing Spark pipelines, whether as Python applications or Jupyter notebooks, all with no markup on compute because I don't think that is a justified narrative.

It took me almost an year to build it with a day job and of course I used AI for frontend design and video narrations, the infrastructue engineering that goes behind it comes with quite a bit of experience in the industry. The backend that orchestrates the cluster is written with the following:

- Django and DRF for API

- Temporal for async workers

- Pulumi that is run via Temporal workers to orchestrate the cluster

- Karpenter for node auto-scaling based on Spark executor workloads and requests

- Librechat for Spark History server and MCP based debugging for Spark pipeline run analysis

There are currently no caps on the CPU limits so you can try this out today in your own personal AWS accounts for free.

Also looking for feedback on HN.

Comments URL: https://news.ycombinator.com/item?id=47075959

Points: 3

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=47038830

Points: 1

# Comments: 0

Currently implemented the following:

- Automated scale in / scale out of nodes for Spark executors and drivers via Karpenter

- Jupyter notebook integration that works as a Spark driver for quick iteration and prototyping

- A simple JSON based IAM permissions managementent via AWS Parameter Store

Work-in-progress this month:

- Jupyterhub based Spark notebook provisioning

- Spark History Server

- Spark History Server MCP support with chat interface to support Spark pipeline debugging and diagnostics

Open to feedback and connecting. Docs at https://docs.orchestera.com/

The platform I am building allows users to launch Spark on Kubernetes in their own AWS account without adding any markup costs to the CPU/Memory on EC2 instances. For example, AWS EMR offering adds a 25% markup cost on top of the EC2 instance pricing. Databricks markup is even higher ranging anywhere from 30% to 100% markup.

To learn more, check out the docs for the project at https://docs.orchestera.com/

Comments URL: https://news.ycombinator.com/item?id=46287648

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=44734198

Points: 2

# Comments: 0

https://homeautomation.substack.com/p/setting-up-home-assist...