1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.

2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.

Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN.

Here’s our talk from Data+AI Summit: Build a Generative AI App on Enterprise Data in 13 Minutes https://www.youtube.com/watch?v=1exLfT-b-GM

Here’s a blog/demo https://www.prophecy.io/blog/prophecy-generative-ai-platform...

1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.

2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.

Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN. Here’s our talk from Data+AI Summit:

Build a Generative AI App on Enterprise Data in 13 Minutes

https://www.youtube.com/watch?v=1exLfT-b-GM

Here’s a blog/demo

https://www.prophecy.io/blog/prophecy-generative-ai-platform...

We provide a low code data transformation product (prophecy.io), and we’ll never close sales at any volume, if we have a to get an MSA that approves this. Might get easier if we become large :)

Work on LLMs and Knowledge Graphs fir English -> ETL data pipelines

You should have previous experience in NLP and solid programming. Shipping a project from scratch is important.

If you have only worked at large companies on tuning a small cog, it will not work

reach out to raj@

Banks are in the business of lending money to home buyers, etc - how is loaning to the US Govt. criminal?

They have a cash crunch due to interest rate rise, and VC slowdown - bad risk management, but everything is above board.

Startup lines of credit should not happen from deposits, too risky an asset class

Expert in Spark, Spark internals or Expert in Query Compilers.

Level variable (2-20 yrs exp), prefer 10+ years

Prophecy.io | Data Engineer | Palo Alto, CA

Expert in writing Spark code and running Spark at scale with substantial production and performance improvement experience. Work with our customers and help product design.

Level variable (2-10 yrs exp), prefer 5+ years

send mail to raj@prophecy.io

We’re a game changing low-code data engineering platform, replacing ETL tools and modernizing the modern data stack.

You know data engineering well, have excellent customer facing experience (consulting works). You will work with our enterprise customers, coordinating with our strong data engineers to make the customers successful. Be their trusted advisor, no upsell responsibility.

email: raj@prophecy.io

send message to raj @ prophecy.io, only primary candidates, thanks!

send mail to raj @ prophecy.io

Comments URL: https://news.ycombinator.com/item?id=30013658

Points: 5

# Comments: 0

When you do visual drag and drop - prophecy is generating high quality code on git that is 100% open source (spark, airflow), you have tests and CI/CD - so you’re visually doing solid data engineering.

You can toggle between code and visual - so if you change the code (some), the visual graph updates - so small edits directly to git don’t break the visual layer.

All visual components are generated from a spec - think a Spark function with a some more info. So the data platform teams will create their own library/framework and roll it out to the wider teams. How it works us that in the visual editor, you start with standard Spark library, but can load visual components for delta, or encryption or data quality.

Our customers are typically fed up of these ETL tools and moving to us. We can also import the ETL formats (AbInitio, Informatica, …) in an automated way (we reverse engineered their formats and created source to source compilers)

Founder here - we’re working to solve this exact problem.