TYPE: full-time

LOCATION: Amsterdam

REMOTE: Optional, Based in the Netherlands is preferred

VISA: No

DESCRIPTION: Polars is the company founded from the Polars OSS project. The company is building a distributed query engine and managed data platform that can run the full polars API with different scaling strategies. (Almost) Our whole tech stack is in Rust and we are building the platform from scratch.

    Rust Software Engineer - Performance: https://hiring.pola.rs/o/performance
    Rust Software Engineer - Query Engines: https://hiring.pola.rs/o/query-engine

CONTACT: hr@polars.tech

However, this is not a Polars issue. Using "fork" can leave ANY MUTEX in the system process invalid (a multi-threaded query engine has plenty of mutexes). It is highly unsafe and has the assumption that none of you libraries in your process hold a lock at that time. That's an assumption that's not PyTorch dataloaders to make.

I highly doubt this. Aside from dataframe generation and series assignment, almost everything in the API surface is different.

Strictness is also not something you can transplant easily. It is checking data types at the IR query planning level before you run the query and being able to resolve schema's independent of the data. In pandas schemas do depend on data within operations and therefore it isn't uncommon that data types change if data gets missing values nor can it check if a correct type is passed to an operation without running the compute.

It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us.

In python, we are able to abstract and provide a stable API.

Cluster configuration is optional if you want this control. Anyhow, this doesn't have much to do with the query API, be it SQL or DataFrame.

However, this should happend during IR-resolving. E.g. the SQL should translate directly to Polars IR, and not LazyFrames. That way we can inspect/resolve all schema's server-side.

It requires a rewrite of our SQL translation in OSS. This should not be too hard, but it is quite some work. Work we eventually get to.

Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's high level DataFrame API.

With regard to implementation, we create stages that run parts of Polars IR (internal representation) on our OSS streaming engine. Those stages run on 1 or many workers create data that will be shuffled in between stages. The scheduler is responsible for creating the distributed query plan and work distribution.

    pc.ComputeContext{
        cpus=4, 
        memory=16
    }

Polars cloud will for the moment only support our DataFrame API. SQL might come later on the roadmap, but since this market is very saturated, we don't feel there is much need there.

At Polars we're building a fast distributed query engine for Polars DataFrames. Our mission is scale DataFrame processing and offer a modern verstatile API to process data fast and easy.

https://hiring.pola.rs/o/database-engineer

Polars is built on the foundation of a vibrant and active open-source community, and we embrace that philosophy in how we run our company. We trust talented people to do their best work without unnecessary constraints. Collaboration is key, but we keep meetings to a minimum to maintain focus. As Polars and Polars Cloud continue to set a new standard in Python data processing, we're looking for like-minded individuals to join us on this journey.

See more: https://pola.rs/careers/#positions

`pc.remote(my_udf, schema)`

Where

`def my_udf() -> DataFrame`

We link the appropiate Python version at cluster startup.

C.blah

The other is that we are only focussing on Polars and honor the Polars semantics and data model. Switching backends via Ibis doesn't honor this, as many architectures have different semantics regarding NaNs, missing data, order of them, decimal arithmetic behavior, regex engines, type upcasting, overflowing, etc.

And lastly, we will ensure it works seamlessly with the Polars landscape, that means that Polars Plugins and IO plugins will also be first class citizens.