WoolyAI Private Multi-agent Inference Stack for DGX Spark is built to enable groups within enterprises to set up their own private, low-cost inference stacks for business agentic workflow apps. It was built with the forward-looking vision that companies will need their own private, low-cost inference setup and that single-model-based inference is not sufficient for complex enterprise workflow agentic apps. These workflows need multiple models of different specializations (hence sizes) for different steps in the workflows. Having dedicated multi-GPU inference stacks for each model is very cost-prohibitive.

First benchmark: We first ran LlamaBenchy tests on our inference server on a 2 DGX Spark cluster across 3 models with no quantization and no speculative decoding. Using speculative decoding would result in even higher numbers.

MODEL LOAD-PREFILL-SYSTEM-TOK/S DECODE-SYSTEM-TOK/S DECODE-TOK/S-PER-REQUEST

DeepSeek V4 Flash C1 1,518.91 21.15 21.15

DeepSeek V4 Flash C4 1,533.15 55.99 14.00

Gemma 4 26B A4B C1 4,579.73 30.22 30.22

Gemma 4 26B A4B C4 4,702.16 63.75 15.94

Nemotron 3 Nano Omni 30B NVFP4 C1 2,607.86 39.42 39.42

Nemotron 3 Nano Omni 30B NVFP4 C4 2,588.87 90.83 22.71

Second Benchmark: One endpoint, three different model-controlled activations. The scheduler batches each burst, coordinates both ranks, and changes the resident model only at a safe boundary

MODEL PREFILL-TOK/S DECODE-TOK/S Model-Activation-Wait

DeepSeek V4 Flash 4,154.34 49.30 16s

Gemma 4 26B A4B 18 4,781.44 64.67 6s

Nemotron 3 Nano Omni 30B NVFP4 2,395.59 93.31 2s

We think we can improve these numbers by 20% with more optimization. Please share your feedback. https://woolyai.com/ai-compute-software/dgx-spark-inference-stack/

Comments URL: https://news.ycombinator.com/item?id=49014048

Points: 4

# Comments: 0

We are looking for teams to give it a try.

More details to get a trial license - https://www.woolyai.com.

Comments URL: https://news.ycombinator.com/item?id=47431664

Points: 2

# Comments: 0

We’ve come up with a different model, similar to how operating systems schedule tasks. Instead of carving up the GPU, we run multiple ML jobs inside a single shared GPU context and schedule their kernels directly. No slices, no preemption windows — just a deterministic, SLA-style kernel scheduler deciding which job’s kernels run when.

This results in the GPU behaving more like an always-on compute fabric rather than a dedicated device. SMs stay busy, memory stays warm, and high-priority jobs still get predictable latency. More details at https://woolyai.com/blog/a-new-approach-to-gpu-kernel-scheduling-for-higher-utilization/ Check out our technology at https://www.woolyai.com.

Comments URL: https://news.ycombinator.com/item?id=46195959

Points: 1

# Comments: 0

Trial is now to triple your GPU Utilization in ML Development.

Comments URL: https://news.ycombinator.com/item?id=46154601

Points: 1

# Comments: 0

https://woolyai.com/signup/

- Higher GPU utilization & lower cost Pack many jobs per GPU with WoolyAI’s server-side scheduler, VRAM deduplication, and SLO-aware controls. - GPU portability Run the same ML container on NVIDIA and AMD backends—no code changes. - Hardware flexibility Develop/run on CPU-only machines; execute kernels on your remote GPU pool.

Comments URL: https://news.ycombinator.com/item?id=45740390

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45327026

Points: 1

# Comments: 0

https://youtu.be/f62s2ORe9H8

Comments URL: https://news.ycombinator.com/item?id=45174863

Points: 1

# Comments: 1

Why this matters

Higher capacity: Share the base model in VRAM; add more adapters or vertical inference stacks per GPU without increasing memory usage.

Isolation & control: Each stack is its own process with independent batching and SLA-aware scheduling.

While vLLM supports multiple adapters on a single vLLM process, many teams need predictable per-adapter SLAs—this is where running independent stacks with a shared base model in VRAM can enable doing it all on the same GPU.

The demo uses LoRA inference using Pytorch, but the same applies when using vLLM. If you’re scaling LoRA inference across business units or model variants and need predictable latency without overprovisioning GPUs, I’d love your feedback. Comment or DM to chat.

Comments URL: https://news.ycombinator.com/item?id=45119743

Points: 7

# Comments: 1

There are some interesting insights just looking at these numbers.

Environment Details Wooly Client: Linux non-GPU container running PyTorch scripts for all ten models Models were downloaded using Hugging Face Transformers library from vendor-specific repositories. Each model was executed 20 times using the same script to collect average Wooly Credits for both CPU and VRAM usage. Models Tested Llama-3.2-1B Llama-3.2-1B-Instruct Llama-3.2-3B Llama-3.2-3B-Instruct Mistral-7B-Instruct Falcon3-7B-Instruct Llama-3.1-8B-Instruct Llama-3.1-8B Dolly-v2-12B Llama-2-13B-Chat-HF Pytorch Script

from transformers import AutoTokenizer, AutoModelForCausalLM import torch torch.manual_seed(100000) # Model name or path model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct" # Load tokenizer and model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda") # Input text input_text = "What is the capital of United States of America" # Tokenize input text inputs = tokenizer(input_text, return_tensors="pt").to(model.device) # Decode and print output for z in range (1, 10): outputs = model.generate(*inputs, max_length=100) generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) print(generated_text) GPU Core and Memory utilization Metrics

Llama-3.2-1B Core Wooly Credits Used - 46000 VRAM Wooly Credits Used - 31072 Llama-3.2-1B-Instruct Core Wooly Credits Used - 94868 VRAM Wooly Credits Used - 60964 Llama-3.2-3B Core Wooly Credits Used - 195936 VRAM Wooly Credits Used - 84715 Llama-3.2-3B-Instruct Core Wooly Credits Used - 502448 VRAM Wooly Credits Used - 258125 Mistral-7b-instr Core Wooly Credits Used - 525689 VRAM Wooly Credits Used - 397181 Falcon3-7B-Instruc Core Wooly Credits Used - 136094 VRAM Wooly Credits Used - 26528 Llama-3.1-8B Core Wooly Credits Used - 283458 VRAM Wooly Credits Used - 167515 Llama-3.1-8B-Instruct Core Wooly Credits Used - 574872 VRAM Wooly Credits Used - 403934 Dolly-v2-12b Core Wooly Credits Used - 767108 VRAM Wooly Credits Used - 342877 Llama-2-13b-chat-hf Core Wooly Credits Used - 313809 VRAM Wooly Credits Used -120067

Comments URL: https://news.ycombinator.com/item?id=43345695

Points: 1

# Comments: 1

We built a GPU Cloud service(WoolyAi Acceleration Service) using this abstraction layer with "Actual GPU Resources Used" billing, NOT "GPU Time Used" billing.

Looking forward to getting your feedback and comments.

Comments URL: https://news.ycombinator.com/item?id=43294543

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=20471657

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=17509471

Points: 1

# Comments: 0