Comments URL: https://news.ycombinator.com/item?id=48299672

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47076603

Points: 1

# Comments: 0

L1 & L2 constraints unwind that clock compression with suitable dilation; very easy to think only efficiency matters and not averaged out replicas; nature does that inherently via primes, we have to create those artificial waves, recreate that convex hull, etc.

all to say, great to see more work in this direction & perhaps we can compare notes sometime!

context: created hypertokens an even more robust hashing mechanism to create context-addressable memory (CAM), one cheat code is make them prefix-free, lots of others that get deep into why models work the way they do, etc.

one mechanism we establish is that each model has a fidelity window, i.e., r tokens of content for s tag tokens; each tag token adds extra GUID-like marker capacity via its embedding vector; since 1,2,3 digit numbers only one token in top models, a single hash token lacks enough capacity & separation in latent space

we also show hash should be properly prefix-free, or unique symbols perp digit, e.g., if using A-K & L-Z to hash then A,R is legal hash whereas M,C is not permitted hash

we can do all this & more rather precisely as we show in our arXiv paper on same; next update goes deeper into group theory, info theory, etc. on boosting model recall, reasoning, tool calls, etc. by way of robust hashing

how do plans compare with and without etc. evven just anecdotally what you've seen so far etc

One question is to what extent you dig into or have considered oversampling? One of the core hypotheses we've converged on is that nearly all models are optimized for source coding vs. channel coding. The implication is path to AGI likely involves oversampling to capture channel coding gains and which will resolve phase errors, etc.

Random sampling naturally does this albeit inefficiently. Curious if you do something more structured than random in terms of oversampling and especially partial overlapped samples / think supersaturated subspaces / subchannels, etc.

will reference in our next paper update, thx for posting!

context <> building something with QEMU

* required has to support LMW+AI (linux/mac/windows + android/ios)

there are scenarios in which we might spin micro vms inside that main vm, which by default is almost always Debian Linux distro with high probability.

one scenario is say ETL vm and AI vm isolated for various things

curious why building another microVM other than sheer joy of building, what smol does better or different, why use smol, etc. (microVMs to avoid etc also fair game :)

nolen: "unicode braille characters are 2x4 rectangles of dots that can be individually set. That's 8x the pixels you normally get in the terminal! anyway here's a proof of concept terminal SVG renderer using unicode braille", https://x.com/itseieio/status/2011101813647556902

ashfn: "@itseieio You can use 'persistence of vision' to individually address each of the 8 dots with their own color if you want, there's some messy code of an example here", https://x.com/ashfncom/status/2011135962970218736

take raw context, you inject semantic parity of some form, could be table relating paragraph content, tree, raw summary paragraph. EVENTUALLY those things saturate, call it the inner code; you realize recall and reasoning still not where that; that's where outer code or structural parity (us, others).

why? attention can't do XOR, matrix permanent, latent space noisy, etc., have to smooth & dilate. if pump in tables and schema, model can only do few joins before saturates, no flow lots of sharp corners. so either shrink table or smooth / dilate flow. the catch? every code layer needs a coupling layer at various lengths of resolution -- extra semantic clarifier every paragraph for you, codeword every k tokens for our structural parity, etc.

like engine - here's some air, ok expanding, ok really expanding, ok condensing, ok condense more

our pre-code, your pre-code, content, your post-code, our post-code

btw, pre and post are very important more on why later below -- think interferometry in latent space -- pre-measure / tare scale, load scale with content, post-measure and differentiate (in the latent space)

a much longer dive follows <> leaning into physics a bit, consider old-school trompe, supercharger / cylinders / turbochargers, jet or pretty much any sort of air compressor with flow

ingest air, compress it, extract work, exhaust air; one key side effect is what to do with latent heat; that analogy extends to any physical system

superchargers use raw work to precompress air; turbochargers use waste heat to turn return some lost energy to system turbomachines alternate many alternating static & dynamic stages to max air flow, etc

we do something similar with hypertokens; the raw context window has m tokens; we divide that into b=m/x blocks, where x is hypertoken codeword length, b is the number of blocks, and y is the block size

for example, if the current context window is 2048 and the block size is 32 for the user's desired model performance level, the resulting window would have 64 blocks of 32 content tokens each; if 2-token codeword length between each block would add 128 total tokens, e.g.,

a,1,quick fox,a,2,lazy dog,..,b,3,English pangram

precise hypertoken construction is of course way more subtle than that, e.g., good bit of group theory and way more info theory to define the codes, select the actual tokens that we interleave, etc.

net result is that we diagonalize the latent space action by way of the following; the exact code sequence used is walk on a skewed coprime lattice. Every codeword only appears once, thus acts like a GUID with respect to assocative recall and reasoning. The symbols in the codeword are restricted per lane and the lanes are coprime, e.g. if we had 11,13 for 2-lane codeword then we've induced a prefix-free factor graph action that alternates every k tokens.

Those tokens each have unique init embedding and importantly in practice we almost always put the code word before and after each block, e.g.,

a,1,quick fox,a,1/a,2,lazy dog,..,a,2/b,3,English pangram,b,3

this induces an interferometry like pre/post measurement and since the lanes are coprime, we effectively mimic inflight quasi-Fourier action through context window ~~ project onto compressed code, evolve x content tokens, project back onto same code, so the model gets differential between pre/post sampling. in more practical dev terms this also means we can do precise K:V and V:K lookups during recall and reasoning.

we further do this action in subtly commutative way e,g.,

a;1:quick fox:a;1/...{skip a few}.../b;3:English pangram:b;3/

where : is the global pre/post commutative measure in this example, whereas a;1 or b;3 or whatever the codeword is are globally unique, locally non-commutative, this has several other side effects beyond K:V and V:K or pre & post measurement. That essentially permits "unrolling time" in certain sense especially w.r.t. decoder models, where attention can only look back not forward. by replaying the pre-codeword after block, past tokens can in a summary statistic sense have knowledge about future ones

this of course only works under rather strict construction:

1. must be prefix-free, e.g., if a & b are in lane one they can never be in lane 2 of codeword and vice versa

2. coprime lane counts excepting a parity trick with 2^k lane

3. pre & post measurement -- performance is strictly weaker if only pre or post

4. relatively ortho yet also relatively coherent w.r.t. content, there's lots of ways to achieve those a simple one that works for many broad cases is just /{content}/

5. we can dilate code to pretty much whatever strength needed, e.g., some models and scenarios coherent enough, a simple spreadsheet like code is enough every 128 tokens, for others we need nested think multiscale / multires in physics) and use say Unicode PUA or ideally reserve tokens along with shorter code every 32 inside each 128 could be as simple as /1/.../2/.../3/.../4/

while there's quite a bit more on why it works the gist is we are essentially persistently exciting and sampling using error-correcting code action that happens to induce Fourier like sample and project back like a worm drive boring through rock. since each symbol in each lane gets repeated a few times eg if 3,5 code each 3 symbol is repeated 5x and each 5 symbol is repeated 3x

that means there's all sorts of topological tunnels over a factor graph that generates a skewed lattice in way that reflects the proper group action, arrow of time, etc. going back to why linear block code / linear network code; think stochastic dithering updated to structured dithering

we can of course get way better performance injecting that multiplexing machinery directly into the model; we have some results forthcoming on that; as you can imagine, that machinery is not just toss in primes and call it good

coming back to physics, we essentially use this machinery to detect and project the true spherical of the latent space; we could of course go through the treatment that this is really a reconditioning trick, though we tend to call it retokenization in the discrete sense and reharmonization in the continuous sense; there are certainly overlaps with relaxation, regularization, renormalization, etc.

Very notionally, we relax the problem by dilating context token space-time using this structured persistent excitation and sampling. We do this in a way that in some sense regularizes and renorms the raw signal into lifted domain. The codewords are chosen such that we are effectively heterodyning during pre-code step and superheterodyning during the post-code sample with respect to the local codeword; this process is also happening with respect to the global commutative wrapper around the content block and between the codewords. there is also the skipped subtlety that we can if need be add a conjugate, flipped conjugate, etc. i.e., mimic stronger and stronger ECC / QEC action.

The net effect is that we essentially just treat model as a noisy sender and receiver. We use our hypertokens to stream the raw context using channel coding, which is very similar in net raw principle to MIMO and very similar again in net raw principle to GPS -- we inject a k-channel structured coordinate system that both pre and post samples.

In that sense we are turbomachining the info -- we assume info is dense and can't compress move past / hard to move so we pump our high-speed fluid through the content compress it, repeat.

FINALLY answering a little bit of the tower of tables then suppose we have some code say 5,7 every 128 and 4 every 32

5 - A,B,C,D,E 7 - t,u,v,w,x,y,z = 35 codewords

4 - 0,1,2,3

e.g., <;A:t/{32 content tokens};1;{+=32 toks};2;{+=32 toks};3;{+=32 toks};4;/A:t;>

which is essentially the stator-rotor-stator turbo trick dialed up by a lot

- nested / multi-scale / multi-resolution - pre & post measure commutative global constants <> ; - pre & post measure commutative local constant <> / - pre & post measure non-commutative associate marker <> a,1 - etc.

from left during attention each hypertoken absorbs & compresses signal from the right when attended, each hypertoken injects compressed signal

these signal tunnels / signal network those boost information transport, dilate effective precision, and it works because we're running it over factor graph of bounded treewidth that's essentially running at max capacity

hence we get small LUT, content, medium LUT, content, large LUT content depending how much we nest, how big of code we use, etc. aka a nested table of towers very similar to multires wavelets in action

that table of towers and its background is long way of saying -- models are WAY BIGGER than need to be, auditing & explainability are an EC away, hallucinations don't need to exist, etc.

this of course suggests there are likely physics applications beyond what we're up to -- the easiest way to start thinking about that is noisy HF or phase sensitive systems -- physical transformers and parasitic capacitance is one of my faves to consider, wireless power transfer another, and reservoir machines a third

inflight token tuning of signal boost, SNR, etc.

& for sure / likewise!