The first version we launched used the exact same approach (MAP_PRIVATE). Later on, we bypassed the file system by using shared memory and using userfaultfd because ultimately the NVMe became the bottleneck (https://codesandbox.io/blog/cloning-microvms-using-userfault... and https://codesandbox.io/blog/how-we-scale-our-microvm-infrast...).

It's important to refresh entropy immediately after clone. Still, there can be code that didn't assume it could be cloned (even though there's always been `fork`, of course). Because of this, we don't live clone across workspaces for unlisted/private sandboxes and limit the use case to dev envs where no secrets are stored.

First, we started cloning VMs using userfaultfd, which allows us to bypass the disk and let children read memory directly from parent VMs [1].

And we also moved to saving memory snapshots compressed. To keep VM boots fast, we need to decompress on the fly as VMs read from the snapshot, so we chunk up snapshots in 4kb-8kb pieces that are zstd compressed [2].

Happy to answer any questions here!

[1]: https://codesandbox.io/blog/cloning-microvms-using-userfault...

[2]: https://codesandbox.io/blog/how-we-scale-our-microvm-infrast...

Comments URL: https://news.ycombinator.com/item?id=42400957

Points: 3

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=40788749

Points: 45

# Comments: 17

Comments URL: https://news.ycombinator.com/item?id=40514085

Points: 4

# Comments: 1

In this case it's because the iframes are loaded/unloaded multiple times, but we also spawn web workers where the same worker is spawned multiple times (for transpiling code in multiple threads, for example). In all those cases we rely on caching so we don't have to download the same worker code more than once.

Comments URL: https://news.ycombinator.com/item?id=37950384

Points: 1

# Comments: 1

Our solution is based on Firecracker, which enables us to "pause" (& clone) a VM at any point in time and resume it later exactly where it left of, within 1.5s. This gives the benefit that you won't have to wait for your environment to spin up when you request one, or when you continue working on one after some inactivity.

However, there's another benefit to that: we can now "preload" development environments. Whenever someone opens a pull request (even from local), we create a VM for it in the background. We run the dev server/LSPs/everything you need, and then pause the VM. Now whenever you want to review that pull request, we resume that environment and you can instantly review the code or check the dev server/preview like a deployment preview.

It also reduces cost. We can pause the VM after 5 minutes of inactivity, and when you come back, we'll resume it so it won't feel like the environment was closed at all. In other solutions you either need to keep a server spinning in the background, or increase the "hibernation timeout" to make sure you don't have the cold boot.

It's kind of like your laptop, if you close it you don't expect it to shut down and boot the whole OS again when you open it. I've written more about how we do the pausing/cloning here (https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...) and here (https://codesandbox.io/blog/cloning-microvms-using-userfault...).

> custom io_uring based driver for the VM block devices? or what do you mean here?

We're using the async io backend that's shipped with Firecracker for our scratch disks.

We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).

It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].

[1]: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...

[2]: https://codesandbox.io/blog/cloning-microvms-using-userfault...