OOM on 15k+ concurrent - suspect VIRT?

[pinkforest]

Whilst I was stress testing the UDP stack - I wrote medium about it:
https://missmissm.medium.com/play-ping-pong-with-lunatic-udp-ef557a22a604

Running lunatic target/wasm32-wasi/debug/udp_ping_pong.wasm
thread 'main' panicked at 'Failed to spawn a process: Insufficient resources: System call failed: Cannot allocate memory (os error 12)', /home/foobar/lunatic/rust-lib/src/mailbox.rs:245:25

repro here:

git clone https://github.com/pinkforest/lunatic-udp-examples.git
cd lunatic-udp-examples; cargo run --bin udp_ping_pong

I hit OOM at around 15k flows on debug target - I suspect OOM (suspect virt mem) will not go away if target is release?

    FLOWS            VIRT      RES      SHR       %CPU     %MEM         TIME+   COMMAND
     1.5k             2.2t   155860    33700      15.6      0.5         0:22.77 lunatic  
     2k               16.4t   192336    42276      18.0      0.6         0:31.89 lunatic                          
     3k               23.5t   252900    56708      21.7      0.8         0:49.47 lunatic                        
     4k               31.9t   326300    73988      27.6      1.0         1:13.94 lunatic                                    
     5k               39.2t   388876    88852      28.2      1.2         1:39.30 lunatic   
     10k              81.9t   755756   176452      39.3      2.3         4:41.42 lunatic 

$ ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127764
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100240
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127764
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

lunatic-solutions/lunatic-rs#33
lunatic-solutions/lunatic-rs#17

[bkolobara]

This issue happens because Wasmtime reserves 8 GB of space for each WebAssembly instance (process) heap. At 15k+ processes you basically end up with over 128 TB of memory reserved.

Modern 64bit CPUs actually don't have 64 bit memory instructions, they only usually support 48 bit in practice (because why would you need all that space??), so this ends up at around 256 TB. And because half of the memory space is split between kernel and user space inside of an OS process, once you pass 128 TB the OS will not give you more memory.

Notice that it doesn't really matter if you compile in debug/release mode. This all is usually virtual memory, but you can't go above 128 TB even with virtual memory because of hardware limitations of CPUs and paging systems.

Why does Wasmtime do this? It allows for bound-check elimination. Wasm instances use 32bit memories. If you do start of memory + pointer + other struct offsets and you only work with 32bit sizes you are guarantee to fall into 8 GB. That way you get almost native memory access performance, but at the same time keep the security guarantee of not escaping your heap memory in Wasm. And because it's "just virtual memory" it doesn't matter, at least until you want to have more than 15k of them simultaneously.

One way of fixing this is just to sacrifice performance and pre-allocate smaller heaps in Wasmtime. Once we fill up the heap, allocate a bigger space and copy everything over from previous heap. Wasmtime already supports this and that's how Erlang does it. I couldn't decide what a reasonable starting size would be, so I left the default. This could also be a flag in the VM, Erlang has the +P flag that sets the maximum number of processes (defaults to 32k, 2x lunatic's default).

There is another related issue here: #79. Wasmtime also uses mmap to allocate these heaps and operating systems usually have a maximum of how many mappings they can hold. You will maybe need to adjust some setting in your OS to be able to spawn more processes.

As an interesting side-note, some Intel CPUs have 5-level paging (57 bit) and can address up to 128 PB of virtual memory.

[MaxGraey]

A little clarification: wasmtime reserves 4 GB + 2 GB (for guard pages) but 8GB per instance and only for 64-bit architectures. But all this limits can be configured: https://github.com/bytecodealliance/wasmtime/blob/59a9bd628568cd804839f174b9219ea7d40440d3/docs/contributing-architecture.md#linear-memory