Add WASI initialization support

[amesgen]

The current unix logic does not work for the new GHC WASM backend for two reasons:

• The filesystem is not necessarily mounted in such a way that /dev/urandom exists (e.g. it is only given access to a confined directory, or the host systen is non-unix, or it is run in a browser context etc.)
• The code fails to link due to the initialization fallback using functions (clock, getpid) not being available.

The current WASI snapshot (as well as future planned versions) do specify an entropy source, random_get which is used to back e.g. getentropy.

In this PR, I added cbits for WASI calling getentropy, but alternative approaches (i.e. merging this into the existing unix logic and e.g. disabling the fallback on WASI; or simply using getentropy directly in the unix logic) are also feasible, happy to change.

I only run two test suites on CI (the initialization one should be most relevant here) as e.g. the splitmix-tests suite does not yet build.

[phadej]

Is it possible to test GHC WASM on the CI. Can I e.g. install GHC-WASM with ghcup?

Until then I consider that experimental and not worth having, it's as simple as: it will bitrot if there is no CI setup.

[amesgen]

Is it possible to test GHC WASM on the CI. Can I e.g. install GHC-WASM with ghcup?

There is a new CI job testing via the WASM backend in this PR already, also see the last sentence in the PR description.

[phadej]

is this pinned bindist? I don't want "moving targets" in CI.

EDIT: I guess FLAVOUR is making it somehow. I'd still prefer something ghcup based, so people can more easily test locally.

[amesgen]

Right now, it would be the latest 9.6 release (see the "flavour" below, see the ghc-wasm-meta README). I will pin ghc-wasm-meta to the latest commit 👍

[amesgen]

ebd790e

[amesgen]

Ah, just noticed your edit; I don't think it is easy ATM to install it via ghcup as you need to install a WASI-specific toolchain in addition to the bindist. Personally, I use ghc-wasm-meta via Nix, but there are also community-provided Docker containers as another alternative to ghc-wasm-meta's setup.sh.

[phadej]

Aren't WASI toolchains available as apt packages? I wish I could get GHC from apt repository instead of ghcup too. I guess every community has their own way to install their stuff nowadays. I honestly don't like that, as one has to (get and) learn N install tools.

[amesgen]

E.g. wasi-sdk seems not to be packaged in Debian/Ubuntu (e.g. via Repology); and the GHC WASM backend needs a wasi-sdk fork anyway.

Pinging @TerrorJack in case you have an estimate whether/when a ghcup- or apt-based installation of the WASM backend could become feasible.

[TerrorJack]

@phadej If you prefer, you can pin to a specific ghc-wasm-meta revision, since each revision pins to specific bindists, and that'll make the installed cross compiler binary-reproducible. @amesgen Would you do that in your next push? Thanks :)

We can't use vanilla wasm32-wasi toolchains from upstreams yet due to certain necessary changes in our fork. I understand the reluctance of using another installation method, but getting ghcup to work with cross toolchains would be a quite non-trivial amount of work that I haven't gotten enough time to implement yet 🙏

[TerrorJack]

Compared to ghcup, it'll be much easier to have an aptitude repo that distributes these tools. I'll look into the possibility this week, though no ETA can be quickly given at this point

[amesgen]

@amesgen Would you do that in your next push? Thanks :)

Already happened in ebd790e 😄 Thanks for providing context!

[Bodigrim]

@phadej how does it look? I understand that it falls somewhat short of "install ghc-wasi via ghcup", but I'm not sure if it's feasible at the moment. Otherwise we have a fairly decent chunk of ecosystem, which can be built and is already built for WASM (e. g., fourmolu), but almost no testing capabilities because they are blocked on splitmix.

FWIW I had a decent experience with this CI job in several projects:

• https://github.com/UnkindPartition/tasty/blob/master/.github/workflows/ci.yaml#L84
• https://github.com/Bodigrim/tasty-bench/blob/master/.github/workflows/other.yml#L88
• https://github.com/haskell/unix/blob/master/.github/workflows/ci-wasm32-wasi.yml

[phadej]

@Bodigrim please understand me. You (as part of CLC) has very strict policy of say changes in base. There are discussions to make GHC change policies stricter too so they don't break code (e.g. -experimental proposal). WASI, on the other hand, is completely experimental now, so you ask me to make splitmix, i.e. "fairly decent chunk of ecosystem" use very experimental GHC features?

Sorry, there's too much dissonance in that.

To be clear. I don't really care if the change is somewhat trivial. it's a principle.

WASI folks can (and in my opinion should) setup wasi.hackage repository (c.f. head.hackage for testing HEAD GHC) and don't bother library maintainers until a lot later, at least when there are proper versioned releases.

---

EDIT: To clarify, the recent GHC stability discussions had made me change my opinions. The ghcup install ghc-wasi <githash> is not enough anymore.

[Bodigrim]

WASI, on the other hand, is completely experimental now, so you ask me to make splitmix, i.e. "fairly decent chunk of ecosystem" use very experimental GHC features?

There might be bugs and missing features, but fundamentally WASI is no more experimental than, say, ARM backend was couple of years ago. It's just another backend for another platform, the more we test it the sooner it gets production ready.

[phadej]

There might be bugs and missing features, but fundamentally WASI is no more experimental than, say, ARM backend was couple of years ago. It's just another backend for another platform, the more we test it the sooner it gets production ready.

ARM backend was in GHC proper, no need to jump through the hoops to test it. I want to test stuff reasonably in CI (thanks for workflow links), but also locally - as debugging on GHA is quite frustrating.

[phadej]

Testing GHCJS was easy. apt-get install ghcjs (thanks to hvr's packaging) and everything worked. haskell-ci was quite easy to adapt to work with that etc. cabal had GHCJS support natively.

And I guess (because I don't know), before GHCJS tooling was made easy, people were using Nix overlays with forks to develop stuff with GHCJS. I don't see why that approach doesn't work with WASI or/and JS backend.

[amesgen]

but also locally

Testing GHCJS was easy. apt-get install ghcjs (thanks to hvr's packaging) and everything worked. haskell-ci was quite easy to adapt to work with that etc. cabal had GHCJS support natively.

In addition to getting the GHC WASM backend via ghc-wasm-meta (both with Nix and without Nix), it is nowadays also possible to install the GHC WASM bindists via ghcup: https://www.haskell.org/ghcup/guide/#ghc-wasm-cross-bindists-experimental (note that one still needs to install wasi-sdk beforehand, as described there, so it is not as smooth as a hypothetical apt-get install ghc-wasm).

And I guess (because I don't know), before GHCJS tooling was made easy, people were using Nix overlays with forks to develop stuff with GHCJS. I don't see why that approach doesn't work with WASI or/and JS backend.

That definitely works (eg a WASM-specific hackage repo is conceivable), but I think it is still desirable to upstream changes if possible (in particular, the corresponding maintainers have to be on board ofc) for less overall complexity, but that indeed doesn't have to happen immediately.

---

How do we want to proceed with this PR? Shall I close it, or do we want keep it open until some time in the future when everything around the WASM backend is more mature (until then, I will only occasionally rebase this PR on top of master), or sth else?

[phadej]

Does GHC WASM works with https://github.com/WebAssembly/wasi-sdk, or do one needs a GHC specific fork?

It would be great if GHC had location / paths of wasi-sdk tools configurable, so one would not need to have them in PATH.

[amesgen]

Does GHC WASM works with https://github.com/WebAssembly/wasi-sdk, or do one needs a GHC specific fork?

The latter: https://gitlab.haskell.org/ghc/wasi-sdk

It would be great if GHC had location / paths of wasi-sdk tools configurable, so one would not need to have them in PATH.

I think that is possible by passing appropriate flags to the bindist configure step, but I don't know the details. cc @TerrorJack

[TerrorJack]

See https://www.haskell.org/ghcup/guide/#cross-support, you do not need wasi-sdk to be in PATH at all. You only need to set some environment variables temporarily when installing the wasm bindists, and later you can use wasm32-wasi-ghc and cabal as usual, as long as you use with-compiler:/with-hc-pkg: in cabal.project.local to tell cabal to use the right installation.

[phadej]

What is wasm32-wasi-cabal then?

[phadej]

By using these GHC variants, you are implicitly signing up to participate in GHC development! If you run into bugs or missing behavior...

I'm sorry, I think I have enough on my plate already.

[TerrorJack]

What is wasm32-wasi-cabal then?

If you use setup.sh from ghc-wasm-meta to set up the entire toolchain including wasm32-wasi-cabal, then setup.sh will generate a tiny wrapper of cabal that really just passes ``--with-compiler=wasm32-wasi-cabaletc, and also sets a distinctCABAL_DIR` to avoid the cabal store from mixing up with host cabal store, in case the `wasm32-wasi-ghc` version number matches host ghc.

Vanilla cabal works fine as long as you tell it to use the right ghc.

[phadej]

Ah, cabal store isn't layered by target architecture, that's unfortunate.

Anyway,

How do we want to proceed with this PR? Shall I close it, or do we want keep it open until some time in the future when everything around the WASM backend is more mature (until then, I will only occasionally rebase this PR on top of master), or sth else?

Up to you. I'm not merging it now. The whole cross-compilation story feels fragile, and I'm not ready to dive deep into it.

I'd say that

eg a WASM-specific hackage repo is conceivable

is your best way forward. There seems to be tools for that https://github.com/andreabedini/foliage-template

[phadej]

Please understand my stance. I have burned multiple times by this pattern:

• someone contributes something (e.g. build.nix file)
• it bitrots
• people complain that it doesn't work, I have no idea how to fix it, original contributor is busy with something else
• everyone is sad.

By having your own repository we can avoid that. People would report issues directly to you.

[amesgen]

@phadej Thanks, that makes a lot of sense 👍 I will then convert this PR to draft for now and occasionally rebase it, and it can be reconsidered at some point in the future.