Feature Request: Use single .merlin file even when subprojects have dune-project or dune-workspace

[jonahbeckford]

Summary

Merlin does not have a way to express using a single .merlin file at the root of a unified project. That means vendored subprojects that have their own dune-project (or dune-workspace) spawn their own dune ocaml-merlin processes and consequently mess up the Merlin configuration, even though there is a single build directory. I suspect this is an issue with opam-monorepo style projects as well.

Reproduction

I am converting some of my build systems (MlFront/DkCoder) to create .merlin files directly. As I transition I have a mix of dune-project and .merlin files. Here is the pertinent structure for one of the projects:

.merlin
#s/
  findlib.conf
src/
  SonicScout_Setup/
    AndroidStudio.ml
us/SonicScoutBackend/
  dune-project
  src/
    SonicScout_Std/
      Aws_manager.ml

Then when I do (PowerShell syntax):

get-content us\SonicScoutBackend\src\SonicScout_Std\Aws_manager.ml | `
.ci\cmake\bin\cmake -E env `
  OCAMLFIND_CONF=Y:\source\scoutapps\#s\findlib.conf `
  --modify PATH=path_list_prepend:Y:/source/dksdk-coder/build/d/o/two/bin `
  -- `
  Y:\source\dksdk-coder\build\d\o\one\.opam-switch\build\merlin.4.14-414\_build\install\default\bin\ocamlmerlin.exe `
  single dump-configuration `
    -filename us\SonicScoutBackend\src\SonicScout_Std\Aws_manager.ml `
  | jq '.value.merlin.failures, (.value.merlin.build_path | length)'

I get a failure:

[
  "No config found for file us/SonicScoutBackend/src/SonicScout_Std/Aws_manager.ml. Try calling 'dune build'."
]
0

When I remove the us/SonicScoutBackend/dune-project I get success (no failures and many build paths):

[]
227

Mitigation

I can't remove the dune-project. But I'll try to submit a PR that will be somewhat hacky since I don't know a clean way to solve this: #1870

[voodoos]

As you said, the solution you propose feels very ad-hoc and hackish.

I know it would take much longer to happen, but I feel that it would be more valuable to actually take the time to adapt Dune (and Merlin) to correctly handle your configuration needs (fixes to some of the issues you encountered like the one with copy files would benefit to a lot of users).

Meanwhile, I will try to think about a less ad-hoc option. Maybe a flag to specify the list of allowed configuration readers ?

[jonahbeckford]

I know it would take much longer to happen, but I feel that it would be more valuable to actually take the time to adapt Dune (and Merlin) to correctly handle your configuration needs (fixes to some of the issues you encountered like the one with copy files would benefit to a lot of users).

I thought so at first. I had pushed back a bit when I was initially told my use case doesn't fit with Dune. But now I realize some deeply embedded Dune designs like the directory-centricity of Dune (rather than the target-centricity of most modern build systems) makes it borderline impossible to do a critical use case I need (distributing bytecode). The directory-centric problem is explained in https://gitlab.com/diskuv/registries/public-packages#creating-portable-cmi-files. TLDR:

• A xyz.cmi produced by a Dune project from a source file <duneproject>/src/xyz.ml will be incompatible with a xyz.cmi produced by the identical source file <duneproject>/vendor/third-party/xyz.ml or the source file <duneproject>/duniverse/third-party/xyz.ml
• In other words, composing Dune projects can't be done correctly without also replicating the directory structure

I'm fine with keeping Dune as a build system for interoperability when the use case matches Dune's primary use case (creating executables, especially native executables).

Meanwhile, I will try to think about a less ad-hoc option. Maybe a flag to specify the list of allowed configuration readers ?

I think that is okay for the medium term. Or perhaps a narrower flag to disable only the Dune configuration reader.

FYI: At the moment the Dune configuration reader does not respect Dune configuration (ex. it doesn't check if a (dune) file contains a (dirs) to stop scanning subdirectories). I don't think there is a perfect solution.