Python Path in R is missing /site-packages/ for Ubuntu 24.04 and Python 3.12

[glatterf42]

Description

This is related to #10636 and possibly #10781.
This weekend, our CI started failing on Ubuntu with this error message:

 Error in IRkernel::installspec() : 
  jupyter-client has to be installed but “jupyter kernelspec --version” exited with code 1.
Execution halted

This is the same error message as we encountered the first time that ubuntu-latest was upgraded to Ubuntu 24.04. Back then, I opened an issue with IRkernel and r-lib, where I summarized the issue in detail and what I tried to work around it. In essence, the issue seems to be that the Python path inside of our Rscript step is not correct on ubuntu-latest-py3.12: the .../site-packages/ directory is missing. If I run jupyter kernelspec --version immediately before starting Rscript, the Python path is fine and the command exits without errors.
I am still not sure what's causing this behavior. Ubuntu 24.04 removed the pre-installed R version, but we're using the latest version of the r-lib/setup-r action, anyway. In the meantime, we started supporting Python 3.13, though, and the same error does not appear there (or on Python 3.11, for that matter; unfortunately, I don't remember if that was the case in October, too). You can see this in the CI of another of our repos: this issue only seems to affect Ubuntu 24.04 and Python 3.12.

With this information, we can already find workarounds (e.g. only use Ubuntu 22.04 for Python 3.12), but I still think this bug should be reported and resolved, if you can. Thanks for your efforts :)

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Version: 20241201.1.0

Link (again): https://github.com/iiasa/message_ix/actions/runs/12229324988

Is it regression?

No, the last successful run used Image: ubuntu-22.04 Version: 20241201.1.0.

Expected behavior

The Python path should be the same inside of Rscript as well as outside so that jupyter is recognized properly.

Actual behavior

.../site-packages/ is missing from the path, so jupyter is not found and the command exits with an error.

Repro steps

Please find our workflow file here: https://github.com/iiasa/message_ix/blob/main/.github/workflows/pytest.yaml
If you run the steps in the tutorials workflow, the step Install R dependencies and tutorial requirements will fail with the error message above using the Ubuntu 24.04 version given above and Python 3.12.

[RaviAkshintala]

Hi @glatterf42 Thank you for bringing this issue to our attention. We are looking into this issue and will update you on this issue after investigating.

[RaviAkshintala]

Hi @glatterf42 Could you please try the workaround below, I hope it will work for this issue. Thanks.
https://github.com/RaviAkshintala/message_ix/actions/runs/12272921313/workflow#L60

From my side, Install system dependencies for R packages step is passed with the mentioned workaround.
https://github.com/RaviAkshintala/message_ix/actions/runs/12272921313/job/34242660447

[glatterf42]

Hi @RaviAkshintala, thanks for providing this workaround! I must say that I'm not sure how this made your workflow pass, but I'm glad there is a lead :)
Unfortunately, I wasn't able to reproduce it. This error actually affects two of our repositories: message_ix and ixmp. On message_ix, this is just affecting the tutorials, for which we can almost switch to Python 3.13 to run them. Most likely, the only obstacles are other in-house packages not supporting it yet. So in that repo, I would prefer to implement this solution once it becomes available.
In ixmp, I started a PR implementing your workaround. The workflow file is very similar to message_ix, but something must be different since the run there is still failing: https://github.com/iiasa/ixmp/actions/runs/12294870217?pr=553
Sorry for shifting the target like this, but do you have any idea why your workaround doesn't work in ixmp?

[RaviAkshintala]

@glatterf42 Could you please try to add a step to install jupyter-client before running IRkernel::installspec().

Example:

- name: Install R dependencies and tutorial requirements
      run: |
        install.packages(c("remotes", "Rcpp"))
        remotes::install_cran(
          c("IRkernel", "reticulate"),
          dependencies = TRUE,
        )
        pip install jupyter
        Rscript -e "IRkernel::installspec()"
      shell: Rscript {0}

[glatterf42]

Unfortunately, this does not work at all, it gives me this error in the Install R dependencies and tutorial requirements step:

Error: Error: unexpected symbol in "pip install"
Execution halted
Error: Process completed with exit code 1.

[RaviAkshintala]

@glatterf42 I tried with the below steps , It is working for me. Could you please try from your side.

  install_dependencies:
    runs-on: ubuntu-24.04

    steps:
    - name: Set up R
      uses: r-lib/actions/setup-r@v2

    - name: Install R dependencies
      run: |
        R -e "install.packages(c('remotes', 'Rcpp'))"
        R -e "remotes::install_cran(c('IRkernel', 'reticulate'), dependencies = TRUE)"
    - name: Install Jupyter via pip
      run: |
        pip install jupyter
    - name: Configure IRkernel
      run: |
        Rscript -e "IRkernel::installspec()"

[glatterf42]

Thanks for your many suggestions :)
I'm sorry but even this latest one does not work. It now gives the original error again, but in the Configure IRkernel step:

 Error in IRkernel::installspec() : 
  jupyter-client has to be installed but “jupyter kernelspec --version” exited with code 1.
Execution halted