Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v 0.3 fails to install #31

Open
ausstein opened this issue Dec 10, 2024 · 5 comments
Open

v 0.3 fails to install #31

ausstein opened this issue Dec 10, 2024 · 5 comments
Assignees
@ion-g-ion
Labels
bug Something isn't working

Comments

@ausstein
Copy link
Contributor

Describe the bug
Installing the newest version gives errors

I am in a slurm environment without sudo rights

first I got an error since the /tmp folder only has 5GB and the installation process needed more, so i specified a custom tmp folder in my home directory. then I tried again and got another error. the full log is attached at the bottom. the relevant message seems to be

from /home/ausstein/github/torchTT/cpp/cpp_ext.cpp:1:
/usr/include/features-time64.h:20:10: fatal error: bits/wordsize.h: No such file or directory

for now I went back to 886ef89 which I had installed previously and still works fine

To Reproduce
Steps to reproduce the behavior:

  1. create clean environment conda create -n torchTT pytorch pytest numpy opt_einsum scipy ninja numba tqdm matplotlib snakeviz pytorch-cuda=12.4 -c pytorch -c nvidia

  2. conda activate torchTT

  3. git clone https://github.com/ion-g-ion/torchTT

  4. TMPDIR=~/my_tmp pip install ./torchTT/

  5. See error

Expected behavior
Installation runs smoothly

Desktop

  • OS: Ubuntu
  • Version: 22.04.4 LTS

Error Log
building 'torchttcpp' extension
creating /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp
Emitting ninja build file /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o.d -pthread -B /home/ausstein/miniconda3/envs/torchHEOM/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -Icpp -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/TH -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/THC -I/home/ausstein/miniconda3/envs/torchHEOM/include/python3.11 -c -c /home/ausstein/github/torchTT/cpp/cpp_ext.cpp -o /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o -std=c++17 -Wno-c++11-narrowing -w -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=torchttcpp -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o
c++ -MMD -MF /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o.d -pthread -B /home/ausstein/miniconda3/envs/torchHEOM/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -Icpp -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/TH -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/THC -I/home/ausstein/miniconda3/envs/torchHEOM/include/python3.11 -c -c /home/ausstein/github/torchTT/cpp/cpp_ext.cpp -o /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o -std=c++17 -Wno-c++11-narrowing -w -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=torchttcpp -D_GLIBCXX_USE_CXX11_ABI=0
In file included from /usr/include/features.h:392,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/x86_64-pc-linux-gnu/bits/c++config.h:679,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/cstddef:49,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/c10/core/DeviceType.h:10,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/c10/core/Device.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/core/TensorBody.h:11,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/core/Tensor.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/Tensor.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/function_hook.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/cpp_hook.h:2,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/variable.h:6,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/autograd.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/extension.h:5,
from /home/ausstein/github/torchTT/cpp/define.h:3,
from /home/ausstein/github/torchTT/cpp/full.h:1,
from /home/ausstein/github/torchTT/cpp/cpp_ext.cpp:1:
/usr/include/features-time64.h:20:10: fatal error: bits/wordsize.h: No such file or directory
20 | #include <bits/wordsize.h>
| ^~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
:60: UserWarning:
C++ implementation not available. Falling back to pure Python.

  Error: Error compiling objects for extension
  Traceback (most recent call last):
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 409, in _build_with_temp_dir
      result_basename = _file_with_extension(tmp_dist_dir, result_extension)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 128, in _file_with_extension
      raise ValueError(
  ValueError: No distribution was found. Ensure that `setup.py` is not empty and that it calls `setup()`.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torchTT
Failed to build torchTT
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchTT)
@MazenAli
Copy link

I think this is due to a lack of pre-installed build tools, had the same issue. Here is a docker setup that works:

FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
    build-essential \
    cmake \
    g++ \
    ninja-build \
    libopenblas-dev \
    liblapack-dev \
    libfftw3-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt ./

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt

RUN rm requirements.txt

@ausstein
Copy link
Contributor Author

@MazenAli as i said a previous version just a few commits back compiles and runs perfectly, are any of these build tools new requirements for v0.3?

@MazenAli
Copy link

I can't say for sure without checking the source code for changes. torchtt uses some c++ compiled routines for dmrg, or defaults to a python implementation. In the previous github version, I would get a warning that the implementation defaults to python. In this one, I don't. Which tells me that the github clone by default installed the python implementation whereas the pypi version by default tries to compile the c++ version.

Python dependency management, unfortunately, can't handle non-python build tools. The author could consider defaulting back to the python build with pypi for ease of install.

For you, the simplest workaround would be to use a docker container. Or use the old version ofc.

@ausstein
Copy link
Contributor Author

ausstein commented Dec 10, 2024 via email

@ion-g-ion ion-g-ion self-assigned this Dec 11, 2024
@ion-g-ion ion-g-ion added the bug Something isn't working label Dec 11, 2024
ion-g-ion pushed a commit that referenced this issue Dec 11, 2024
issue #31 change to setup.py
1b27da6
@ion-g-ion
Copy link
Owner

I did some changes to the setup.py to address this issue. Now it should work fine even if the c++ extensions cannot be compiled.

BR,
Ion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ion-g-ion ion-g-ion

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MazenAli @ion-g-ion @ausstein