parquet-tools: 0.2.14 -> 0.2.16

[flokli]

Needs #301032 too to build.

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[flokli]

Rebasing this on top of #301220, it fails to build:

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /build/source
plugins: mock-3.12.0
collected 48 items / 3 deselected / 45 selected                                

tests/test_csv.py ......                                                 [ 13%]
tests/test_inspect.py .                                                  [ 15%]
tests/test_parquet.py .....F                                             [ 28%]
tests/test_show.py ........                                              [ 46%]
tests/test_util.py ........................                              [100%]

=================================== FAILURES ===================================
_________________________ TestGetMetaData.test_inspect _________________________

self = <tests.test_parquet.TestGetMetaData object at 0x7fff464bde90>

    def test_inspect(self):
        tests_dir = dirname(__file__)
        test0_parquet_path = path.join(tests_dir, 'test0.parquet')
>       actual = check_output(['parquet-tools', 'inspect', test0_parquet_path]).decode()

tests/test_parquet.py:228: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/subprocess.py:466: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/subprocess.py:548: in run
    with Popen(*popenargs, **kwargs) as process:
/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/subprocess.py:1026: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['parquet-tools', 'inspect', '/build/source/te...>
args = ['parquet-tools', 'inspect', '/build/source/tests/test0.parquet']
executable = b'parquet-tools', preexec_fn = None, close_fds = True
pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0
shell = False, p2cread = -1, p2cwrite = -1, c2pread = 11, c2pwrite = 12
errread = -1, errwrite = -1, restore_signals = True, gid = None, gids = None
uid = None, umask = -1, start_new_session = False, process_group = -1

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session, process_group):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/nix/store/a1s263pmsci9zykm5xcdf7x9rv26w6d5-bash-5.2p26/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and process_group == -1
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        process_group, gid, gids, uid, umask,
                        preexec_fn, _USE_VFORK)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                if err_msg == "noexec:chdir":
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                elif err_msg == "noexec":
                    err_msg = ""
                    err_filename = None
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                if err_filename is not None:
>                   raise child_exception_type(errno_num, err_msg, err_filename)
E                   FileNotFoundError: [Errno 2] No such file or directory: 'parquet-tools'

/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/lib/python3.11/subprocess.py:1953: FileNotFoundError
=============================== warnings summary ===============================
tests/test_util.py::TestS3ParquetFile::test_local_path
tests/test_util.py::TestGetDataframeFromObjs::test_s3_single_file
tests/test_util.py::TestGetDataframeFromObjs::test_s3_double_file
tests/test_util.py::TestGetDataframeFromObjs::test_s3_double_file
tests/test_util.py::TestGetDataframeFromObjs::test_s3_wildcard_file
tests/test_util.py::TestGetDataframeFromObjs::test_s3_wildcard_file
tests/test_util.py::TestGetDataframeFromObjs::test_local_and_s3_files
  /nix/store/c9q2j3q1gr0b9n71197pzrbndhkghyg4-python3.11-halo-0.0.31/lib/python3.11/site-packages/halo/halo.py:497: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead
    self._spinner_thread.setDaemon(True)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_parquet.py::TestGetMetaData::test_inspect - FileNotFoundError: [Errno 2] No such file or directory: 'parquet-tools'
============ 1 failed, 44 passed, 3 deselected, 7 warnings in 1.33s ============
/nix/store/v5lsd029lz5lfhamivbgqyp3zdv94ah2-stdenv-linux/setup: line 1578: pop_var_context: head of shell_variables not a function context
error: build of '/nix/store/djyk2aayhajyc4wxprpzv5b7ilqcaw2s-parquet-tools-0.2.16.drv' on 'ssh://root@box.flokli.de' failed: builder for '/nix/store/djyk2aayhajyc4wxprpzv5b7ilqcaw2s-parquet-tools-0.2.16.drv' failed with exit code 1
error: builder for '/nix/store/djyk2aayhajyc4wxprpzv5b7ilqcaw2s-parquet-tools-0.2.16.drv' failed with exit code 1;
       last 10 log lines:
       > tests/test_util.py::TestGetDataframeFromObjs::test_s3_wildcard_file
       > tests/test_util.py::TestGetDataframeFromObjs::test_local_and_s3_files
       >   /nix/store/c9q2j3q1gr0b9n71197pzrbndhkghyg4-python3.11-halo-0.0.31/lib/python3.11/site-packages/halo/halo.py:497: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead
       >     self._spinner_thread.setDaemon(True)
       >
       > -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
       > =========================== short test summary info ============================
       > FAILED tests/test_parquet.py::TestGetMetaData::test_inspect - FileNotFoundError: [Errno 2] No such file or directory: 'parquet-tools'
       > ============ 1 failed, 44 passed, 3 deselected, 7 warnings in 1.33s ============
       > /nix/store/v5lsd029lz5lfhamivbgqyp3zdv94ah2-stdenv-linux/setup: line 1578: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/djyk2aayhajyc4wxprpzv5b7ilqcaw2s-parquet-tools-0.2.16.drv'.

It seems to rely on being able to invoke parquet-tools from $PATH during the test session now?

[mweinelt]

  preCheck = ''
    export PATH=$out/bin:$PATH
  '';

[flokli]

Thanks, updated! Now also with less disabled tests!

[edef1c]

LGTM