[BUG]: Error loading nvrtc-builtins64_131.dll

[pcgeek86]

Version

1.0.1

Version

13.1

Which installation method(s) does this occur on?

Pip

Describe the bug.

I copied the example code from the cuTile quickstart and tried running it.

https://docs.nvidia.com/cuda/cutile-python/quickstart.html

Project Setup:

uv init .
uv add cuda-tile
uv add numpy
uv add pytest
uv add cup-cuda13x

# Copy the quickstart sample code into main.py

uv run main.py

This results in the following error:

Traceback (most recent call last):
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 731, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 125, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 138, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 53, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_BUILTIN_OPERATION_FAILURE (7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\git\cutile-test\main.py", line 60, in <module>
    test()
  File "C:\git\cutile-test\main.py", line 37, in test
    a = cp.random.uniform(-1, 1, vector_size)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_distributions.py", line 843, in uniform
    return rs.uniform(low, high, size=size, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_generator.py", line 984, in uniform
    rand = self.random_sample(size=size, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_generator.py", line 624, in random_sample
    RandomState._mod1_kernel(out)
  File "cupy\\_core\\_kernel.pyx", line 920, in cupy._core._kernel.ElementwiseKernel.__call__
  File "cupy\\_core\\_kernel.pyx", line 945, in cupy._core._kernel.ElementwiseKernel._get_elementwise_kernel
  File "cupy\\_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "cupy\\_core\\_kernel.pyx", line 728, in cupy._core._kernel._get_elementwise_kernel
  File "cupy\\_core\\_kernel.pyx", line 82, in cupy._core._kernel._get_simple_elementwise_kernel_from_code
  File "cupy\\_core\\core.pyx", line 2377, in cupy._core.core.compile_with_cache
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 536, in _compile_module_with_cache
    return _compile_with_cache_cuda(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 580, in _compile_with_cache_cuda
    base = _preprocess('', options, arch, backend)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 473, in _preprocess
    result, _ = prog.compile(options)
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 750, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: nvrtc: error: failed to open nvrtc-builtins64_131.dll.
  Make sure that nvrtc-builtins64_131.dll is installed correctly.

Minimum reproducible example

Provided above

Relevant log output

Provided above

Full env printout

Doesn't work on Windows 11. I don't have Bash. Bash refers to Windows Subsystem for Linux (WSL), not Bash itself running on Windows.

• Windows 11 Pro
• PNY NVIDIA GeForce RTX 5070 Ti
• NVIDIA Driver 591.74
• NVIDIA CUDA Toolkit 13.1 installed

Output of nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 591.74                 Driver Version: 591.74         CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5070 Ti   WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   50C    P0             36W /  300W |    2989MiB /  16303MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Other/Misc.

No response

Contributing Guidelines

• I agree to follow cuTile Python's contributing guidelines
• I have searched the open bugs and have found no duplicates for this bug report

[haijieg]

@pcgeek86 this is a cupy setup issue. Please refer to cupy installation guide. https://docs.cupy.dev/en/stable/install.html

[ZhangZhiPku]

I encountered a similar issue on a B200 GPU, and I resolved it by reinstalling CUDA Toolkit 13.1.
https://developer.nvidia.com/cuda-downloads

[leofang]

NVRTC is the bare minimum for CuPy to run. In CuPy v13.x and earlier, CuPy assumes that a system CUDA Toolkit (CTK) is installed. However, NVRTC and other CTK components can be installed via pip/conda to the user space as well, and to use them today we have a few options:

1. pip install "cuda-toolkit[nvrtc]==13.1.*" cuda-pathfinder after installing cuTile Python and CuPy, and then call from cuda import pathfinder; pathfinder.load_nvidia_dynamic_lib("nvrtc") after importing CuPy. This will force the OS to load NVRTC from a non-system installation location (aka pip and conda) for CuPy to make use of it.
2. Wait for CuPy v14.0.0 which we start supporting non-system CTK installations out of box (currently targeting end of this month).

[leofang]

Sorry, was juggling between meetings and did not give the full picture here. CuPy has been fully working with conda CTK for many years, so here is the Option 3 (if you use conda/pixi to manage your Python env):

3. Install CuPy from conda-forge via conda install -c conda-forge cupy cuda-version=13.1 and then pip install cuTile Python to the same env.

[pcgeek86]

Conda seems really bloated. There isn't a way to just use the uv package manager? It's used on lots of different Python projects, and does not seem as bloated as Conda. I would much prefer not to install Conda.

[pcgeek86]

I just restarted my development workstation, and it seems like the CUDA Toolkit installation is working fine now. I was able to run the cuTile example code.

Here's the values of the CUDA environment variables:

CUDA_PATH                      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1
CUDA_PATH_V13_1                C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1

And the actual program running:

PS C:\git\cutile-test> uv run main.py       
✓ vector_add_example passed!

I am still not using Conda at all. I simply installed the dependencies with the uv package manager. Here's the corresponding pyproject.toml file, just for reference:

[project]
name = "cutile-test"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "cuda-tile>=1.0.1",
    "cupy-cuda13x>=13.6.0",
    "numpy>=2.4.1",
    "pytest>=9.0.2",
]

[pcgeek86]

Since the print_env.sh script doesn't work for reporting environment details on Windows 11, I added a pull request with a PowerShell script, that writes out info to a text file.

#64

[leofang]

Conda is by-design a self-contained, cross-platform, user-space package manager. It is meant to be used in isolation from most of the system package management, meaning what's already installed on the system is simply ignored, and for that there are extra packages that need to be installed as required dependency for consistency and safe usage. This is a feature not a bug 🙂

FWIW, all NVIDIA CUDA conda packages on the conda-forge channel are officially maintained by NVIDIA.