python -m jittor.test.test_cudnn_op测试不通过

Pluto · 2024 年3 月 29 日 09:05

错误信息、
按照官方推荐的安装步骤进行并进行测试，测试不通过；core和example的测试已通过

Jittor 运行的完整 log（建议复制粘贴运行输出，如果可能请不要使用截图）、

[i 0329 17:11:12.446000 68 compiler.py:956] Jittor(1.3.8.5) src: f:\jittor_learning\pythonproject1\.venv\lib\site-packages\jittor
[i 0329 17:11:12.471000 68 compiler.py:957] cl at C:\Users\mulin\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe(19.29.30133)
[i 0329 17:11:12.472000 68 compiler.py:958] cache_path: C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\IntelRCoreTMi5x0d\default
[i 0329 17:11:12.475000 68 install_cuda.py:93] cuda_driver_version: [12, 4, 0]
[i 0329 17:11:12.500000 68 __init__.py:411] Found C:\Users\mulin\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe(11.2.67) at C:\Users\mulin\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe.
[i 0329 17:11:12.596000 68 __init__.py:411] Found addr2line(2.28) at D:\mingw\bin\addr2line.EXE.
[i 0329 17:11:12.636000 68 compiler.py:1011] cuda key:cu11.2.67
[i 0329 17:11:12.638000 68 __init__.py:227] Total mem: 15.92GB, using 5 procs for compiling.
[i 0329 17:11:13.770000 68 jit_compiler.cc:28] Load cc_path: C:\Users\mulin\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe
[i 0329 17:11:13.777000 68 init.cc:62] Found cuda archs: [61,]
[w 0329 17:11:14.032000 68 compile_extern.py:203] CUDA related path found in LD_LIBRARY_PATH or PATH(['', 'C', '\\Users\\mulin\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\mkl\\
dnnl_win_2.2.0_cpu_vcomp\\bin', '', 'C', '\\Users\\mulin\\.cache\\jittor\\mkl\\dnnl_win_2.2.0_cpu_vcomp\\lib', '', 'C', '\\Users\\mulin\\.cache\\jittor\\jt1.3.8\\cl\\py3.12.0\\Windows-10-10.x9d\\IntelRCoreTMi5x0d\\default', '', 
'C', '\\Users\\mulin\\.cache\\jittor\\jt1.3.8\\cl\\py3.12.0\\Windows-10-10.x9d\\IntelRCoreTMi5x0d\\default\\cu11.2.67', '', 'C', '\\Users\\mulin\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\bin', '', 'C', '\\Users\\mulin\\.cach
e\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib\\x64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\win10_kits\\lib\\ucrt\\x64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\win10_kits\\lib\\um\\x64', '', 'C', '\\Users\\mulin\
\.cache\\jittor\\msvc\\VC\\lib', '', 'C', '\\Users\\mulin\\AppData\\Local\\Programs\\Python\\Python312\\libs', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\VC\\_\\_\\_\\_\\_\\bin', 'F', '\\jittor_learning\\pythonProject1\\.venv/S
cripts', 'D', '\\Anaconda3', 'D', '\\Anaconda3\\Library\\mingw-w64\\bin', 'D', '\\Anaconda3\\Library\\usr\\bin', 'D', '\\Anaconda3\\Library\\bin', 'D', '\\Anaconda3\\Scripts', 'D', '\\Anaconda3\\bin', 'D', '\\Anaconda3\\condabin
', 'C', '\\Program Files\\NVIDIA\\CUDNN\\v9.0\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\bin', 'C', '\\Program Files\\NVIDIA G
PU Computing Toolkit\\CUDA\\v12.3\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\libnvvp', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4\\libnvvp', 'C', '\\Program Files\\NVIDIA GPU C
omputing Toolkit\\CUDA\\v11.2\\libnvvp', 'C', '\\Windows\\system32', 'C', '\\Windows', 'C', '\\Windows\\System32\\Wbem', 'C', '\\Windows\\System32\\WindowsPowerShell\\v1.0', 'C', '\\Windows\\System32\\OpenSSH', 'C', '\\Program F
iles\\NVIDIA Corporation\\NVIDIA NvDLISR', 'C', '\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common', 'D', '\\Program Files\\nodejs', 'D', '\\Program Files\\Git\\cmd', 'D', '\\Anaconda3\\envs\\torch\\Lib\\site-packages\\to
rch\\lib', 'D', '\\Anaconda3\\envs\\torch\\Lib\\site-packages\\pyqt5_tools', 'D', '\\mingw\\bin', 'D', '\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\Hostx64\\x64', 'D', '\\P
rogram Files\\MATLAB\\R2017a\\runtime\\win64', 'D', '\\Program Files\\MATLAB\\R2017a\\bin', 'D', '\\Program Files\\Graphviz\\bin', 'D', '\\Program Files (x86)\\Xshell7', 'D', '\\Program Files (x86)\\Xftp 7', 'C', '\\Program File
s\\NVIDIA Corporation\\Nsight Compute 2020.3.1', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\lib', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\include', 'C', '\\Program Files\\NVIDIA G
PU Computing Toolkit\\CUDA\\v12.3\\lib', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include', 'C', '\\Users\\mulin\\AppData\\Local\\Programs\\Python\\Launcher', 'D', '\\Anaconda3', 'D', '\\Anaconda3\\Libra
ry\\mingw-w64\\bin', 'D', '\\Anaconda3\\Library\\usr\\bin', 'D', '\\Anaconda3\\Library\\bin', 'D', '\\Anaconda3\\Scripts', 'C', '\\Users\\mulin\\AppData\\Local\\Microsoft\\WindowsApps', 'D', '\\Program Files\\Bandizip', 'D', '\\
Program Files\\Microsoft VS Code\\bin', 'C', '\\Users\\mulin\\AppData\\Roaming\\npm', 'E', '\\Program Files\\JetBrains\\IntelliJ IDEA Community Edition 2021.2.3\\bin', 'F', '\\PyCharm Community Edition 2023.3.5\\bin']), This pat
h may cause jittor found the wrong libs, please unset LD_LIBRARY_PATH and remove cuda lib path in Path.
Or you can let jittor install cuda for you: `python3.x -m jittor_utils.install_cuda`
[w 0329 17:11:14.033000 68 compile_extern.py:203] CUDA related path found in LD_LIBRARY_PATH or PATH(['', 'C', '\\Users\\mulin\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\mkl\\
dnnl_win_2.2.0_cpu_vcomp\\bin', '', 'C', '\\Users\\mulin\\.cache\\jittor\\mkl\\dnnl_win_2.2.0_cpu_vcomp\\lib', '', 'C', '\\Users\\mulin\\.cache\\jittor\\jt1.3.8\\cl\\py3.12.0\\Windows-10-10.x9d\\IntelRCoreTMi5x0d\\default', '', 
'C', '\\Users\\mulin\\.cache\\jittor\\jt1.3.8\\cl\\py3.12.0\\Windows-10-10.x9d\\IntelRCoreTMi5x0d\\default\\cu11.2.67', '', 'C', '\\Users\\mulin\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\bin', '', 'C', '\\Users\\mulin\\.cach
e\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib\\x64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\win10_kits\\lib\\ucrt\\x64', '', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\win10_kits\\lib\\um\\x64', '', 'C', '\\Users\\mulin\
\.cache\\jittor\\msvc\\VC\\lib', '', 'C', '\\Users\\mulin\\AppData\\Local\\Programs\\Python\\Python312\\libs', 'C', '\\Users\\mulin\\.cache\\jittor\\msvc\\VC\\_\\_\\_\\_\\_\\bin', 'F', '\\jittor_learning\\pythonProject1\\.venv/S
cripts', 'D', '\\Anaconda3', 'D', '\\Anaconda3\\Library\\mingw-w64\\bin', 'D', '\\Anaconda3\\Library\\usr\\bin', 'D', '\\Anaconda3\\Library\\bin', 'D', '\\Anaconda3\\Scripts', 'D', '\\Anaconda3\\bin', 'D', '\\Anaconda3\\condabin
', 'C', '\\Program Files\\NVIDIA\\CUDNN\\v9.0\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\bin', 'C', '\\Program Files\\NVIDIA G
PU Computing Toolkit\\CUDA\\v12.3\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\libnvvp', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.4\\libnvvp', 'C', '\\Program Files\\NVIDIA GPU C
omputing Toolkit\\CUDA\\v11.2\\libnvvp', 'C', '\\Windows\\system32', 'C', '\\Windows', 'C', '\\Windows\\System32\\Wbem', 'C', '\\Windows\\System32\\WindowsPowerShell\\v1.0', 'C', '\\Windows\\System32\\OpenSSH', 'C', '\\Program F
iles\\NVIDIA Corporation\\NVIDIA NvDLISR', 'C', '\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common', 'D', '\\Program Files\\nodejs', 'D', '\\Program Files\\Git\\cmd', 'D', '\\Anaconda3\\envs\\torch\\Lib\\site-packages\\to
rch\\lib', 'D', '\\Anaconda3\\envs\\torch\\Lib\\site-packages\\pyqt5_tools', 'D', '\\mingw\\bin', 'D', '\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\Hostx64\\x64', 'D', '\\P
rogram Files\\MATLAB\\R2017a\\runtime\\win64', 'D', '\\Program Files\\MATLAB\\R2017a\\bin', 'D', '\\Program Files\\Graphviz\\bin', 'D', '\\Program Files (x86)\\Xshell7', 'D', '\\Program Files (x86)\\Xftp 7', 'C', '\\Program File
s\\NVIDIA Corporation\\Nsight Compute 2020.3.1', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\lib', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.2\\include', 'C', '\\Program Files\\NVIDIA G
PU Computing Toolkit\\CUDA\\v12.3\\lib', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include', 'C', '\\Users\\mulin\\AppData\\Local\\Programs\\Python\\Launcher', 'D', '\\Anaconda3', 'D', '\\Anaconda3\\Libra
ry\\mingw-w64\\bin', 'D', '\\Anaconda3\\Library\\usr\\bin', 'D', '\\Anaconda3\\Library\\bin', 'D', '\\Anaconda3\\Scripts', 'C', '\\Users\\mulin\\AppData\\Local\\Microsoft\\WindowsApps', 'D', '\\Program Files\\Bandizip', 'D', '\\
Program Files\\Microsoft VS Code\\bin', 'C', '\\Users\\mulin\\AppData\\Roaming\\npm', 'E', '\\Program Files\\JetBrains\\IntelliJ IDEA Community Edition 2021.2.3\\bin', 'F', '\\PyCharm Community Edition 2023.3.5\\bin']), This pat
h may cause jittor found the wrong libs, please unset LD_LIBRARY_PATH and remove cuda lib path in Path.
Or you can let jittor install cuda for you: `python3.x -m jittor_utils.install_cuda`
[i 0329 17:11:14.907000 68 cuda_flags.cc:49] CUDA enabled.
[i 0329 17:11:14.912000 68 v10 op.cc:272] Jit op key not found: curand_random?T:float32?R:uniform?JIT:1?JIT_cuda:1?index_t:int32
[i 0329 17:11:14.920000 68 v10 op.cc:279] Get jit op entry: 00007FFEA0AF1150
[i 0329 17:11:14.925000 68 v100 op.cc:268] Jit op key found: curand_random?T:float32?R:uniform?JIT:1?JIT_cuda:1?index_t:int32 jit op entry: 00007FFEA0AF1150
[e 0329 17:11:16.536000 68 log.cc:565] cl : Command line warning D9025 : overriding '/EHa' with '/EHs'
__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.cc
f:/jittor_learning/pythonproject1/.venv/lib/site-packages/jittor/src\misc/cuda_atomic.h(138): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, unsigned short)

f:/jittor_learning/pythonproject1/.venv/lib/site-packages/jittor/src\misc/cuda_atomic.h(153): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, unsigned short)

2 errors detected in the compilation of "C:/Users/mulin/.cache/jittor/jt1.3.8/cl/py3.12.0/Windows-10-10.x9d/IntelRCoreTMi5x0d/default/cu11.2.67/jit/__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__X
DIM_4__YDI___hash_27f85cac8782b9a7_op.cc".
__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.cc

[e 0329 17:11:17.983000 68 log.cc:565] cl : Command line warning D9025 : overriding '/EHa' with '/EHs'
__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.cc
f:/jittor_learning/pythonproject1/.venv/lib/site-packages/jittor/src\misc/cuda_atomic.h(138): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, unsigned short)

f:/jittor_learning/pythonproject1/.venv/lib/site-packages/jittor/src\misc/cuda_atomic.h(153): error: no instance of overloaded function "atomicCAS" matches the argument list
            argument types are: (unsigned short *, unsigned short, unsigned short)

2 errors detected in the compilation of "C:/Users/mulin/.cache/jittor/jt1.3.8/cl/py3.12.0/Windows-10-10.x9d/IntelRCoreTMi5x0d/default/cu11.2.67/jit/__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__X
DIM_4__YDI___hash_27f85cac8782b9a7_op.cc".
__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.cc


======================================================================
ERROR: test_conv_transpose3d (__main__.TestCudnnConvOp.test_conv_transpose3d)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "F:\jittor_learning\pythonProject1\.venv\Lib\site-packages\jittor\test\test_cudnn_op.py", line 184, in test_conv_transpose3d
    check((2,5,10,10,10), (5,4,3,3,3), (1,1,1), (1,1,1))
  File "F:\jittor_learning\pythonProject1\.venv\Lib\site-packages\jittor\test\test_cudnn_op.py", line 166, in check
    jt.sync_all()
RuntimeError: [f 0329 17:11:20.880000 68 executor.cc:686] 
Execute fused operator(0/10) failed.
[JIT Source]: C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\IntelRCoreTMi5x0d\default\cu11.2.67\jit\__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8
782b9a7_op.cc
[OP TYPE]: fused_op:( broadcast_to, reindex, binary.multiply, reduce.add,)
[Input]: float32[5,3,3,3,], float32[10,100,100,3,],
[Output]: float32[10,49,49,5,],
[Async Backtrace]: not found, please set env JT_SYNC=1, trace_py_var=3
[Reason]: [f 0329 17:11:20.880000 68 log.cc:608] Check failed ret(1) == 0(0) Run cmd failed: "C:\Users\mulin\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe" "C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.
x9d\IntelRCoreTMi5x0d\default\cu11.2.67\jit\__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.cc"            -shared  -L"C:\Users\mulin\AppData\Local\Programs\P
ython\Python312\libs" -lpython312  -Xcompiler -EHa -Xcompiler -MD -Xcompiler -utf-8   -I"C:\Users\mulin\.cache\jittor\msvc\VC\include" -I"C:\Users\mulin\.cache\jittor\msvc\win10_kits\include\ucrt" -I"C:\Users\mulin\.cache\jittor
\msvc\win10_kits\include\shared" -I"C:\Users\mulin\.cache\jittor\msvc\win10_kits\include\um" -DNOMINMAX  -L"C:\Users\mulin\.cache\jittor\msvc\VC\lib" -L"C:\Users\mulin\.cache\jittor\msvc\win10_kits\lib\um\x64" -L"C:\Users\mulin\
.cache\jittor\msvc\win10_kits\lib\ucrt\x64"  -I"f:\jittor_learning\pythonproject1\.venv\lib\site-packages\jittor\src" -I"C:\Users\mulin\AppData\Local\Programs\Python\Python312\include" -DHAS_CUDA -DIS_CUDA -I"C:\Users\mulin\.cac
he\jittor\jtcuda\cuda11.2_cudnn8_win\include" -I"f:\jittor_learning\pythonproject1\.venv\lib\site-packages\jittor\extern\cuda\inc"  -lcudart -L"C:\Users\mulin\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\lib\x64" -L"C:\Users\mulin\.
cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin"  -I"C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\IntelRCoreTMi5x0d\default\cu11.2.67" -L"C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\Intel
RCoreTMi5x0d\default\cu11.2.67" -L"C:\Users\mulin\.cache\jittor\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\IntelRCoreTMi5x0d\default"  -l"jit_utils_core.cp312-win_amd64"  -l"jittor_core.cp312-win_amd64"  -x cu --cudart=shared -ccbin=
"C:\Users\mulin\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe" --use_fast_math  -w  -I"f:\jittor_learning\pythonproject1\.venv\lib\site-packages\jittor\extern/cuda/inc"  -arch=compute_61  -code=sm_61  -o "C:\Users\mulin\.cache\jitt
or\jt1.3.8\cl\py3.12.0\Windows-10-10.x9d\IntelRCoreTMi5x0d\default\cu11.2.67\jit\__opkey0_broadcast_to__Tx_float32__DIM_7__BCAST_7__opkey1_reindex__Tx_float32__XDIM_4__YDI___hash_27f85cac8782b9a7_op.dll" -Xlinker -EXPORT:"?jit_r
un@FusedOp@jittor@@QEAAXXZ"
**********
Async error was detected. To locate the async backtrace and get better error report, please rerun your code with two enviroment variables set:
cmd:
>>> set JT_SYNC=1
>>> set trace_py_var=3
powershell:
>>> $env:JT_SYNC=1
>>> $env:trace_py_var=3


----------------------------------------------------------------------
Ran 5 tests in 5.975s

FAILED (errors=3)

复现此问题的代码或者描述、
显卡驱动已更新至最新，本地安装了cuda v11.2以及cuDNN v8.9.3；执行jittor推荐的cuda安装输出如下；使用的IDE为PyCharm，安装的jittor版本为1.3.8.5

(base) (.venv) PS F:\jittor_learning\pythonProject1> python -m jittor_utils.install_cuda
[i 0329 17:15:05.890743 96 install_cuda.py:93] cuda_driver_version: [12, 4, 0]
[i 0329 17:15:05.890743 96 install_cuda.py:164] nvcc is installed at C:\Users\mulin\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe

Pluto · 2024 年4 月 3 日 06:21

本地测试后发现与显卡有一定关系，使用1080ti显卡会出现该问题，3070显卡没有问题，希望官方可以对显卡要求做出说明更新

LDY · 2024 年4 月 10 日 10:03

尝试更新github上最新版本，是否还会出现相同的问题，谢谢

wangg12 · 2024 年4 月 29 日 06:05

在2080Ti和3090上都有问题，cudnn测试无法通过，按提示设置环境变量啥的之后也没法解决

ERROR: test_conv_transpose3d (__main__.TestCudnnConvOp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/test/test_cudnn_op.py", line 184, in test_conv_transpose3d
    check((2,5,10,10,10), (5,4,3,3,3), (1,1,1), (1,1,1))
  File "/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/test/test_cudnn_op.py", line 169, in check
    jt.sync_all()
RuntimeError: Wrong inputs arguments, Please refer to examples(help(jt.sync_all)).

Types of your inputs are:
 self   = module,
 args   = (),

The function declarations are:
 void sync_all(bool device_sync=false)

Failed reason:[f 0429 14:16:02.437796 80 parallel_compiler.cc:331] Error happend during compilation:
 [Error] source file location:/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/jit/cudnn_conv3d_backward_x__Tx_float32__Ty_float32__Tw_float32__JIT_1__JIT_cuda_1__index_t_int32_hash_c6c8fbf71eff8711_op.cc
Compile operator(0/1)failed:Op(427:2:1:1:i2:o1:s0:g1,cudnn_conv3d_backward_x->428)

Reason: [f 0429 14:16:02.437692 80 log.cc:605] Check failed: ret>=0 && ret<=256  Run cmd failed: "/usr/local/cuda-12.1/bin/nvcc" "/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/jit/cudnn_conv3d_backward_x__Tx_float32__Ty_float32__Tw_float32__JIT_1__JIT_cuda_1__index_t_int32_hash_c6c8fbf71eff8711_op.cc"      -std=c++14 -Xcompiler -fPIC  -Xcompiler -march=native  -Xcompiler -fdiagnostics-color=always  -lstdc++ -ldl -shared  -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/src" -I/home/gu/miniforge3/envs/foundationpose/include/python3.9 -I/home/gu/miniforge3/envs/foundationpose/include/python3.9 -DHAS_CUDA -DIS_CUDA -I"/usr/local/cuda-12.1/include" -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/inc"  -lcudart -L"/usr/local/cuda-12.1/lib64" -Xlinker -rpath="/usr/local/cuda-12.1/lib64"  -I"/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75" -L"/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75" -Xlinker -rpath="/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75" -L"/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default" -Xlinker -rpath="/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default"  -l:"jit_utils_core.cpython-39-x86_64-linux-gnu".so  -l:"jittor_core.cpython-39-x86_64-linux-gnu".so  -x cu --cudart=shared -ccbin="/usr/bin/g++" --use_fast_math  -w  -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/inc"  -arch=compute_75  -code=sm_75  -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/cudnn/inc"  -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/cudnn/ops"  -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/inc" -I"/home/gu/miniforge3/envs/foundationpose/lib/python3.9/site-packages/jittor/extern/cuda/cudnn/inc" -lcudnn -L"/usr/local/cuda-12.1/lib64" -Xlinker -rpath="/usr/local/cuda-12.1/lib64"  -L"/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/cuda" -Xlinker -rpath="/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/cuda" -l:libcuda_extern.so   -L"/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/custom_ops" -Xlinker -rpath="/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/custom_ops" -l:"gen_ops_cudnn_conv_backward_w_cudnn_conv3d_backwar___hash3d0727.cpython-39-x86_64-linux-gnu".so   -o "/home/gu/.cache/jittor/jt1.3.9/g++9.4.0/py3.9.19/Linux-5.4.0-17x95/IntelRCoreTMi7x48/a493/default/cu12.1.66_sm_75/jit/cudnn_conv3d_backward_x__Tx_float32__Ty_float32__Tw_float32__JIT_1__JIT_cuda_1__index_t_int32_hash_c6c8fbf71eff8711_op.so" 
return 512. This might be an overcommit issue or out of memory. Try : sudo sysctl vm.overcommit_memory=1, or set enviroment variable `export DISABLE_MULTIPROCESSING=1`