jt.sync_all()函数报错,按提示设置JT_SYNC和trace_py_var后仍未解决

运行官方模型pix2pix(https://github.com/Jittor/gan-jittor/tree/master/models/pix2pix)时报错:

[i 0211 21:48:22.952000 04 compiler.py:944] Jittor(1.3.1.38) src: d:\software\anaconda\anaconda\envs\jittor\lib\site-packages\jittor
[i 0211 21:48:22.979000 04 compiler.py:945] cl at C:\Users\33038\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe(19.29.30133)
[i 0211 21:48:22.979000 04 compiler.py:946] cache_path: C:\Users\33038\.cache\jittor\jt1.3.1\cl\py3.8.12\Windows-10-10.x53\IntelRCoreTMi7x56\default
[i 0211 21:48:22.982000 04 install_cuda.py:51] cuda_driver_version: [11, 1, 0]
[i 0211 21:48:23.003000 04 __init__.py:372] Found C:\Users\33038\.cache\jittor\jtcuda\cuda11.0_cudnn8_win\bin\nvcc.exe(11.0.194) at C:\Users\33038\.cache\jittor\jtcuda\cuda11.0_cudnn8_win\bin\nvcc.exe.
[i 0211 21:48:23.081000 04 __init__.py:372] Found gdb(7.6.1) at D:\Software\MinGW\bin\gdb.EXE.
[i 0211 21:48:23.112000 04 __init__.py:372] Found addr2line(2.32) at D:\Software\MinGW\bin\addr2line.EXE.
[i 0211 21:48:23.341000 04 compiler.py:999] cuda key:cu11.0.194_sm_61
[i 0211 21:48:23.342000 04 __init__.py:187] Total mem: 31.96GB, using 10 procs for compiling.
[i 0211 21:48:24.004000 04 jit_compiler.cc:27] Load cc_path: C:\Users\33038\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe
[i 0211 21:48:24.005000 04 init.cc:62] Found cuda archs: [61,]
[i 0211 21:48:24.116000 04 compile_extern.py:497] mpicc not found, distribution disabled.
[i 0211 21:48:24.889000 04 cuda_flags.cc:32] CUDA enabled.
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Namespace(b1=0.5, b2=0.999, batch_size=1, channels=3, checkpoint_interval=30, dataset_name='facades', decay_epoch=100, epoch=0, img_height=256, img_width=256, lr=0.0002, n_cpu=8, n_epochs=200, sample_interval=5)
Traceback (most recent call last):
  File "D:/Scientific Research/BRDF/Jittor/Program/pix2px/pix2pix.py", line 213, in <module>
    main()
  File "D:/Scientific Research/BRDF/Jittor/Program/pix2px/pix2pix.py", line 175, in main
    jt.sync_all()
RuntimeError: [f 0211 21:48:39.814000 04 executor.cc:584] 
Execute fused operator(558/624) failed. 
[JIT Source]: C:\Users\33038\.cache\jittor\jt1.3.1\cl\py3.8.12\Windows-10-10.x53\IntelRCoreTMi7x56\default\cu11.0.194_sm_61\jit\_opkey0_array_T_float32__JIT_1__JIT_cuda_1__index_t_int32___opkey1_array_T_int32__o_0__JIT___hash_94630827edcac943_op.cc 
[OP TYPE]: fused_op:( array, array, array, broadcast_to, array, array, broadcast_to, broadcast_to, broadcast_to, broadcast_to, broadcast_to, broadcast_to, binary.multiply, binary.multiply, binary.multiply, binary.add, binary.multiply, binary.multiply, binary.add, binary.multiply, binary.add,)
[Input]: float32[1,], float32[1,], float32[512,512,4,4,], float32[512,512,4,4,], 
[Output]: float32[512,512,4,4,], float32[512,512,4,4,], 
[Async Backtrace]: not found, please set env JT_SYNC=1, trace_py_var=3 
[Reason]: [f 0211 21:48:39.813000 04 helper_cuda.h:126] CUDA error at d:\software\anaconda\anaconda\envs\jittor\lib\site-packages\jittor\src\mem\allocator\cuda_managed_allocator.cc:23  code=2( cudaErrorMemoryAllocation ) cudaMallocManaged(&ptr, size)
**********
Async error was detected. To locate the async backtrace and get better error report, please rerun your code with two enviroment variables set:
cmd: 
>>> set JT_SYNC=1
>>> set trace_py_var=3
powershell: 
>>> $env:JT_SYNC=1
>>> $env:trace_py_var=3


Process finished with exit code 1

您是在什么操作系统跑的呢,以及运行时显存占用有满吗?

用的win10跑的,运行时显存还剩一大半,并且把batchsize设置成1,数据集图片删减到十张也是这样。

您的显卡是啥型号呢,以及运行时显存占用了多少G?

显卡为1050Ti,显存占用2个G