runtime error

Exit code: 1. Reason: afetensors: 100%|██████████| 508M/508M [00:04<00:00, 112MB/s] Loading pipeline components...: 0%| | 0/6 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s] Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 13.98it/s] Loading pipeline components...: 100%|██████████| 6/6 [00:00<00:00, 6.40it/s] Optimizing pipeline... SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=10 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=12 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 139, in worker_init torch.move(callback=callback) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 444, in move e.submit(copy_context().run, _move, callback=callback).result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 427, in _move original_cuda = original.pin_memory().cuda(non_blocking=True) RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1131, please report a bug to PyTorch. Traceback (most recent call last): File "/home/user/app/app.py", line 70, in <module> optimize_pipeline_(pipe, File "/home/user/app/optimization.py", line 125, in optimize_pipeline_ compiled_transformer_1, compiled_transformer_2 = compile_transformer() File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 210, in gpu_executor raise ctx.exception( spaces.zero.context.ZeroGPUException: RuntimeError

Container logs: