runner.logger.info(‘loss become infinite or NaN
or INFO Loss Logger nan Runner
2023-09-14 09:05:45 时间
在使用mmdetection训练的时候出现了如下错误:
2022-05-07 21:35:13,488 - mmdet - INFO - Epoch [1][50/5111] lr: 1.978e-04, eta: 18:35:19, time: 0.546, data_time: 0.217, memory: 1781, loss_cls: 17.7358, loss_bbox: 4.1217, loss: 21.8
575
2022-05-07 21:35:24,322 - mmdet - INFO - Epoch [1][100/5111] lr: 3.976e-04, eta: 12:58:42, time: 0.217, data_time: 0.003, memory: 1781, loss_cls: 13.2629, loss_bbox: 4.4149, loss: 17.6
778
2022-05-07 21:35:35,155 - mmdet - INFO - Epoch [1][150/5111] lr: 5.974e-04, eta: 11:06:25, time: 0.217, data_time: 0.003, memory: 1781, loss_cls: 9.6519, loss_bbox: 4.5696, loss: 14.22
16
2022-05-07 21:35:45,901 - mmdet - INFO - Epoch [1][200/5111] lr: 7.972e-04, eta: 10:09:16, time: 0.215, data_time: 0.003, memory: 1781, loss_cls: 13.5727, loss_bbox: 5.6242, loss: 19.1
969
runner.run(data_loaders, cfg.workflow)
File "D:\ProgramData\Anaconda3\envs\mytorch\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 127, in run
epoch_runner(data_loaders[i], **kwargs)
File "D:\ProgramData\Anaconda3\envs\mytorch\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 51, in train
self.call_hook('after_train_iter')
File "D:\ProgramData\Anaconda3\envs\mytorch\lib\site-packages\mmcv\runner\base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "D:\ProgramData\Anaconda3\envs\mytorch\lib\site-packages\mmdet-2.24.1-py3.7.egg\mmdet\core\hook\checkloss_hook.py", line 24, in after_train_iter
runner.logger.info('loss become infinite or NaN!')
AssertionError: None
出现这个问题是因为学习率太大了,需要降低学习率;
把学习率改为2e-4就可以解决问题了。
相关文章
- Map or switch
- HTTP Status 500 - The absolute uri: http://java.sun.com/jsp/jstl/core cannot be resolved in either web.xml or the jar files deployed with this application
- Maven报错Please ensure you are using JDK 1.4 or above and not a JRE解决方法!
- JDK1.3安装出现/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory Done.
- [Javascript] Broadcaster + Operator + Listener pattern -- 24. Design choice, ifElse or merge
- 【收藏】goland报错:报错package xxx is not in GOROOT or GOPATH 或者 cannot find package “xxx“ in any of
- cms_install.sh No such file or directory
- 成功解决TypeError: slice indices must be integers or None or have an __index__ method
- 成功解决FileNotFoundError: [Errno 2] No such file or directory: '/home/bai/Myprojects/Tfexamples/data/kn
- 已解决ValueError: expected a simple type, a tuple or a list
- Pixhawk---fatal: Not a git repository (or any of the parent directories)
- sql 语句中count()有条件的时候为什么要加上or null
- Mac FFmpeg编译和解决nasm/yasm not found or too old错误
- 1098 Insertion or Heap Sort C++版
- 【异常】前端提示Cannot use [chunkhash] or [contenthash] for chunk in ‘[name].[chunkhash].js‘