### 问题描述 使用model提供的yolov4和coco2017的数据集训练模型，内存占用越来越大 ### 环境信息 - **Hardware Environment(`Ascend`/`GPU`/`CPU`) / 硬件环境**: > Please delete the backend not involved / 请删除不涉及的后端: > /device Ascend * 1 > /cpu kunpeng920 > /mem 96G - **Software Environment / 软件环境 (Mandatory / 必填)**: -- MindSpore version (e.g., 2.0.0) :MindSpore2.1.0 -- Python version (e.g., Python 3.7.5) :Python 3.9.18 -- OS platform and distribution (e.g., Linux Ubuntu 16.04):openEuler22.03 - **Execute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: > Please delete the mode not involved / 请删除不涉及的模式: > /mode pynative ### 关联用例 yolov4模型训练 ### 重现步骤 1. 使用models 2.1分支的代码以及coco2017的数据防止obs路径； 2.启用ModelArts的训练作业，填写相关的训练超参数； 3.启动训练作业； ### 预期结果 能够正常跑完训练作业，并且内存不能一直增长，控制在一定范围； ### 日志/截图 [在这里上传图片]   2024-06-22 15:48:44,721:INFO:epoch[6], iter[93699], loss:399.882875, per step time: 263.88 ms, fps: 30.32, lr:0.031962066888809204 2024-06-22 15:49:11,941:INFO:epoch[6], iter[93799], loss:371.868206, per step time: 272.20 ms, fps: 29.39, lr:0.03199617937207222 2024-06-22 15:49:42,031:INFO:epoch[6], iter[93899], loss:378.378537, per step time: 300.89 ms, fps: 26.59, lr:0.032030291855335236 2024-06-22 15:50:07,784:INFO:epoch[6], iter[93999], loss:395.540974, per step time: 257.50 ms, fps: 31.07, lr:0.03206440061330795 2024-06-22 15:50:34,984:INFO:epoch[6], iter[94099], loss:375.826539, per step time: 271.99 ms, fps: 29.41, lr:0.03209851309657097 2024-06-22 15:51:03,124:INFO:epoch[6], iter[94199], loss:377.883690, per step time: 281.38 ms, fps: 28.43, lr:0.032132625579833984 2024-06-22 15:51:28,048:INFO:epoch[6], iter[94299], loss:366.131171, per step time: 249.23 ms, fps: 32.10, lr:0.0321667343378067 2024-06-22 15:51:54,840:INFO:epoch[6], iter[94399], loss:393.184814, per step time: 267.91 ms, fps: 29.86, lr:0.03220084682106972 2024-06-22 15:52:18,553:INFO:epoch[6], iter[94499], loss:385.793442, per step time: 237.11 ms, fps: 33.74, lr:0.032234955579042435 2024-06-22 15:52:43,902:INFO:epoch[6], iter[94599], loss:396.595579, per step time: 253.49 ms, fps: 31.56, lr:0.03226906806230545 2024-06-22 15:53:10,707:INFO:epoch[6], iter[94699], loss:396.697080, per step time: 268.03 ms, fps: 29.85, lr:0.032303180545568466 2024-06-22 15:53:37,147:INFO:epoch[6], iter[94799], loss:410.300946, per step time: 264.40 ms, fps: 30.26, lr:0.032337289303541183 2024-06-22 15:54:02,186:INFO:epoch[6], iter[94899], loss:406.076060, per step time: 250.39 ms, fps: 31.95, lr:0.0323714017868042 2024-06-22 15:54:25,655:INFO:epoch[6], iter[94999], loss:376.711527, per step time: 234.68 ms, fps: 34.09, lr:0.032405510544776917 2024-06-22 15:54:53,972:INFO:epoch[6], iter[95099], loss:410.441413, per step time: 283.17 ms, fps: 28.25, lr:0.03243962302803993 2024-06-22 15:55:22,878:INFO:epoch[6], iter[95199], loss:387.909025, per step time: 289.05 ms, fps: 27.68, lr:0.03247373551130295 2024-06-22 15:55:46,615:INFO:epoch[6], iter[95299], loss:377.746292, per step time: 237.37 ms, fps: 33.70, lr:0.032507844269275665 2024-06-22 15:56:16,978:INFO:epoch[6], iter[95399], loss:365.889424, per step time: 303.62 ms, fps: 26.35, lr:0.03254195675253868 2024-06-22 15:56:46,943:INFO:epoch[6], iter[95499], loss:390.484753, per step time: 299.64 ms, fps: 26.70, lr:0.0325760692358017 2024-06-22 15:57:15,355:INFO:epoch[6], iter[95599], loss:381.341825, per step time: 284.12 ms, fps: 28.16, lr:0.032610177993774414 2024-06-22 15:57:41,677:INFO:epoch[6], iter[95699], loss:369.704940, per step time: 263.21 ms, fps: 30.39, lr:0.03264429047703743 2024-06-22 15:58:05,980:INFO:epoch[6], iter[95799], loss:392.960310, per step time: 243.02 ms, fps: 32.92, lr:0.03267839923501015 2024-06-22 15:58:35,570:INFO:epoch[6], iter[95899], loss:407.040228, per step time: 295.90 ms, fps: 27.04, lr:0.03271251171827316 2024-06-22 15:59:01,163:INFO:epoch[6], iter[95999], loss:403.285904, per step time: 255.93 ms, fps: 31.26, lr:0.03274662420153618 time="2024-06-22T15:59:18+08:00" level=error msg="start run_train script failed, error: failed to get process exit code, exit code: -1" file="run_train.go:141" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T15:59:18+08:00" level=info msg="the post-training script(ma-post-end.sh) is not found in the directory where the bootfile is located, skipping execution" file="run_train.go:873" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T15:59:18+08:00" level=info msg="post runtime info for diagnose started" file="run_train.go:950" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T15:59:18+08:00" level=warning msg="failed to collect gpu_info, err: run nvidia-smi -q failed, this may not be a GPU job, err: exit status 127, skipping" file="collector.go:77" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T15:59:18+08:00" level=warning msg="failed to collect ib_stats, err: run command ibstat failed, err: exit status 127, skipping" file="collector.go:77" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T15:59:18+08:00" level=warning msg="failed to collect nvlink_status, err: run command nvidia-smi nvlink -s failed, err: exit status 127, skipping" file="collector.go:77" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:18+08:00" level=error msg="failed to execute envirenment diagnosis and information collect, error: environment diagnosis and info collect time out" file="run_train.go:1038" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=info msg="report event TrainingExit success" file="event.go:63" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=error msg="bootstrap is exiting with exit code 1" file="bootstrap.go:280" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=info msg="retCode 1 has been written to the retCode file /home/ma-user/modelarts/retCode" file="bootstrap.go:258" Command=bootstrap/run Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=info msg="[sidecar] training is completed" Component=ShellScripts Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=info msg="[sidecar] the reason for the failure of the training job is under analysis" Component=ShellScripts Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=warning msg="the log-preview-size parameter exceeds the limit and will be set to the default value 5242880" file="cli.go:234" Command=analyze Component=ma-training-toolkit Platform=ModelArts-Service time="2024-06-22T16:00:19+08:00" level=info msg="[sidecar] stop toolkit_obs_upload_by_channels_pid = 55 by signal SIGTERM" Component=ShellScripts Platform=ModelArts-Service time="2024-06-22T16:00:20+08:00" level=info msg="the final upload is in progress..." file="upload.go:230" Command=obs/upload_by_channels Component=ma-training-toolkit Platform=ModelArts-Service Task=srt_log_collection time="2024-06-22T16:00:20+08:00" level=info msg="the final upload is in progress..." file="upload.go:230" Command=obs/upload_by_channels Component=ma-training-toolkit Platform=ModelArts-Service Task=train_url time="2024-06-22T16:00:20+08:00" level=info msg="the final upload is in progress..." file="upload.go:230" Command=obs/upload_by_channels Component=ma-training-toolkit Platform=ModelArts-Service Task=log_url ### 备注