[Bug]: 【910B1】【训练解决方案】【模型训练】【自动化】MindSpore 2.2.0 B020版本，pod环境wide_and_deep性能不达标

### 问题描述

【910B1】【训练解决方案】【模型训练】【自动化】MindSpore 2.2.0 B020版本，pod环境wide_and_deep性能不达标

### 环境信息

**Hardware Environment(Ascend/GPU/CPU)
版本：Ascend Training Solution 23.0.RC3.B030
驱动和固件:
MindSpore 2.2.0 B020
https://cmc.rnd.huawei.com/cmcversion/index/releaseView?deltaId=6855363598680064&isSelect=Software

CANN 7.0.RC1.B030
https://cmc-szv.clouddragon.huawei.com/cmcversion/index/releaseView?deltaId=8685077575369088&isSelect=Software

Ascend HDK 23.0.RC3.B030
https://cmc-szv.clouddragon.huawei.com/cmcversion/index/releaseView?deltaId=8724961893286400&isSelect=Software

Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 2.0.0) :MindSpore 2.2.0 B020
-- Python version (e.g., Python 3.7.5) :Python 3.7.5
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):openEuler 22.03 LTS
-- GCC/Compiler version (if compiled from source): 7.3.0

### 关联用例

/01xs5itoqrp/03210uq2ba2ln/030f0uu5e90p4/030f0uu5e90q0/03090uu5ea5vp697/03090uu5ea6qf6b3/03090uu5ea7336br/03090uu5ea73f/03090uu5ea73q6bs/03090uu5ea7456bt/03090uu5eaf15j12/03090uu5eaf1vj3f/
/01xs5itoqrp/03210uq2ba2ln/030f0uu5e90p4/030f0uu5e90q0/03090uu5ea5vp697/03090uu5ea6qf6b3/03090uu5ea7336br/03090uu5ea73f/03090uu5ea73q6bs/03090uu5ea7456bt/03090uu5eafl0ksq/03090uu5eaflskvl/

### 重现步骤

1.在环境安装MindSpore 2.2.0 B020和CANN 7.0.RC1.B030
2.拉起训练
 bash run_multinpu_train_for_910b.sh 8 5 /home/data/criteo_mindrecord/ /home/CI_daily/Train_MS/rank_table.json

### 预期结果

性能达标

### 日志/截图

![输入图片说明](https://foruda.gitee.com/images/1694677640329389288/ccc7351a_11896310.png "屏幕截图")

### 备注

定位人：代宇鑫 30023125

版本信息：
MindSpore 2.2.0.B060 910A + ge per step time：23ms 性能正常
CANN 7.0.RC1.B070 910B + ge per step time：36ms 性能劣化
对比profiler数据：
发现性能差异几乎来自于GatherV2和UnsortedSegmentSum
![输入图片说明](https://foruda.gitee.com/images/1696840382091710835/470f9552_9647040.png "屏幕截图")
当前网络GatherV2算子有两种规格：
```
{"Input_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "200000,80"}, "Input_1": {"format": "NCHW", "data_type": "INT32", "shape": "16000,39"}, "Input_2": {"format": "NCHW", "data_type": "INT32", "shape": ""}, "Output_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "16000,39,80"}}
{"Input_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "200000,1"}, "Input_1": {"format": "NCHW", "data_type": "INT32", "shape": "16000,39"}, "Input_2": {"format": "NCHW", "data_type": "INT32", "shape": ""}, "Output_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "16000,39,1"}}
```
当前网络UnsortedSegmentSum算子有两种规格：
```
{"Input_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "16000,39,8"}, "Input_1": {"format": "NCHW", "data_type": "INT32", "shape": "16000,39"}, "Input_2": {"format": "NCHW", "data_type": "INT32", "shape": ""}, "Output_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "200000,8"}}
{"Input_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "16000,39,80"}, "Input_1": {"format": "NCHW", "data_type": "INT32", "shape": "16000,39"}, "Input_2": {"format": "NCHW", "data_type": "INT32", "shape": ""}, "Output_0": {"format": "NCHW", "data_type": "FLOAT", "shape": "200000,80"}}
```

MindSpore/models

内容风险标识

评论 (4)

MindSpore/models .gitee-modal { width: 500px !important; }

内容风险标识