[Bug]: [master][cnnctc模型训练报错]

### 问题描述

[cnnctc](https://gitee.com/mindspore/models/tree/master/research/cv/cnnctc#%E8%AE%AD%E7%BB%83%E8%BF%87%E7%A8%8B)训练报EZ3002: Optype [CTCLoss] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: data_type DT_FLOAT16 of input[0, inputs] is unsupported, op type[CTCLoss]..

### 环境信息

- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device Ascend

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 2.0.0) :2.2.0
-- Python version (e.g., Python 3.7.5) :3.9.18
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):euler aarch64
-- GCC/Compiler version (if compiled from source): 7.3.0

- **Execute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode pynative

### 关联用例

/

### 重现步骤

1.预处理数据集：python src/preprocess_dataset.py
2.分布式训练脚本：bash scripts/run_distribute_train_ascend.sh /user/config/jobstart_hccl.json

### 预期结果

```
# 分布式训练结果（8P）
epoch: 1 step: 1 , loss is 76.25, average time per step is 0.335177839748392712
epoch: 1 step: 2 , loss is 73.46875, average time per step is 0.36798572540283203
epoch: 1 step: 3 , loss is 69.46875, average time per step is 0.3429678678512573
epoch: 1 step: 4 , loss is 64.3125, average time per step is 0.33512671788533527
epoch: 1 step: 5 , loss is 58.375, average time per step is 0.33149147033691406
epoch: 1 step: 6 , loss is 52.7265625, average time per step is 0.3292975425720215
...
epoch: 1 step: 8689 , loss is 9.706798802612482, average time per step is 0.3184656601312549
epoch: 1 step: 8690 , loss is 9.70612545289855, average time per step is 0.3184725407765116
epoch: 1 step: 8691 , loss is 9.70695776049204, average time per step is 0.31847309686135555
epoch: 1 step: 8692 , loss is 9.707279624277456, average time per step is 0.31847339290613375
epoch: 1 step: 8693 , loss is 9.70763437950938, average time per step is 0.3184720295013031
epoch: 1 step: 8694 , loss is 9.707695425072046, average time per step is 0.31847410284595573
epoch: 1 step: 8695 , loss is 9.708408273381295, average time per step is 0.31847338271072345
epoch: 1 step: 8696 , loss is 9.708703753591953, average time per step is 0.3184726025560777
epoch: 1 step: 8697 , loss is 9.709536406025824, average time per step is 0.31847212061114694
epoch: 1 step: 8698 , loss is 9.708542263610315, average time per step is 0.3184715309307257
```

### 日志/截图

```
[ERROR] GE_ADPT(2724911,ffff980ba010,python):2024-06-24-21:34:07.522.555 [mindspore/ccsrc/transform/graph_ir/graph_runner.cc:396] CompileGraph] Call GE CompileGraph Failed, ret is: 1343225857
Traceback (most recent call last):
  File "/home/ma-user/work/aicc/models/research/cv/cnnctc/train/train.py", line 149, in <module>
    train()
  File "/home/ma-user/work/aicc/models/research/cv/cnnctc/train/src/model_utils/moxing_adapter.py", line 113, in wrapped_func
    run_func(*args, **kwargs)
  File "/home/ma-user/work/aicc/models/research/cv/cnnctc/train/train.py", line 142, in train
    model.train(config.TRAIN_EPOCHS,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 1068, in train
    self._train(epoch,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 114, in wrapper
    func(self, *args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 617, in _train
    self._train_process(epoch, train_dataset, list_callback, cb_params, initial_epoch, valid_infos)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 919, in _train_process
    outputs = self._train_network(*next_element)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 680, in __call__
    out = self.compile_and_run(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1020, in compile_and_run
    self.compile(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 997, in compile
    _cell_graph_executor.compile(self, phase=self.phase,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 1547, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: Compile graph kernel_graph_0 failed.

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ3002: Optype [CTCLoss] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: data_type DT_FLOAT16 of input[0, inputs] is unsupported, op type[CTCLoss]..
        Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
        Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
        TraceBack (most recent call last):
        Optype [CTCLoss] of Ops kernel [aicpu_tf_kernel] is unsupported. Reason: data_type DT_FLOAT16 of input[0, inputs] is unsupported, op type[CTCLoss]..
        No supported Ops kernel and engine are found for [Default/network-WithLossCell/_loss_fn-ctc_loss/CTCLoss-op767], optype [CTCLoss].
        [Call][PreRun] Failed, graph_id:1, session_id:0.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4113]
        [Compile][Graph]Compile graph failed, error code:1343225857, session_id:0, graph_id:1.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1153]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- Ascend Warning Message:
----------------------------------------------------
W11001: Op [Default/network-WithLossCell/_loss_fn-ctc_loss/CTCLoss-op767] does not hit the high-priority operator information library, which might result in compromised performance.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_graph_executor.cc:813 CompileGraph

```

### 备注

MindSpore/models

内容风险标识

评论 (3)

MindSpore/models .gitee-modal { width: 500px !important; }

内容风险标识