AscendIndexSQ 建库规模设置为特定值时，search报错

### 该问题是怎么引起的？
AscendIndexSQ 建库规模设置为特定值时，用例会失败

### 重现步骤
https://gitee.com/ascend/ascend-referenceapps/blob/master/mxIndex_samples/TestAscendIndexSQ.cpp 的 TEST(TestAscendIndexSQ, QPS)测试用例中如果将建库规模设置为 [2049,6144]范围内的大小（通过抽样测试推出，该区间并非全部验证，不排除其他范围的值也可能报错），search的时候就会报错。

```
TEST(TestAscendIndexSQ, QPS)
{
    std::vector<int> dim = { 512 };
    std::vector<size_t> ntotal = { 5000 };//{ 7000000 };
    std::vector<int> searchNum = {1};//{ 1, 2, 4, 8, 16, 32, 48, 64, 96 };

size_t maxSize = ntotal.back() * dim.back();
    std::vector<float> data(maxSize);
    for (size_t i = 0; i < maxSize; i++) {
        data[i] = 1.0 * FastRand() / FAST_RAND_MAX;
    }
    
    Norm(data.data(), ntotal.back(), dim.back());

for (size_t i = 0; i < dim.size(); i++) {
        faiss::ascend::AscendIndexSQConfig conf({ 0 }, 1024 * 1024 * 1500);
        //faiss::ascend::AscendIndexSQConfig conf({ 0,1,2,3 }, 1024 * 1024 * 1500);
        faiss::ascend::AscendIndexSQ index(dim[i], faiss::ScalarQuantizer::QuantizerType::QT_8bit, faiss::METRIC_INNER_PRODUCT,
            conf);

for (size_t j = 0; j < ntotal.size(); j++) {
            index.reset();
            for (auto deviceId : conf.deviceList) {
                int len = index.getBaseSize(deviceId);
                ASSERT_EQ(len, 0);
            }

index.train(ntotal[j], data.data());
            index.add(ntotal[j], data.data());
            {
                int getTotal = 0;
                for (size_t k = 0; k < conf.deviceList.size(); k++) {
                    int tmpTotal = index.getBaseSize(conf.deviceList[k]);
                    getTotal += tmpTotal;
                }
                EXPECT_EQ(getTotal, ntotal[j]);
            }
            {
                for (size_t n = 0; n < searchNum.size(); n++) {
                    int k = 100;
                    int loopTimes = 1;//100;
                    std::vector<float> dist(searchNum[n] * k, 0);
                    std::vector<faiss::Index::idx_t> label(searchNum[n] * k, 0);
                    double ts = GetMillisecs();
                    for (int l = 0; l < loopTimes; l++) {
                        index.search(searchNum[n], data.data(), k, dist.data(), label.data());
                        //index.search_with_masks(searchNum[n], data.data(), k, dist.data(), label.data(), (void*)masks4search);
                        for(int i = 0; i < k; i++){
                            printf("AscendIndexSQ====Top%d: dist[%d]=%f, label[%d]=%d\n",i, i, dist[i], i, label[i]);
                        }
                    }
                    double te = GetMillisecs();
                    int cases = i * ntotal.size() * searchNum.size() + j * searchNum.size() + n;
                    printf("case[%d]: base:%zu, dim:%d, search num:%d, QPS:%.4f\n", cases, ntotal[j], dim[i],
                        searchNum[n], 1000 * searchNum[n] * loopTimes / (te - ts));
                }
            }
        }
    }
}
```

### 报错信息
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.143 [engine.cc:1355]876065 ReportExceptProc:Task exception! device_id=0, stream_id=7, task_id=4, type=0, retCode=0x26, [aicore exception]
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.382 [device_error_proc.cc:510]876065 PrintCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.392 [device_error_proc.cc:510]876065 PrintCoreErrorInfo:The error from device(0), serial number is 52, there is an aicore error, core id is 5, error code = 0x800000, dump info: pc start: 0x800124080118000, current: 0x124080118938, vec error info: 0x1139def3, mte error info: 0x304814e, ifu error info: 0xb3adeafc7b80, ccu error info: 0xba69e3f7004bc17d, cube error info: 0x34, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x124000569000, errorStr: The DDR address of the MTE instruction is out of range.
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.432 [device_error_proc.cc:541]876065 PrintCoreErrorInfo:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.440 [device_error_proc.cc:541]876065 PrintCoreErrorInfo:The extend info from device(0), serial number is 52, there is aicore error, core id is 5, aicore int: 0x10, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101d14000000000, biu status1: 0x80000201020000, clk gate mask: 0, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0xd1, dbg data3: 0, dfx data: 0
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.472 [task.cc:1151]876065 PreCheckTaskErr:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.481 [task.cc:1151]876065 PreCheckTaskErr:Kernel task happen error, retCode=0x26, [aicore exception].
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.533 [task.cc:1122]876065 PrintErrorInfo:Aicore kernel execute failed, device_id=0, stream_id=7, report_stream_id=7, task_id=4, flip_num=0, fault kernel_name=608/4_-1_3_DistanceSQ8IPMaxs_tvmbin, program id=0, hash=5123669073413599788.
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.554 [stream.cc:1128]876065 GetError:Stream Synchronize failed, stream_id=7, retCode=0x26, [aicore exception].
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.562 [stream.cc:1131]876065 GetError:report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.568 [stream.cc:1131]876065 GetError:Aicore kernel execute failed, device_id=0, stream_id=7, report_stream_id=7, task_id=4, flip_num=0, fault kernel_name=608/4_-1_3_DistanceSQ8IPMaxs_tvmbin, program id=0, hash=5123669073413599788.
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.579 [logger.cc:341]876065 StreamSynchronize:Stream synchronize failed
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.592 [api_c.cc:724]876065 rtStreamSynchronize:ErrCode=507015, desc=[aicore exception], InnerCode=0x7150026
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.599 [error_message_manage.cc:49]876065 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.607 [error_message_manage.cc:49]876065 FuncErrorReason:rtStreamSynchronize execute failed, reason=[aicore exception]
[ERROR] ASCENDCL(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.619 [stream.cpp:126]876065 aclrtSynchronizeStream: synchronize stream failed, runtime result = 507015
[ERROR] APP(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.633 [log_inner.cpp:76]876065 searchPaged:ascendhost/src/index/IndexSQIPAicpu.cpp:168: "Err: 'ret == ACL_SUCCESS' failed: aclrtSynchronizeStream default stream: 507015
"
[ERROR] APP(876065,TestAscendIndexSQ):2023-07-12-15:11:23.933.646 [log_inner.cpp:76]876065 RpcIndexSearch:ascendhost/src/rpc-local/AscendRpcLocal.cpp:75: "Err: 'ret == APP_ERR_OK' failed: Failed to search index id: 0
terminate called after throwing an instance of 'faiss::FaissException'
  what():  Error in faiss::ascend::AscendIndexImpl::searchImpl(int, const float*, int, float*, faiss::Index::idx_t*) const::<lambda(int)> at ascend/impl/AscendIndexImpl.cpp:358: Error: 'ret == RPC_ERROR_NONE' failed: Search implement failed(1).

Ascend/ascend-referenceapps

内容风险标识

评论 (2)

Ascend/ascend-referenceapps .gitee-modal { width: 500px !important; }

内容风险标识