Skip to content

test: add qwen3 tilelet pto kernels#426

Open
HecreReed wants to merge 16 commits intohw-native-sys:mainfrom
HecreReed:codex/qwen-scope2-case
Open

test: add qwen3 tilelet pto kernels#426
HecreReed wants to merge 16 commits intohw-native-sys:mainfrom
HecreReed:codex/qwen-scope2-case

Conversation

@HecreReed
Copy link
Copy Markdown
Collaborator

Summary

  • add test/samples/Qwen3Scope2/ with 13 qwen3_32b_decode_scope2.py generated .pto kernels
  • teach test/samples/runop.sh to include Qwen3Scope2 in direct .pto coverage
  • pass --pto-arch a5 --pto-level=level3 for Qwen3Scope2 when no explicit override is provided
  • skip these cases by default in remote validation workflow inputs for now

Details

These kernels are generated from the pypto-lib Qwen3 scope2 decode example and are intended to provide compile-regression coverage for pypto-generated A5 .pto inputs.

The kernels compile with the current ptoas flow when using A5 + level3 lowering. They are added as direct .pto samples instead of handwritten IR.

Remote board validation is intentionally left conservative in this draft: the workflow defaults skip these cases so this PR can land compile coverage first without changing the current board-run default surface.

Validation

  • PTOAS_BIN=/Users/laoda/pto/PTOAS/build/tools/ptoas/ptoas bash test/samples/runop.sh -t Qwen3Scope2

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Qwen3 scope2 PTO kernels and updates the runop.sh script to include the Qwen3Scope2 directory in the test suite. The script was modified to apply default architecture and lowering level flags for these kernels. A review comment suggests decoupling the logic for these default flags to ensure that --pto-level=level3 is applied even if a user provides an explicit architecture override.

Comment on lines +175 to +190
local has_pto_arch_override=0
if ((${#ptoas_flags[@]})); then
for ((idx=0; idx<${#ptoas_flags[@]}; ++idx)); do
if [[ "${ptoas_flags[idx]}" == "--pto-arch" && $((idx + 1)) -lt ${#ptoas_flags[@]} ]]; then
target_arch="${ptoas_flags[idx + 1]}"
has_pto_arch_override=1
elif [[ "${ptoas_flags[idx]}" == --pto-arch=* ]]; then
target_arch="${ptoas_flags[idx]#--pto-arch=}"
has_pto_arch_override=1
fi
done
fi
if [[ "$A" == "Qwen3Scope2" && $has_pto_arch_override -eq 0 ]]; then
ptoas_flags+=(--pto-arch a5 --pto-level=level3)
target_arch="a5"
fi
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for applying default flags to the Qwen3Scope2 directory is currently coupled to the presence of the --pto-arch flag. If a user provides an explicit --pto-arch override in PTOAS_FLAGS but omits --pto-level, the required --pto-level=level3 default will not be applied, which will cause compilation failures for these specific kernels as they require Level-3 lowering. It is better to detect and apply these overrides independently.

  local has_pto_arch_override=0
  local has_pto_level_override=0
  if ((${#ptoas_flags[@]})); then
    for ((idx=0; idx<${#ptoas_flags[@]}; ++idx)); do
      if [[ "${ptoas_flags[idx]}" == "--pto-arch" && $((idx + 1)) -lt ${#ptoas_flags[@]} ]]; then
        target_arch="${ptoas_flags[idx + 1]}"
        has_pto_arch_override=1
      elif [[ "${ptoas_flags[idx]}" == --pto-arch=* ]]; then
        target_arch="${ptoas_flags[idx]#--pto-arch=}"
        has_pto_arch_override=1
      elif [[ "${ptoas_flags[idx]}" == "--pto-level" && $((idx + 1)) -lt ${#ptoas_flags[@]} ]]; then
        has_pto_level_override=1
      elif [[ "${ptoas_flags[idx]}" == --pto-level=* ]]; then
        has_pto_level_override=1
      fi
    done
  fi
  if [[ "$A" == "Qwen3Scope2" ]]; then
    if [[ $has_pto_arch_override -eq 0 ]]; then
      ptoas_flags+=(--pto-arch a5)
      target_arch="a5"
    fi
    if [[ $has_pto_level_override -eq 0 ]]; then
      ptoas_flags+=(--pto-level=level3)
    fi
  fi

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3
decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7
decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11
decode_attention_incore_12 --pto-level=level3

@HecreReed HecreReed marked this pull request as ready for review April 3, 2026 02:48
@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:fc6110570fca
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260403_104907_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:internal / RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3

日志尾部

SES], test/samples/planmemory/plan_memory_fragmentation_hole_fit.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_for_iter_args_yield.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_bind_tile_alias_liveness.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_golden.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_compare.py [not in RUN_ONLY_CASES], test/samples/Xors/xors.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_golden.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_compare.py [not in RUN_ONLY_CASES], test/samples/Xor/xor.py [not in RUN_ONLY_CASES], ... (+422 more)

===== STAGE sample-build-and-test @ 2026-04-03 10:50:42 =====
bash test/samples/runop.sh --enablebc all
PTOAS_OUT_DIR=/tmp/ptoas-board-monitor-a5/runs/20260403_104907_manual_pr426/payload/test/samples
========== SUMMARY ==========
-----------------------------
OK=0  FAIL=0  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=0 @ 2026-04-03 10:50:44 =====

===== INTERNAL ERROR =====
Traceback (most recent call last):
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1952, in run_once
    summary = runner.run()
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1380, in run
    self.generate_payload()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1344, in generate_payload
    self.resolve_payload_run_only_cases()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1032, in resolve_payload_run_only_cases
    raise RuntimeError(f"RUN_ONLY_CASES matched zero buildable cases: {self.run_only_cases}")
RuntimeError: RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45dbf6790c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +193 to +194
if [[ "$A" == "Qwen3Scope2" && $has_pto_arch_override -eq 0 ]]; then
ptoas_flags+=(--pto-arch a5 --pto-level=level3)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve level3 default when arch is overridden

For Qwen3Scope2, this branch only injects --pto-level=level3 when no --pto-arch override is present, so a common override like PTOAS_FLAGS='--pto-arch=a5' disables the level3 injection and runs these kernels at ptoas default level2. These new kernels use pto.alloc_tile addr=... (which ptoas documents as level3-only), so this path causes avoidable compile failures even though the user requested the correct arch. Consider detecting --pto-level independently and only auto-injecting level3 when it is not explicitly set.

Useful? React with 👍 / 👎.

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:fc6110570fca
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260403_105206_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:internal / RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12

日志尾部

.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_golden.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_compare.py [not in RUN_ONLY_CASES], test/samples/Xors/xors.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_golden.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_compare.py [not in RUN_ONLY_CASES], test/samples/Xor/xor.py [not in RUN_ONLY_CASES], ... (+422 more)

===== STAGE sample-build-and-test @ 2026-04-03 10:53:43 =====
bash test/samples/runop.sh --enablebc all
PTOAS_OUT_DIR=/tmp/ptoas-board-monitor-a5/runs/20260403_105206_manual_pr426/payload/test/samples
========== SUMMARY ==========
-----------------------------
OK=0  FAIL=0  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=0 @ 2026-04-03 10:53:44 =====

===== INTERNAL ERROR =====
Traceback (most recent call last):
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1952, in run_once
    summary = runner.run()
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1380, in run
    self.generate_payload()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1344, in generate_payload
    self.resolve_payload_run_only_cases()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1032, in resolve_payload_run_only_cases
    raise RuntimeError(f"RUN_ONLY_CASES matched zero buildable cases: {self.run_only_cases}")
RuntimeError: RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测成功

  • 触发方式:manual
  • 源码提交:fc6110570fca
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260403_110405_manual_pr426.log
  • 结果 TSV:/root/ptoas-board-monitor-a5/logs/20260403_110405_manual_pr426.tsv
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:fc6110570fca
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260403_163706_manual_pr426.log
  • 结果 TSV:/root/ptoas-board-monitor-a5/logs/20260403_163706_manual_pr426.tsv
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:internal / board validation produced zero testcase rows for RUN_ONLY_CASES: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9

日志尾部

/usr/local/Ascend/cann-9.0.0/aarch64-linux/simulator/Ascend910B1/lib
[2026-04-03 16:38:49] SIM_SOC_VERSION=Ascend950
[2026-04-03 16:38:49] === NPU Device Check ===
uid=0(root) gid=0(root) groups=0(root),1001(HwHiAiUser)
crw-rw---- 1 HwHiAiUser HwHiAiUser 508, 0 Mar 26 19:44 /dev/davinci0
crw-rw---- 1 HwHiAiUser HwHiAiUser 508, 1 Mar 26 19:44 /dev/davinci1
crw-rw---- 1 HwHiAiUser HwHiAiUser 509, 0 Mar 26 19:39 /dev/davinci_manager
[2026-04-03 16:38:49] Using vendored pto-isa tree at /tmp/ptoas-board-monitor-a5/runs/20260403_163706_manual_pr426/payload/pto-isa (no .git); skipping clone/fetch/checkout.
[2026-04-03 16:38:49] === SUMMARY ===
[2026-04-03 16:38:49] OK=0 FAIL=0 SKIP=0
[2026-04-03 16:38:49] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260403_163706_manual_pr426/remote_npu_validation_results.tsv
===== END STAGE board-validation rc=0 @ 2026-04-03 16:38:49 =====

===== INTERNAL ERROR =====
Traceback (most recent call last):
  File "/root/ptoas-board-monitor-a5/monitor.py", line 2071, in run_once
    summary = runner.run()
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1501, in run
    self.board_validate()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1490, in board_validate
    raise RuntimeError(
        "board validation produced zero testcase rows for RUN_ONLY_CASES: " + self.payload_run_only_cases
    )
RuntimeError: board validation produced zero testcase rows for RUN_ONLY_CASES: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:fc6110570fca
  • 结果汇总:OK 10 / FAIL 3 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260403_170006_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • decode_attention_incore_7 (run, exit=2)
  • decode_attention_incore_2 (run, exit=1)
  • decode_attention_incore_10 (run, exit=2)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

decode_attention_incore_7

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_7_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_7_kernel.dir/decode_attention_incore_7_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_7_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-03 17:05:25] ERROR: testcase failed (exit 2): decode_attention_incore_7
decode_attention_incore_2

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_2/main.cpp:109)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 1977663] 2026-04-03-17:11:01.806.916 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 131, there is an aivec error exception, core id is 0, error code = 95, dump info: pc start: 0x100040800000, current: 0x100040800178, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080031000310047, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0x80000000.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(95) errorStr: The DDR address of the MTE instruction is out of range. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z25decode_attention_incore_2Pu6__bf16PfS_S_ii, fault kernel info ext=_Z25decode_attention_incore_2Pu6__bf16PfS_S_ii, program id=0, hash=1092065175157927996.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-03 17:11:37] ERROR: testcase failed (exit 1): decode_attention_incore_2
decode_attention_incore_10

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260403_170006_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_10_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_10_kernel.dir/decode_attention_incore_10_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_10_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-03 17:15:48] ERROR: testcase failed (exit 2): decode_attention_incore_10

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3
decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7
decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11
decode_attention_incore_12

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:45dbf6790cf8
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_094510_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:sample-build-and-test / exit=1

日志尾部

y_nested_loops.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_loop_no_reuse_outer_live.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_loop_in_if.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_if_yield.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_if_in_loop.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_fragmentation_two_holes.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_fragmentation_hole_fit.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_for_iter_args_yield.py [not in RUN_ONLY_CASES], test/samples/planmemory/plan_memory_bind_tile_alias_liveness.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_golden.py [not in RUN_ONLY_CASES], test/samples/Xors/xors_compare.py [not in RUN_ONLY_CASES], test/samples/Xors/xors.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_golden.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_compare.py [not in RUN_ONLY_CASES], test/samples/Xor/xor.py [not in RUN_ONLY_CASES], ... (+418 more)

===== STAGE sample-build-and-test @ 2026-04-07 09:46:45 =====
bash test/samples/runop.sh --enablebc all
PTOAS_OUT_DIR=/tmp/ptoas-board-monitor-a5/runs/20260407_094510_manual_pr426/payload/test/samples
========== SUMMARY ==========
Qwen3Scope2(decode_attention_incore_0.pto) FAIL ptoas failed: decode_attention_incore_0.pto
Qwen3Scope2(decode_attention_incore_1.pto) FAIL ptoas failed: decode_attention_incore_1.pto
Qwen3Scope2(decode_attention_incore_2.pto) FAIL ptoas failed: decode_attention_incore_2.pto
Qwen3Scope2(decode_attention_incore_3.pto) FAIL ptoas failed: decode_attention_incore_3.pto
-----------------------------
OK=0  FAIL=4  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=1 @ 2026-04-07 09:46:47 =====

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:45dbf6790cf8
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_094805_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:sample-build-and-test / exit=1

日志尾部

rs/xors.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_golden.py [not in RUN_ONLY_CASES], test/samples/Xor/xor_compare.py [not in RUN_ONLY_CASES], test/samples/Xor/xor.py [not in RUN_ONLY_CASES], ... (+409 more)

===== STAGE sample-build-and-test @ 2026-04-07 09:49:46 =====
bash test/samples/runop.sh --enablebc all
PTOAS_OUT_DIR=/tmp/ptoas-board-monitor-a5/runs/20260407_094805_manual_pr426/payload/test/samples
========== SUMMARY ==========
Qwen3Scope2(decode_attention_incore_0.pto) FAIL ptoas failed: decode_attention_incore_0.pto
Qwen3Scope2(decode_attention_incore_10.pto) FAIL ptoas failed: decode_attention_incore_10.pto
Qwen3Scope2(decode_attention_incore_11.pto) FAIL ptoas failed: decode_attention_incore_11.pto
Qwen3Scope2(decode_attention_incore_12.pto) FAIL ptoas failed: decode_attention_incore_12.pto
Qwen3Scope2(decode_attention_incore_1.pto) FAIL ptoas failed: decode_attention_incore_1.pto
Qwen3Scope2(decode_attention_incore_2.pto) FAIL ptoas failed: decode_attention_incore_2.pto
Qwen3Scope2(decode_attention_incore_3.pto) FAIL ptoas failed: decode_attention_incore_3.pto
Qwen3Scope2(decode_attention_incore_4.pto) FAIL ptoas failed: decode_attention_incore_4.pto
Qwen3Scope2(decode_attention_incore_5.pto) FAIL ptoas failed: decode_attention_incore_5.pto
Qwen3Scope2(decode_attention_incore_6.pto) FAIL ptoas failed: decode_attention_incore_6.pto
Qwen3Scope2(decode_attention_incore_7.pto) FAIL ptoas failed: decode_attention_incore_7.pto
Qwen3Scope2(decode_attention_incore_8.pto) FAIL ptoas failed: decode_attention_incore_8.pto
Qwen3Scope2(decode_attention_incore_9.pto) FAIL ptoas failed: decode_attention_incore_9.pto
-----------------------------
OK=0  FAIL=13  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=1 @ 2026-04-07 09:49:48 =====

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 issue428_cube_sync_regression --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:45dbf6790cf8
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_101805_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 issue428_cube_sync_regression --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12,issue428_cube_sync_regression
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:sample-build-and-test / exit=1

日志尾部

c_a3_dyn.py) SKIP requires --pto-arch=a3
Sync(test_intercore_sync_a3_missing_setffts.py) SKIP requires --pto-arch=a3
Sync(test_intercore_sync_a3_modes.py) SKIP requires --pto-arch=a3
Sync(test_intercore_sync_a3.py) SKIP requires --pto-arch=a3
Sync(test_intercore_sync_a5_dyn.py) OK   generated: test_intercore_sync_a5_dyn-pto.cpp
Sync(test_intercore_sync_a5_functional.py) OK   generated: test_intercore_sync_a5_functional-pto.cpp
Sync(test_intercore_sync_a5_ptoisa_vec.py) OK   generated: test_intercore_sync_a5_ptoisa_vec-pto.cpp
Sync(test_intercore_sync_a5.py) OK   generated: test_intercore_sync_a5-pto.cpp
Sync(test_mem_inject_sync_basic.py) FAIL ptoas failed: test_mem_inject_sync_basic-pto-ir.pto
Sync(test_set_wait_unified_api.py) OK   generated: test_set_wait_unified_api-pto.cpp
Sync(tmatmulk_autosync_a5.py) FAIL ptoas failed: tmatmulk_autosync_a5-pto-ir.pto
Sync(tmatmulk_autosync.py) FAIL ptoas failed: tmatmulk_autosync-pto-ir.pto
Tcvt(tcvt.py) FAIL ptoas failed: tcvt-pto-ir.pto
TileSetGetValue(tile_getval_mat_invalid.py) XFAIL python failed as expected
TileSetGetValue(tileSetGetValue.py) FAIL ptoas failed: tileSetGetValue-pto-ir.pto
TInsert(tinsert.py) FAIL ptoas failed: tinsert-pto-ir.pto
Trans(trans.py) FAIL ptoas failed: trans-pto-ir.pto
Trap(trap.py) OK   generated: trap-pto.cpp
VectorAddition(vadd_pto_ir.py) FAIL ptoas failed: vadd_pto_ir-pto-ir.pto
VectorAddition(vadd_validshape_hyper.py) FAIL ptoas failed: vadd_validshape_hyper-pto-ir.pto
VectorAddition(vectorAddition.py) FAIL ptoas failed: vectorAddition-pto-ir.pto
Xors(xors.py) FAIL ptoas failed: xors-pto-ir.pto
Xor(xor.py)  FAIL ptoas failed: xor-pto-ir.pto
-----------------------------
OK=28  FAIL=162  SKIP=4
=============================
===== END STAGE sample-build-and-test rc=1 @ 2026-04-07 10:21:11 =====

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:45dbf6790cf8
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_103405_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:internal / RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12

日志尾部

wen3Scope2(decode_attention_incore_4.pto) OK   generated: decode_attention_incore_4.cpp
Qwen3Scope2(decode_attention_incore_5.pto) OK   generated: decode_attention_incore_5.cpp
Qwen3Scope2(decode_attention_incore_6.pto) OK   generated: decode_attention_incore_6.cpp
Qwen3Scope2(decode_attention_incore_7.pto) OK   generated: decode_attention_incore_7.cpp
Qwen3Scope2(decode_attention_incore_8.pto) OK   generated: decode_attention_incore_8.cpp
Qwen3Scope2(decode_attention_incore_9.pto) OK   generated: decode_attention_incore_9.cpp
-----------------------------
OK=13  FAIL=0  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=0 @ 2026-04-07 10:35:48 =====
patched payload runner case discovery: samples/basic *-pto.cpp|*_pto.cpp

===== INTERNAL ERROR =====
Traceback (most recent call last):
  File "/root/ptoas-board-monitor-a5/monitor.py", line 2195, in run_once
    summary = runner.run()
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1623, in run
    self.generate_payload()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1565, in generate_payload
    self.resolve_payload_run_only_cases()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/ptoas-board-monitor-a5/monitor.py", line 1200, in resolve_payload_run_only_cases
    raise RuntimeError(f"RUN_ONLY_CASES matched zero buildable cases: {self.run_only_cases}")
RuntimeError: RUN_ONLY_CASES matched zero buildable cases: decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:116927660e9d
  • 结果汇总:OK 10 / FAIL 3 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_105505_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • decode_attention_incore_7 (run, exit=2)
  • decode_attention_incore_2 (run, exit=1)
  • decode_attention_incore_10 (run, exit=2)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

decode_attention_incore_7

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_7_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_7_kernel.dir/decode_attention_incore_7_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_7_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-07 10:59:19] ERROR: testcase failed (exit 2): decode_attention_incore_7
decode_attention_incore_2

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_2/main.cpp:109)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 2983618] 2026-04-07-11:04:47.674.346 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 132, there is an aivec error exception, core id is 0, error code = 95, dump info: pc start: 0x100040800000, current: 0x100040800178, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080031000310047, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0x80000000.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(95) errorStr: The DDR address of the MTE instruction is out of range. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z25decode_attention_incore_2Pu6__bf16PfS_S_ii, fault kernel info ext=_Z25decode_attention_incore_2Pu6__bf16PfS_S_ii, program id=0, hash=1092065175157927996.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-07 11:05:23] ERROR: testcase failed (exit 1): decode_attention_incore_2
decode_attention_incore_10

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_105505_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_10_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_10_kernel.dir/decode_attention_incore_10_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_10_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-07 11:09:34] ERROR: testcase failed (exit 2): decode_attention_incore_10

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:1e19329390f0
  • 结果汇总:OK 11 / FAIL 2 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260407_114405_manual_pr426.log
  • 手动指令:/run a5 decode_attention_incore_0 decode_attention_incore_1 decode_attention_incore_2 decode_attention_incore_3 decode_attention_incore_4 decode_attention_incore_5 decode_attention_incore_6 decode_attention_incore_7 decode_attention_incore_8 decode_attention_incore_9 decode_attention_incore_10 decode_attention_incore_11 decode_attention_incore_12 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:decode_attention_incore_0,decode_attention_incore_1,decode_attention_incore_2,decode_attention_incore_3,decode_attention_incore_4,decode_attention_incore_5,decode_attention_incore_6,decode_attention_incore_7,decode_attention_incore_8,decode_attention_incore_9,decode_attention_incore_10,decode_attention_incore_11,decode_attention_incore_12
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • decode_attention_incore_7 (run, exit=2)
  • decode_attention_incore_10 (run, exit=2)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

decode_attention_incore_7

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_7/decode_attention_incore_7_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_7_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_7_kernel.dir/decode_attention_incore_7_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_7_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-07 11:48:20] ERROR: testcase failed (exit 2): decode_attention_incore_7
decode_attention_incore_10

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260407_114405_manual_pr426/npu_validation/Qwen3Scope2/decode_attention_incore_10/decode_attention_incore_10_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/decode_attention_incore_10_kernel.dir/build.make:76: CMakeFiles/decode_attention_incore_10_kernel.dir/decode_attention_incore_10_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/decode_attention_incore_10_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-07 11:58:43] ERROR: testcase failed (exit 2): decode_attention_incore_10

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

qwen3_decode_layer_incore_2

stage=run info=exit=2

ld.lld: error: undefined symbol: qwen3_decode_layer_incore_2_aic(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-088333.o-08c357.extract:(_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_2_aiv(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-088333.o-08c357.extract:(_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_2_aic(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-088333.o-08c357.extract:(_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii_mix_aiv)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_2_aiv(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-088333.o-08c357.extract:(_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii_mix_aiv)
cceld: ccec ReturnCode: 1
cceld: ExecutionFailed: 0
cceld: ErrMsg:
bisheng: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_2_kernel.dir/build.make:90: CMakeFiles/qwen3_decode_layer_incore_2_kernel.dir/launch.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_2_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-08 16:58:06] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_2
qwen3_decode_layer_incore_1

stage=run info=exit=2

ld.lld: error: undefined symbol: qwen3_decode_layer_incore_1_aic(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-e9ae36.o-bb5cb9.extract:(_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_1_aiv(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-e9ae36.o-bb5cb9.extract:(_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_1_aic(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-e9ae36.o-bb5cb9.extract:(_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii_mix_aiv)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_1_aiv(__bf16 AS1*, float AS1*, float AS1, __bf16 AS1, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-e9ae36.o-bb5cb9.extract:(_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii_mix_aiv)
cceld: ccec ReturnCode: 1
cceld: ExecutionFailed: 0
cceld: ErrMsg:
bisheng: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_1_kernel.dir/build.make:90: CMakeFiles/qwen3_decode_layer_incore_1_kernel.dir/launch.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_1_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-08 16:58:09] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_1
qwen3_decode_layer_incore_14

stage=run info=exit=2

ld.lld: error: undefined symbol: qwen3_decode_layer_incore_14_aic(float AS1*, __bf16 AS1*, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-88bbb8.o-12ff89.extract:(_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_14_aiv(float AS1*, __bf16 AS1*, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-88bbb8.o-12ff89.extract:(_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_14_aic(float AS1*, __bf16 AS1*, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-88bbb8.o-12ff89.extract:(_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii_mix_aiv)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_14_aiv(float AS1*, __bf16 AS1*, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-88bbb8.o-12ff89.extract:(_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii_mix_aiv)
cceld: ccec ReturnCode: 1
cceld: ExecutionFailed: 0
cceld: ErrMsg:
bisheng: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_14_kernel.dir/build.make:90: CMakeFiles/qwen3_decode_layer_incore_14_kernel.dir/launch.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_14_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-08 16:58:12] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_14
qwen3_decode_layer_incore_13

stage=run info=exit=2

ld.lld: error: undefined symbol: qwen3_decode_layer_incore_13_aic(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, __bf16 AS1, __bf16 AS1, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-e7cf9a.o-b3e80b.extract:(_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_13_aiv(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, __bf16 AS1, __bf16 AS1, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-e7cf9a.o-b3e80b.extract:(_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_13_aic(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, __bf16 AS1, __bf16 AS1, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-e7cf9a.o-b3e80b.extract:(_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i_mix_aiv)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_13_aiv(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, __bf16 AS1, __bf16 AS1, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-e7cf9a.o-b3e80b.extract:(_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i_mix_aiv)
cceld: ccec ReturnCode: 1
cceld: ExecutionFailed: 0
cceld: ErrMsg:
bisheng: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_13_kernel.dir/build.make:90: CMakeFiles/qwen3_decode_layer_incore_13_kernel.dir/launch.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_13_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-08 16:58:15] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_13
qwen3_decode_layer_incore_10

stage=run info=exit=2

ld.lld: error: undefined symbol: qwen3_decode_layer_incore_10_aic(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-ff03f3.o-bff058.extract:(_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_10_aiv(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, int, int) (.cube)
>>> referenced by launch.cpp
>>>               /tmp/launch-ff03f3.o-bff058.extract:(_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii_mix_aic)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_10_aic(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-ff03f3.o-bff058.extract:(_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii_mix_aiv)
ld.lld: error: undefined symbol: qwen3_decode_layer_incore_10_aiv(float AS1*, __bf16 AS1*, float AS1*, __bf16 AS1, int, int) (.vector)
>>> referenced by launch.cpp
>>>               /tmp/launch-ff03f3.o-bff058.extract:(_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii_mix_aiv)
cceld: ccec ReturnCode: 1
cceld: ExecutionFailed: 0
cceld: ErrMsg:
bisheng: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_10_kernel.dir/build.make:90: CMakeFiles/qwen3_decode_layer_incore_10_kernel.dir/launch.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_10_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-08 16:58:18] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_10
[2026-04-08 16:58:18] === SUMMARY ===
[2026-04-08 16:58:18] OK=0 FAIL=5 SKIP=0
[2026-04-08 16:58:18] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260408_165610_manual_pr426/remote_npu_validation_results.tsv

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:ce7d04111082
  • 结果汇总:OK 0 / FAIL 5 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260408_171705_manual_pr426.log
  • 手动指令:/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • qwen3_decode_layer_incore_2 (run, exit=1)
  • qwen3_decode_layer_incore_1 (run, exit=1)
  • qwen3_decode_layer_incore_14 (run, exit=1)
  • qwen3_decode_layer_incore_13 (run, exit=1)
  • qwen3_decode_layer_incore_10 (run, exit=1)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

qwen3_decode_layer_incore_2

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_2/main.cpp:133)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3304384] 2026-04-08-17:37:14.387.339 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 144, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408002b0, current: 0x1000408005b8, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 145, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800e2c, current: 0x100040800f28, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 145, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800e2c, current: 0x100040800974, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii, fault kernel info ext=_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii, program id=0, hash=17167445014040612000.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 17:37:48] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_2
qwen3_decode_layer_incore_1

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_1/main.cpp:117)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3306885] 2026-04-08-17:56:06.324.878 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 146, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408001f8, current: 0x100040800470, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 147, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800bac, current: 0x1000408012b0, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 147, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800bac, current: 0x100040800f14, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii, fault kernel info ext=_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii, program id=0, hash=13072310816159174184.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 17:56:38] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_1
qwen3_decode_layer_incore_14

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_14/main.cpp:101)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3309476] 2026-04-08-18:14:55.994.685 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 148, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408002bc, current: 0x100040800620, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 149, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800864, current: 0x100040800a70, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 149, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800864, current: 0x10004080095c, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii, fault kernel info ext=_Z28qwen3_decode_layer_incore_14PfPu6__bf16S0_ii, program id=0, hash=12704120089081622103.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 18:15:36] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_14
qwen3_decode_layer_incore_13

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_13/main.cpp:124)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3311989] 2026-04-08-18:33:54.248.945 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 150, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x10004080027c, current: 0x100040800554, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 151, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800998, current: 0x100040800ddc, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 151, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800998, current: 0x100040800b2c, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i, fault kernel info ext=_Z28qwen3_decode_layer_incore_13PfPu6__bf16S_S0_S0_S0_i, program id=0, hash=1621789960468670568.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 18:34:25] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_13
qwen3_decode_layer_incore_10

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_10/main.cpp:109)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3314565] 2026-04-08-18:52:43.082.768 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 152, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408001f8, current: 0x100040800470, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 153, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800a48, current: 0x100040800fe4, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 153, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800a48, current: 0x100040800de4, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii, fault kernel info ext=_Z28qwen3_decode_layer_incore_10PfPu6__bf16S_S0_ii, program id=0, hash=12922822057091878848.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 18:53:22] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_10
[2026-04-08 18:53:22] === SUMMARY ===
[2026-04-08 18:53:22] OK=0 FAIL=5 SKIP=0
[2026-04-08 18:53:22] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260408_171705_manual_pr426/remote_npu_validation_results.tsv

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:ce7d04111082
  • 结果汇总:OK 0 / FAIL 2 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260408_190706_manual_pr426.log
  • 手动指令:/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=-15

失败用例

  • qwen3_decode_layer_incore_2 (run, exit=1)
  • qwen3_decode_layer_incore_1 (run, exit=1)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

qwen3_decode_layer_incore_2

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_190706_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_2/main.cpp:133)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3321533] 2026-04-08-19:27:17.484.006 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 154, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408002b0, current: 0x1000408005b8, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 155, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800e2c, current: 0x100040800f28, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 155, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800e2c, current: 0x100040800974, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii, fault kernel info ext=_Z27qwen3_decode_layer_incore_2Pu6__bf16PfS0_S_S_S_S_ii, program id=0, hash=17167445014040612000.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 19:27:50] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_2
qwen3_decode_layer_incore_1

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507014 (/tmp/ptoas-board-monitor-a5/runs/20260408_190706_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_1/main.cpp:117)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 3325001] 2026-04-08-19:46:08.579.468 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 156, there is an aicore error exception, core id is 0, error code = 0, dump info: pc start: 0x1000408001f8, current: 0x100040800470, sc error info: 0xffffffffffff, su error info: 0xffffffed1efe9387,0x6fddfefff8007fff, mte error info: 0xdc1fffe70007fbff, vec error info: 0, cube error info: 0, l1 error info: 0xffff001ff6ee, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
        TraceBack (most recent call last):
       The extend info: errcode:(0) errorStr: timeout or trap error. subErrType: 0x4.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:583]
       The error from device(chipId:0, dieId:0), serial number is 157, there is an aivec error exception, core id is 0, error code = 0, dump info: pc start: 0x100040800bac, current: 0x1000408012b0, sc error info: 0xffffffffffff, su error info: 0xe7ffd23d1fdc0017,0x4240141410009bfd, mte error info: 0x2005d, vec error info: 0x4080021000310036, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       The error from device(chipId:0, dieId:0), serial number is 157, there is an aivec error exception, core id is 1, error code = 0, dump info: pc start: 0x100040800bac, current: 0x100040800f14, sc error info: 0xffffffffffff, su error info: 0x2985b4fc1dfeefdb,0xe64ef56bc000acdb, mte error info: 0xdebf637b0007deff, vec error info: 0x4d6c3f7f001cfccf, cube error info: 0, l1 error info: 0, aic error mask: 0x395856, para base: 0x100040200000, mte error: 0.[FUNC:ProcessDavidStarsCoreErrorInfo][FILE:device_error_proc_c.cc][LINE:580]
       Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1728]
       AICORE Kernel task happen error, retCode=0x25.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [AIC_INFO] after execute:(no result)[FUNC:GetError][FILE:stream.cc][LINE:1478]
       [DFX_INFO]Aicore kernel execute failed, device_id=1, stream_id=62, report_stream_id=62, task_id=0, flip_num=0, fault kernel_name=_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii, fault kernel info ext=_Z27qwen3_decode_layer_incore_1Pu6__bf16PfS0_S_S_ii, program id=0, hash=13072310816159174184.[FUNC:GetError][FILE:stream.cc][LINE:1478]
       rtStreamSynchronize execution failed, reason=aicore timeout[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-04-08 19:46:40] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_1

# Conflicts:
#	test/samples/validation_runtime.py
@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:7e9b8e26ea06
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_091405_manual_pr426.log
  • 手动指令:/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=-15

日志尾部

/runs/20260409_091405_manual_pr426/payload/test/samples/Qwen3Tilelet/qwen3_decode_layer_incore_2-pto.cpp ===
-- The C compiler identification is Clang 15.0.5
-- The CXX compiler identification is Clang 15.0.5
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/Ascend/cann-9.0.0/bin/bisheng - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/Ascend/cann-9.0.0/bin/bisheng - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.3s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    SOC_VERSION


-- Build files have been written to: /tmp/ptoas-board-monitor-a5/runs/20260409_091405_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_2/build
[ 20%] Building CXX object CMakeFiles/qwen3_decode_layer_incore_2_kernel.dir/qwen3_decode_layer_incore_2_kernel.cpp.o
[ 40%] Building CXX object CMakeFiles/qwen3_decode_layer_incore_2_kernel.dir/launch.cpp.o
bishengbisheng: : warning: argument unused during compilation: '--cce-aicore-enable-tl' [-Wunused-command-line-argument]
warning: argument unused during compilation: '--cce-aicore-enable-tl' [-Wunused-command-line-argument]
[ 60%] Linking CXX shared library libqwen3_decode_layer_incore_2_kernel.so
[ 60%] Built target qwen3_decode_layer_incore_2_kernel
[ 80%] Building CXX object CMakeFiles/qwen3_decode_layer_incore_2.dir/main.cpp.o
[100%] Linking CXX executable qwen3_decode_layer_incore_2
[100%] Built target qwen3_decode_layer_incore_2
===== END STAGE board-validation rc=-15 @ 2026-04-09 09:28:33 =====

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测成功

  • 触发方式:manual
  • 源码提交:2efe3ffc1cc5
  • 结果汇总:OK 5 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_143606_manual_pr426.log
  • 结果 TSV:/root/ptoas-board-monitor-a5/logs/20260409_143606_manual_pr426.tsv
  • 手动指令:/run a5 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:2efe3ffc1cc5
  • 结果汇总:OK 0 / FAIL 20 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_155807_manual_pr426.log
  • 手动指令:/run a5 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_0,qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_3,qwen3_decode_layer_incore_4,qwen3_decode_layer_incore_5,qwen3_decode_layer_incore_6,qwen3_decode_layer_incore_7,qwen3_decode_layer_incore_8,qwen3_decode_layer_incore_9,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_11,qwen3_decode_layer_incore_12,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14,qwen3_decode_layer_incore_15,qwen3_decode_layer_incore_16,qwen3_decode_layer_incore_17,qwen3_decode_layer_incore_18,qwen3_decode_layer_incore_19
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • qwen3_decode_layer_incore_9 (run, exit=1)
  • qwen3_decode_layer_incore_8 (run, exit=2)
  • qwen3_decode_layer_incore_7 (run, exit=1)
  • qwen3_decode_layer_incore_6 (run, exit=1)
  • qwen3_decode_layer_incore_5 (run, exit=2)
  • qwen3_decode_layer_incore_4 (run, exit=2)
  • qwen3_decode_layer_incore_3 (run, exit=2)
  • qwen3_decode_layer_incore_2 (run, exit=1)
  • qwen3_decode_layer_incore_1 (run, exit=1)
  • qwen3_decode_layer_incore_19 (run, exit=1)
  • qwen3_decode_layer_incore_18 (run, exit=1)
  • qwen3_decode_layer_incore_17 (run, exit=1)
  • qwen3_decode_layer_incore_16 (run, exit=1)
  • qwen3_decode_layer_incore_15 (run, exit=1)
  • qwen3_decode_layer_incore_14 (run, exit=1)
  • qwen3_decode_layer_incore_13 (run, exit=1)
  • qwen3_decode_layer_incore_12 (run, exit=1)
  • qwen3_decode_layer_incore_11 (run, exit=1)
  • qwen3_decode_layer_incore_10 (run, exit=1)
  • qwen3_decode_layer_incore_0 (run, exit=1)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

qwen3_decode_layer_incore_9

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_9/main.cpp:84)
[ERROR] RecentErrMsg: [PID: 3551750] 2026-04-09-16:00:06.652.577 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:07] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_9
qwen3_decode_layer_incore_8

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/qwen3_decode_layer_incore_8_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:00:08] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_8
qwen3_decode_layer_incore_7

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_7/main.cpp:105)
[ERROR] RecentErrMsg: [PID: 3552482] 2026-04-09-16:00:11.630.957 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:11] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_7
qwen3_decode_layer_incore_6

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_6/main.cpp:101)
[ERROR] RecentErrMsg: [PID: 3552984] 2026-04-09-16:00:14.889.476 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:15] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_6
qwen3_decode_layer_incore_5

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:97:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:97:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:99:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:106:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:109:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:111:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:114:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:116:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:117:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:120:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:120:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:122:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:122:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:123:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/qwen3_decode_layer_incore_5_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:00:17] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_5
qwen3_decode_layer_incore_4

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:97:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:97:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:99:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:106:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:109:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:111:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:114:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:116:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:117:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:120:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:120:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:122:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:122:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:123:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/qwen3_decode_layer_incore_4_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:00:18] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_4
qwen3_decode_layer_incore_3

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:92:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:96:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:98:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:113:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:115:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:115:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:119:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:119:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:121:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:121:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:122:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/qwen3_decode_layer_incore_3_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:00:20] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_3
qwen3_decode_layer_incore_2

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_2/main.cpp:97)
[ERROR] RecentErrMsg: [PID: 3554390] 2026-04-09-16:00:24.855.189 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:25] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_2
qwen3_decode_layer_incore_1

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_1/main.cpp:89)
[ERROR] RecentErrMsg: [PID: 3554962] 2026-04-09-16:00:29.347.674 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:29] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_1
qwen3_decode_layer_incore_19

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_19/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3555468] 2026-04-09-16:00:32.405.653 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:32] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_19
qwen3_decode_layer_incore_18

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_18/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3556017] 2026-04-09-16:00:36.584.995 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:36] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_18
qwen3_decode_layer_incore_17

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_17/main.cpp:84)
[ERROR] RecentErrMsg: [PID: 3556563] 2026-04-09-16:00:40.955.098 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:41] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_17
qwen3_decode_layer_incore_16

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_16/main.cpp:87)
[ERROR] RecentErrMsg: [PID: 3557172] 2026-04-09-16:00:44.109.894 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:44] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_16
qwen3_decode_layer_incore_15

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_15/main.cpp:75)
[ERROR] RecentErrMsg: [PID: 3557684] 2026-04-09-16:00:47.291.262 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:47] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_15
qwen3_decode_layer_incore_14

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_14/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3558253] 2026-04-09-16:00:51.470.395 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:51] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_14
qwen3_decode_layer_incore_13

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_13/main.cpp:92)
[ERROR] RecentErrMsg: [PID: 3558801] 2026-04-09-16:00:55.992.172 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:56] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_13
qwen3_decode_layer_incore_12

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_12/main.cpp:80)
[ERROR] RecentErrMsg: [PID: 3559405] 2026-04-09-16:00:59.118.026 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:00:59] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_12
qwen3_decode_layer_incore_11

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_11/main.cpp:104)
[ERROR] RecentErrMsg: [PID: 3559944] 2026-04-09-16:01:02.317.326 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:01:02] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_11
qwen3_decode_layer_incore_10

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_10/main.cpp:85)
[ERROR] RecentErrMsg: [PID: 3560502] 2026-04-09-16:01:06.716.949 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:01:07] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_10
qwen3_decode_layer_incore_0

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_0/main.cpp:76)
[ERROR] RecentErrMsg: [PID: 3561025] 2026-04-09-16:01:09.762.503 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:01:10] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_0
[2026-04-09 16:01:10] === SUMMARY ===
[2026-04-09 16:01:10] OK=0 FAIL=20 SKIP=0
[2026-04-09 16:01:10] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260409_155807_manual_pr426/remote_npu_validation_results.tsv

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:2efe3ffc1cc5
  • 结果汇总:OK 0 / FAIL 20 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_161006_manual_pr426.log
  • 手动指令:/run a5 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_0,qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_3,qwen3_decode_layer_incore_4,qwen3_decode_layer_incore_5,qwen3_decode_layer_incore_6,qwen3_decode_layer_incore_7,qwen3_decode_layer_incore_8,qwen3_decode_layer_incore_9,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_11,qwen3_decode_layer_incore_12,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14,qwen3_decode_layer_incore_15,qwen3_decode_layer_incore_16,qwen3_decode_layer_incore_17,qwen3_decode_layer_incore_18,qwen3_decode_layer_incore_19
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • qwen3_decode_layer_incore_9 (run, exit=1)
  • qwen3_decode_layer_incore_8 (run, exit=2)
  • qwen3_decode_layer_incore_7 (run, exit=1)
  • qwen3_decode_layer_incore_6 (run, exit=1)
  • qwen3_decode_layer_incore_5 (run, exit=2)
  • qwen3_decode_layer_incore_4 (run, exit=2)
  • qwen3_decode_layer_incore_3 (run, exit=2)
  • qwen3_decode_layer_incore_2 (run, exit=1)
  • qwen3_decode_layer_incore_1 (run, exit=1)
  • qwen3_decode_layer_incore_19 (run, exit=1)
  • qwen3_decode_layer_incore_18 (run, exit=1)
  • qwen3_decode_layer_incore_17 (run, exit=1)
  • qwen3_decode_layer_incore_16 (run, exit=1)
  • qwen3_decode_layer_incore_15 (run, exit=1)
  • qwen3_decode_layer_incore_14 (run, exit=1)
  • qwen3_decode_layer_incore_13 (run, exit=1)
  • qwen3_decode_layer_incore_12 (run, exit=1)
  • qwen3_decode_layer_incore_11 (run, exit=1)
  • qwen3_decode_layer_incore_10 (run, exit=1)
  • qwen3_decode_layer_incore_0 (run, exit=1)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

qwen3_decode_layer_incore_9

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_9/main.cpp:84)
[ERROR] RecentErrMsg: [PID: 3566559] 2026-04-09-16:12:03.651.146 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:04] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_9
qwen3_decode_layer_incore_8

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:97:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:100:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:105:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:108:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:110:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:114:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_8/qwen3_decode_layer_incore_8_kernel.cpp:114:21: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_M, PIPE_FIX, EVENT_ID0);
                    ^
10 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/qwen3_decode_layer_incore_8_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_8_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:12:05] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_8
qwen3_decode_layer_incore_7

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_7/main.cpp:105)
[ERROR] RecentErrMsg: [PID: 3567266] 2026-04-09-16:12:08.638.088 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:08] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_7
qwen3_decode_layer_incore_6

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_6/main.cpp:101)
[ERROR] RecentErrMsg: [PID: 3567768] 2026-04-09-16:12:11.835.341 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:12] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_6
qwen3_decode_layer_incore_5

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:97:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:97:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:99:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:106:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:109:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:111:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:114:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:116:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:117:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:120:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:120:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:122:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:122:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_5/qwen3_decode_layer_incore_5_kernel.cpp:123:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/qwen3_decode_layer_incore_5_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_5_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:12:13] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_5
qwen3_decode_layer_incore_4

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:97:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:97:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:99:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:106:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:109:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:111:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:114:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:116:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:117:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:120:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:120:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:122:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:122:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_4/qwen3_decode_layer_incore_4_kernel.cpp:123:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/qwen3_decode_layer_incore_4_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_4_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:12:15] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_4
qwen3_decode_layer_incore_3

stage=run info=exit=2

/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:92:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID2);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:93:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID3);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:94:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID4);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:95:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID5);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:96:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:96:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:98:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:105:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:108:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:110:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:113:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID1);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:115:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:115:23: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                      ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:116:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID1);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:119:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
            ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:119:24: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
                       ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:121:12: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
           ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:121:20: error: the ranges of 2nd parameter must be [0, 1], [4, 5]
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
                   ^
/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_3/qwen3_decode_layer_incore_3_kernel.cpp:122:13: error: the ranges of 1st parameter must be [0, 1], [4, 5]
  wait_flag(PIPE_MTE1, PIPE_MTE2, EVENT_ID0);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/build.make:76: CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/qwen3_decode_layer_incore_3_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/qwen3_decode_layer_incore_3_kernel.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
[2026-04-09 16:12:17] ERROR: testcase failed (exit 2): qwen3_decode_layer_incore_3
qwen3_decode_layer_incore_2

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_2/main.cpp:97)
[ERROR] RecentErrMsg: [PID: 3568971] 2026-04-09-16:12:21.664.096 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:22] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_2
qwen3_decode_layer_incore_1

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_1/main.cpp:89)
[ERROR] RecentErrMsg: [PID: 3569530] 2026-04-09-16:12:26.152.862 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:26] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_1
qwen3_decode_layer_incore_19

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_19/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3570040] 2026-04-09-16:12:29.236.882 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:29] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_19
qwen3_decode_layer_incore_18

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_18/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3570822] 2026-04-09-16:12:33.584.942 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:33] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_18
qwen3_decode_layer_incore_17

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_17/main.cpp:84)
[ERROR] RecentErrMsg: [PID: 3571386] 2026-04-09-16:12:37.934.407 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:38] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_17
qwen3_decode_layer_incore_16

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_16/main.cpp:87)
[ERROR] RecentErrMsg: [PID: 3571883] 2026-04-09-16:12:41.078.831 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:41] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_16
qwen3_decode_layer_incore_15

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_15/main.cpp:75)
[ERROR] RecentErrMsg: [PID: 3572384] 2026-04-09-16:12:44.192.489 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:44] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_15
qwen3_decode_layer_incore_14

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_14/main.cpp:81)
[ERROR] RecentErrMsg: [PID: 3572940] 2026-04-09-16:12:48.443.426 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:48] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_14
qwen3_decode_layer_incore_13

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_13/main.cpp:92)
[ERROR] RecentErrMsg: [PID: 3573499] 2026-04-09-16:12:53.038.717 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:53] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_13
qwen3_decode_layer_incore_12

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_12/main.cpp:80)
[ERROR] RecentErrMsg: [PID: 3573997] 2026-04-09-16:12:56.109.615 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:56] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_12
qwen3_decode_layer_incore_11

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_11/main.cpp:104)
[ERROR] RecentErrMsg: [PID: 3574500] 2026-04-09-16:12:59.375.161 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:12:59] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_11
qwen3_decode_layer_incore_10

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_10/main.cpp:85)
[ERROR] RecentErrMsg: [PID: 3575067] 2026-04-09-16:13:03.873.528 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:13:04] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_10
qwen3_decode_layer_incore_0

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/npu_validation/Qwen3Tilelet/qwen3_decode_layer_incore_0/main.cpp:76)
[ERROR] RecentErrMsg: [PID: 3575579] 2026-04-09-16:13:07.030.303 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:13:07] ERROR: testcase failed (exit 1): qwen3_decode_layer_incore_0
[2026-04-09 16:13:07] === SUMMARY ===
[2026-04-09 16:13:07] OK=0 FAIL=20 SKIP=0
[2026-04-09 16:13:07] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260409_161006_manual_pr426/remote_npu_validation_results.tsv

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a5 abs

@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:2efe3ffc1cc5
  • 结果汇总:OK 0 / FAIL 1 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_162709_manual_pr426.log
  • 手动指令:/run a5 abs
  • 触发人:HecreReed
  • 指定用例:abs
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • abs (run, exit=1)

1 similar comment
@reedhecre
Copy link
Copy Markdown

A5 板测失败

  • 触发方式:manual
  • 源码提交:2efe3ffc1cc5
  • 结果汇总:OK 0 / FAIL 1 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260409_162709_manual_pr426.log
  • 手动指令:/run a5 abs
  • 触发人:HecreReed
  • 指定用例:abs
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:board-validation / exit=1

失败用例

  • abs (run, exit=1)

@reedhecre
Copy link
Copy Markdown

A5 板测失败详情:PR #426

abs

stage=run info=exit=1

[ERROR] aclrtSetDevice(deviceId) failed: 507033 (/tmp/ptoas-board-monitor-a5/runs/20260409_162709_manual_pr426/npu_validation/Abs/abs/main.cpp:75)
[ERROR] RecentErrMsg: [PID: 3595747] 2026-04-09-16:29:03.855.444 Invalid_Argument(EE1001): The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        TsdOpen failed. devId=1, tdt error=1[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3321]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6120]
        The argument is invalid.Reason: rtGetDevMsg execution failed, the context is a null pointer.
[2026-04-09 16:29:04] ERROR: testcase failed (exit 1): abs
[2026-04-09 16:29:04] === SUMMARY ===
[2026-04-09 16:29:04] OK=0 FAIL=1 SKIP=0
[2026-04-09 16:29:04] RESULTS_TSV=/tmp/ptoas-board-monitor-a5/runs/20260409_162709_manual_pr426/remote_npu_validation_results.tsv

@HecreReed
Copy link
Copy Markdown
Collaborator Author

/run a3 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3

@reedhecre
Copy link
Copy Markdown

A3 板测失败

  • 触发方式:manual
  • 源码提交:fdfe1abd9484
  • 结果汇总:OK 0 / FAIL 0 / SKIP 0
  • 日志:/home/zhongxuan/ptoas-board-monitor/runtime/logs/20260409_172205_manual_pr426.log
  • 手动指令:/run a3 qwen3_decode_layer_incore_0 qwen3_decode_layer_incore_1 qwen3_decode_layer_incore_2 qwen3_decode_layer_incore_3 qwen3_decode_layer_incore_4 qwen3_decode_layer_incore_5 qwen3_decode_layer_incore_6 qwen3_decode_layer_incore_7 qwen3_decode_layer_incore_8 qwen3_decode_layer_incore_9 qwen3_decode_layer_incore_10 qwen3_decode_layer_incore_11 qwen3_decode_layer_incore_12 qwen3_decode_layer_incore_13 qwen3_decode_layer_incore_14 qwen3_decode_layer_incore_15 qwen3_decode_layer_incore_16 qwen3_decode_layer_incore_17 qwen3_decode_layer_incore_18 qwen3_decode_layer_incore_19 --pto-level=level3
  • 触发人:HecreReed
  • 指定用例:qwen3_decode_layer_incore_0,qwen3_decode_layer_incore_1,qwen3_decode_layer_incore_2,qwen3_decode_layer_incore_3,qwen3_decode_layer_incore_4,qwen3_decode_layer_incore_5,qwen3_decode_layer_incore_6,qwen3_decode_layer_incore_7,qwen3_decode_layer_incore_8,qwen3_decode_layer_incore_9,qwen3_decode_layer_incore_10,qwen3_decode_layer_incore_11,qwen3_decode_layer_incore_12,qwen3_decode_layer_incore_13,qwen3_decode_layer_incore_14,qwen3_decode_layer_incore_15,qwen3_decode_layer_incore_16,qwen3_decode_layer_incore_17,qwen3_decode_layer_incore_18,qwen3_decode_layer_incore_19
  • PTOAS 参数:--pto-level=level3
  • 触发评论:test: add qwen3 tilelet pto kernels #426 (comment)
  • 失败阶段:sample-build-and-test / exit=1

日志尾部

e_layer_incore_12.pto) OK   generated: qwen3_decode_layer_incore_12-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_13.pto) FAIL ptoas failed: qwen3_decode_layer_incore_13.pto
Qwen3Tilelet(qwen3_decode_layer_incore_14.pto) FAIL ptoas failed: qwen3_decode_layer_incore_14.pto
Qwen3Tilelet(qwen3_decode_layer_incore_15.pto) OK   generated: qwen3_decode_layer_incore_15-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_16.pto) OK   generated: qwen3_decode_layer_incore_16-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_17.pto) FAIL ptoas failed: qwen3_decode_layer_incore_17.pto
Qwen3Tilelet(qwen3_decode_layer_incore_18.pto) FAIL ptoas failed: qwen3_decode_layer_incore_18.pto
Qwen3Tilelet(qwen3_decode_layer_incore_19.pto) OK   generated: qwen3_decode_layer_incore_19-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_1.pto) FAIL ptoas failed: qwen3_decode_layer_incore_1.pto
Qwen3Tilelet(qwen3_decode_layer_incore_2.pto) FAIL ptoas failed: qwen3_decode_layer_incore_2.pto
Qwen3Tilelet(qwen3_decode_layer_incore_3.pto) FAIL ptoas failed: qwen3_decode_layer_incore_3.pto
Qwen3Tilelet(qwen3_decode_layer_incore_4.pto) FAIL ptoas failed: qwen3_decode_layer_incore_4.pto
Qwen3Tilelet(qwen3_decode_layer_incore_5.pto) FAIL ptoas failed: qwen3_decode_layer_incore_5.pto
Qwen3Tilelet(qwen3_decode_layer_incore_6.pto) OK   generated: qwen3_decode_layer_incore_6-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_7.pto) OK   generated: qwen3_decode_layer_incore_7-pto.cpp
Qwen3Tilelet(qwen3_decode_layer_incore_8.pto) FAIL ptoas failed: qwen3_decode_layer_incore_8.pto
Qwen3Tilelet(qwen3_decode_layer_incore_9.pto) OK   generated: qwen3_decode_layer_incore_9-pto.cpp
-----------------------------
OK=9  FAIL=11  SKIP=0
=============================
===== END STAGE sample-build-and-test rc=1 @ 2026-04-09 17:24:17 =====

@HecreReed HecreReed force-pushed the codex/qwen-scope2-case branch from 0ccdea1 to f0bb59d Compare April 9, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants