Skip to content

Commit

Permalink
Bump test suite to include punet tests and fix xfail script. (iree-or…
Browse files Browse the repository at this point in the history
…g#17897)

Highlights from this update:

*
nod-ai/SHARK-TestSuite@fc9fd62
Fixing the `update_config_xfails.py` script
*
nod-ai/SHARK-TestSuite@2c0a312
(and follow-ups) Adding the [Partitioned UNet
(punet)](https://github.com/nod-ai/sharktank/tree/main/sharktank/sharktank/models/punet)
model to the test suite, using data from
https://huggingface.co/amd-shark/sdxl-quant-models

This also changes the 'models' pytest command to include all output,
even for xfail'd tests. This will let us see the latest compile/run
errors for these programs using the tested flags.

ci-exactly: build_packages,regression_test
  • Loading branch information
ScottTodd authored Jul 15, 2024
1 parent 9d6b425 commit 5943426
Show file tree
Hide file tree
Showing 4 changed files with 19 additions and 12 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/pkgci_regression_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:
uses: actions/checkout@v4.1.7
with:
repository: nod-ai/SHARK-TestSuite
ref: a06e730ce325c12db40bb89b43e8e6e897052e96
ref: 1da89a12a8b00dc77506d702f61300803b6240b7
path: SHARK-TestSuite
submodules: false
lfs: false
Expand Down Expand Up @@ -198,7 +198,7 @@ jobs:
uses: actions/checkout@v4.1.7
with:
repository: nod-ai/SHARK-TestSuite
ref: a06e730ce325c12db40bb89b43e8e6e897052e96
ref: 1da89a12a8b00dc77506d702f61300803b6240b7
path: SHARK-TestSuite
submodules: false
lfs: true
Expand All @@ -220,7 +220,7 @@ jobs:
pytest \
SHARK-TestSuite/iree_tests/pytorch/models \
SHARK-TestSuite/iree_tests/sharktank \
-rpfE \
-rA \
-k real_weights \
--no-skip-tests-missing-files \
--capture=no \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@
"pytorch/models/resnet50",
"pytorch/models/sdxl-vae-decode-tank",

// error: 'builtin.module' op failed to run transform dialect passes
// (transform spec file is specific to SDXL?)
"sharktank/llama/open-llama-3b-v2-f16"
// TODO(#17874): error: a handle passed as operand #0 and consumed by this operation points to a payload entity more than once
"sharktank/llama/open-llama-3b-v2-f16",

"sharktank/punet/fp16",
"sharktank/punet/int8"
],
"expected_run_failures": []
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@
"pytorch/models/resnet50",
"pytorch/models/sdxl-vae-decode-tank",

// error: 'builtin.module' op failed to run transform dialect passes
// (transform spec file is specific to SDXL?)
"sharktank/llama/open-llama-3b-v2-f16"
// TODO(#17874): error: a handle passed as operand #0 and consumed by this operation points to a payload entity more than once
"sharktank/llama/open-llama-3b-v2-f16",

"sharktank/punet/fp16",
"sharktank/punet/int8"
],
"expected_run_failures": []
}
9 changes: 6 additions & 3 deletions build_tools/pkgci/external_test_suite/models_gpu_vulkan.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,16 @@
// TODO(#17344): need to regenerate .mlirbc
"pytorch/models/opt-125M",
"pytorch/models/resnet50",

"pytorch/models/sdxl-prompt-encoder-tank",
"pytorch/models/sdxl-scheduled-unet-3-tank",
"pytorch/models/sdxl-vae-decode-tank",

// error: 'builtin.module' op failed to run transform dialect passes
// (transform spec file is specific to SDXL?)
"sharktank/llama/open-llama-3b-v2-f16"
// TODO(#17874): error: a handle passed as operand #0 and consumed by this operation points to a payload entity more than once
"sharktank/llama/open-llama-3b-v2-f16",

"sharktank/punet/fp16",
"sharktank/punet/int8"
],
"expected_run_failures": []
}

0 comments on commit 5943426

Please sign in to comment.