Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Alright this should add
mkl
support. I also created a quick benchmark script to only test thematmul
performance between torch and candle: https://gist.github.com/LLukas22/b58adc148e771afdeaebc4074f0644f7Results without
--features mkl
:With
--features mkl
:I'm running this on an AMD Ryzen 7 3700x, ans seeing these results i guess Sarah did a pretty good job optimizing the
gemm
create.I also re-ran the embedding "benchmark" and candle still is a bit slower there, but faster than without
mkl
. I'm guessing we lose some speed in other operations thanmatmul
or trough the python wrapper.With
mkl
:Without
mkl
:That being said i'm probably gonna re-run these tests tomorrow on an intel based system and see if that changes anything.