To recreate issue:

pip install -r requirements.txt
python main.py

My results are (CPU):

Keras 100 predict iterations took 6.44 seconds
Keras: 64.44 ms per prediction
Torch 100 predict iterations took 0.05 seconds
Torch: 0.51 ms per prediction

Updated Results with variatons (CPU/m2)

Keras with .predict() in a loop
Keras 100 predict iterations took 2.14 seconds
Keras: 21.37 ms per prediction
---
Keras with __call__ in a loop:
Keras 100 predict iterations took 0.73 seconds
Keras: 7.33 ms per prediction
---
Keras with predict dataset:
Keras 100 predict iterations took 0.21 seconds
Keras: 2.09 ms per prediction
---
Torch with numpy in a loop:
Torch 100 predict iterations took 0.04 seconds
Torch: 0.41 ms per prediction
---
Torch with tensor in a loop:
Torch 100 predict iterations took 0.03 seconds
Torch: 0.25 ms per prediction