Extension to visualize how tokenizer encodes the tokens #2018
Interpause
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
As in title. I might work on this if I have time. Right now, would be useful to confirm special tokens added inside my character prompts are tokenized correctly.
For now, I added some stuff to
text_generation.py
:Beta Was this translation helpful? Give feedback.
All reactions