max lenght of the model
#70
by eqemen - opened
Incase you need to know the max lenght of the model is 5104 characters.
Has it changed recently? It was 1024 tokens and not 5104 tokens.
I had to create chunk and do recursive summarization to process long text.
Don't know. when I try directly from text it has some issues with generating.
Some of them solved when I reduce the character size to 5104. Other problems solve when I completely discard all the punctuations.
But these all resolved completely when I use tokenizer instead of pipeline.
With importing Tokenizier method, I will response all my text files and no need to remove any punctuation.
This comment has been hidden
eqemen changed discussion status to closed
god why I cannot delete my comments? ☹️
eqemen changed discussion status to open
what is the min_length of the model?