79107575

Date: 2024-10-20 16:56:14
Score: 0.5
Natty:
Report link

You can change the tokenizer's vocabulary:

tokenizer.add_tokens(["asadaf", "sdfsaf"])
model.resize_token_embeddings(len(tokenizer)) # change input embeddings size
input_text = "This is asadaf and sdfsaf"
print(tokenizer(input_text))

As a result, asadaf and sdfsaf would be tokenized as unique words.

Reasons:
  • Low length (0.5):
  • Has code block (-0.5):
  • Low reputation (0.5):
Posted by: Treetagger is a nightmare