FIX: Disable truncation and padding in all-mpnet-base-v2 tokenizer (#105)

Browse Source

The tokenizer was truncating and padding to 128 tokens, and we try append
new post content until we hit 384 tokens. This was causing the tokenizer
to accept all posts in a topic, wasting CPU and memory.

...

This commit is contained in:

Rafael dos Santos Silva

2023-07-13 21:09:46 -03:00

committed by GitHub

parent 703762a7a9

commit d692ecc7de

1 changed files with 1 additions and 1 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

tokenizers/all-mpnet-base-v2.json

+1 -1

File diff suppressed because one or more lines are too long