Discourse/discourse-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
101 Commits 45 Branches 0 Tags
a028309cbdb0565bb1c485f5cefd2af41ac7ecb2
Commit Graph
5 Commits
Author SHA1 Message Date
Rafael dos Santos SilvaandJoffrey JAFFEUX e457c687ca FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91) 
* FIX: OpenAI Tokenizer was failing to truncate mid emojis

* Update spec/shared/tokenizer.rb

Co-authored-by: Joffrey JAFFEUX <[email protected]>

---------

Co-authored-by: Joffrey JAFFEUX <[email protected]>
 2023-06-16 15:15:36 -03:00
Rafael dos Santos Silva 739b314312 Fixes for embeddings and truncate (#67) 2023-05-18 09:21:28 +10:00
Rafael dos Santos Silva 3c9513e754 Refinements to embeddings and tokenizers (#61) 
* Refinements to embeddings and tokenizers

* lint

* Truncate with tokenizers for summary

* fix
 2023-05-15 15:10:42 -03:00
Sam e76fc77189 fixes (#53) 
* Minor... use username suggester in case username already exists

* FIX: ensure we truncate long prompts

Previously we

1. Used raw length instead of token counts for counting length
2. We totally dropped a prompt if it was too long

New implementation will truncate "raw" if it gets too long maintaining
meaning.
 2023-05-06 07:31:53 -03:00
Rafael dos Santos Silva 9783e3b025 FEATURE: Add a basic tokenizer API (#37) 
* FEATURE: Add a basic tokenizer API

* Add tests

* lint
 2023-04-19 11:55:59 -03:00