Discourse/discourse-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a028309cbdb0565bb1c485f5cefd2af41ac7ecb2
discourse-ai/lib/shared/tokenizer
T
History
Rafael dos Santos SilvaandJoffrey JAFFEUX e457c687ca FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91) 
* FIX: OpenAI Tokenizer was failing to truncate mid emojis

* Update spec/shared/tokenizer.rb

Co-authored-by: Joffrey JAFFEUX <[email protected]>

---------

Co-authored-by: Joffrey JAFFEUX <[email protected]>
2023-06-16 15:15:36 -03:00
..
tokenizer.rb
FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91)
2023-06-16 15:15:36 -03:00