This website requires JavaScript.
Explore
Help
Sign In
Discourse
/
discourse-ai
Watch
1
Star
0
Fork
0
Code
Issues
Pull Requests
Actions
Packages
Projects
Releases
Wiki
Activity
Files
37dbd48513c30d7cce4cc6a4d07761ed665cc0a8
discourse-ai
/
lib
/
tokenizer
T
History
Rafael dos Santos Silva
9dccc1eb93
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (
#1440
)
2025-06-17 10:25:03 -03:00
..
all_mpnet_base_v2_tokenizer.rb
…
anthropic_tokenizer.rb
…
basic_tokenizer.rb
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (
#1440
)
2025-06-17 10:25:03 -03:00
bert_tokenizer.rb
…
bge_large_en_tokenizer.rb
…
bge_m3_tokenizer.rb
FEATURE: Add BGE-M3 embeddings support (
#569
)
2024-04-10 17:24:01 -03:00
gemini_tokenizer.rb
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (
#1440
)
2025-06-17 10:25:03 -03:00
llama3_tokenizer.rb
FEATURE: Llama 3 tokenizer (
#615
)
2024-05-13 12:45:52 -03:00
mixtral_tokenizer.rb
Mixtral (
#376
)
2023-12-26 14:49:55 -03:00
multilingual_e5_large_tokenizer.rb
…
open_ai_gpt4o_tokenizer.rb
FEATURE: GPT4o Tokenizer (
#721
)
2024-07-22 15:26:14 -03:00
open_ai_tokenizer.rb
FIX/REFACTOR: FoldContent revamp (
#866
)
2024-10-25 11:51:17 -03:00
qwen_tokenizer.rb
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (
#1440
)
2025-06-17 10:25:03 -03:00
