Rafael dos Santos Silva 9dccc1eb93 FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)

2025-06-17 10:25:03 -03:00

..

all_mpnet_base_v2_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

anthropic_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

basic_tokenizer.rb

FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)

2025-06-17 10:25:03 -03:00

bert_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

bge_large_en_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

bge_m3_tokenizer.rb

FEATURE: Add BGE-M3 embeddings support (#569)

2024-04-10 17:24:01 -03:00

gemini_tokenizer.rb

FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)

2025-06-17 10:25:03 -03:00

llama3_tokenizer.rb

FEATURE: Llama 3 tokenizer (#615)

2024-05-13 12:45:52 -03:00

mixtral_tokenizer.rb

Mixtral (#376)

2023-12-26 14:49:55 -03:00

multilingual_e5_large_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

open_ai_gpt4o_tokenizer.rb

FEATURE: GPT4o Tokenizer (#721)

2024-07-22 15:26:14 -03:00

open_ai_tokenizer.rb

FIX/REFACTOR: FoldContent revamp (#866)

2024-10-25 11:51:17 -03:00

qwen_tokenizer.rb

FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)

2025-06-17 10:25:03 -03:00