Rafael dos Santos SilvaandRoman Rizzi 5db7bf6e68 Mixtral (#376) ...

Add both Mistral and Mixtral support. Also includes vLLM-openAI inference support.

Co-authored-by: Roman Rizzi <[email protected]>

2023-12-26 14:49:55 -03:00

..

all_mpnet_base_v2_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

anthropic_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

basic_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

bert_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

bge_large_en_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

llama2_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

mixtral_tokenizer.rb

Mixtral (#376)

2023-12-26 14:49:55 -03:00

multilingual_e5_large_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

open_ai_tokenizer.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00