Rafael dos Santos Silva
7357280e88
FEATURE: Add old OpenAI tokenizer to embeddings ( #1487 )
2025-07-07 15:07:27 -03:00
Rafael dos Santos Silva and Roman Rizzi
d792919ddf
DEV: Move tokenizers to a gem ( #1481 )
...
Also renames the Mixtral tokenizer to Mistral.
See gem at github.com/discourse/discourse_ai-tokenizers
Co-authored-by: Roman Rizzi <[email protected]
2025-07-02 14:43:03 -03:00
Rafael dos Santos Silva
9dccc1eb93
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 ( #1440 )
2025-06-17 10:25:03 -03:00
Roman Rizzi
df63e36ad8
FEATURE: Make Mixtral tokenizer available for embeddings ( #1258 )
2025-04-11 12:01:38 -03:00
Kris
99e73f09ff
UX: improve embeddings config styles ( #1085 )
...
* WIP: improve embeddings config styles
* switch to textarea, fix back button
* remove log, update button, fix tests
* stree
* fix spec
* spec fix
* remove comment
2025-01-24 16:24:59 +11:00
Rafael dos Santos Silva
67a1257b89
FEATURE: Gemini Tokenizer ( #1088 )
2025-01-23 18:20:35 -03:00
Roman Rizzi
e2e753d73c
FEATURE: Formalize support for matryoshka dimensions. ( #1083 )
...
We have a flag to signal we are shortening the embeddings of a model.
Only used in Open AI's text-embedding-3-*, but we plan to use it for other services.
2025-01-22 11:26:46 -03:00
Roman Rizzi
a5e5ae72a8
FIX: Open AI embedding shortening is only available for some models ( #1080 )
2025-01-21 17:50:40 -03:00
Roman Rizzi
3b66fb3e87
FIX: Restore the accidentally deleted query prefix. ( #1079 )
...
Additionally, we add a prefix for embedding generation.
Both are stored in the definitions table.
2025-01-21 14:10:31 -03:00
Roman Rizzi
f5cf1019fb
FEATURE: configurable embeddings ( #1049 )
...
* Use AR model for embeddings features
* endpoints
* Embeddings CRUD UI
* Add presets. Hide a couple more settings
* system specs
* Seed embedding definition from old settings
* Generate search bit index on the fly. cleanup orphaned data
* support for seeded models
* Fix run test for new embedding
* fix selected model not set correctly
2025-01-21 12:23:19 -03:00
Rafael dos Santos Silva
Kris