FEATURE: allow tuning of RAG generation (#565)

* FEATURE: allow tuning of RAG generation

- change chunking to be token based vs char based (which is more accurate)
- allow control over overlap / tokens per chunk and conversation snippets inserted
- UI to control new settings

* improve ui a bit

* fix various reindex issues

* reduce concurrency

* try ultra low queue ... concurrency 1 is too slow.

test/javascripts/unit/models/ai-persona-test.js

@@ -49,6 +49,9 @@ module("Discourse AI | Unit | Model | ai-persona", function () {
       vision_enabled: true,
       vision_max_pixels: 100,
       rag_uploads: [],
+      rag_chunk_tokens: 374,
+      rag_chunk_overlap_tokens: 10,
+      rag_conversation_chunks: 10,
     };
 
     const aiPersona = AiPersona.create({ ...properties });