Discourse/discourse-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FEATURE: PDF support for rag pipeline (#1118)

Browse Source 
This PR introduces several enhancements and refactorings to the AI Persona and RAG (Retrieval-Augmented Generation) functionalities within the discourse-ai plugin. Here's a breakdown of the changes:

**1. LLM Model Association for RAG and Personas:**

-   **New Database Columns:** Adds `rag_llm_model_id` to both `ai_personas` and `ai_tools` tables. This allows specifying a dedicated LLM for RAG indexing, separate from the persona's primary LLM.  Adds `default_llm_id` and `question_consolidator_llm_id` to `ai_personas`.
-   **Migration:**  Includes a migration (`20250210032345_migrate_persona_to_llm_model_id.rb`) to populate the new `default_llm_id` and `question_consolidator_llm_id` columns in `ai_personas` based on the existing `default_llm` and `question_consolidator_llm` string columns, and a post migration to remove the latter.
-   **Model Changes:**  The `AiPersona` and `AiTool` models now `belong_to` an `LlmModel` via `rag_llm_model_id`. The `LlmModel.proxy` method now accepts an `LlmModel` instance instead of just an identifier.  `AiPersona` now has `default_llm_id` and `question_consolidator_llm_id` attributes.
-   **UI Updates:**  The AI Persona and AI Tool editors in the admin panel now allow selecting an LLM for RAG indexing (if PDF/image support is enabled).  The RAG options component displays an LLM selector.
-   **Serialization:** The serializers (`AiCustomToolSerializer`, `AiCustomToolListSerializer`, `LocalizedAiPersonaSerializer`) have been updated to include the new `rag_llm_model_id`, `default_llm_id` and `question_consolidator_llm_id` attributes.

**2. PDF and Image Support for RAG:**

-   **Site Setting:** Introduces a new hidden site setting, `ai_rag_pdf_images_enabled`, to control whether PDF and image files can be indexed for RAG. This defaults to `false`.
-   **File Upload Validation:** The `RagDocumentFragmentsController` now checks the `ai_rag_pdf_images_enabled` setting and allows PDF, PNG, JPG, and JPEG files if enabled.  Error handling is included for cases where PDF/image indexing is attempted with the setting disabled.
-   **PDF Processing:** Adds a new utility class, `DiscourseAi::Utils::PdfToImages`, which uses ImageMagick (`magick`) to convert PDF pages into individual PNG images. A maximum PDF size and conversion timeout are enforced.
-   **Image Processing:** A new utility class, `DiscourseAi::Utils::ImageToText`, is included to handle OCR for the images and PDFs.
-   **RAG Digestion Job:** The `DigestRagUpload` job now handles PDF and image uploads. It uses `PdfToImages` and `ImageToText` to extract text and create document fragments.
-   **UI Updates:**  The RAG uploader component now accepts PDF and image file types if `ai_rag_pdf_images_enabled` is true. The UI text is adjusted to indicate supported file types.

**3. Refactoring and Improvements:**

-   **LLM Enumeration:** The `DiscourseAi::Configuration::LlmEnumerator` now provides a `values_for_serialization` method, which returns a simplified array of LLM data (id, name, vision_enabled) suitable for use in serializers. This avoids exposing unnecessary details to the frontend.
-   **AI Helper:** The `AiHelper::Assistant` now takes optional `helper_llm` and `image_caption_llm` parameters in its constructor, allowing for greater flexibility.
-   **Bot and Persona Updates:** Several updates were made across the codebase, changing the string based association to a LLM to the new model based.
-   **Audit Logs:** The `DiscourseAi::Completions::Endpoints::Base` now formats raw request payloads as pretty JSON for easier auditing.
- **Eval Script:** An evaluation script is included.

**4. Testing:**

-    The PR introduces a new eval system for LLMs, this allows us to test how functionality works across various LLM providers. This lives in `/evals`
This commit is contained in:
Sam
2025-02-14 12:15:07 +11:00
committed by GitHub
parent e2afbc26d3
commit 5e80f93e4c
54 changed files with 1329 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+2
View File
@@ -2,3 +2,5 @@ node_modules
/gems
/auto_generated
.env
evals/log
evals/cases
admin/assets/javascripts/discourse/routes/admin-plugins-show-discourse-ai-tools-edit.js
+2
View File
@@ -14,5 +14,7 @@ export default class DiscourseAiToolsEditRoute extends DiscourseRoute {
controller.set("allTools", toolsModel);
controller.set("presets", toolsModel.resultSetMeta.presets);
controller.set("llms", toolsModel.resultSetMeta.llms);
controller.set("settings", toolsModel.resultSetMeta.settings);
}
}
admin/assets/javascripts/discourse/routes/admin-plugins-show-discourse-ai-tools-new.js
+2
View File
@@ -11,5 +11,7 @@ export default class DiscourseAiToolsNewRoute extends DiscourseRoute {
controller.set("allTools", toolsModel);
controller.set("presets", toolsModel.resultSetMeta.presets);
controller.set("llms", toolsModel.resultSetMeta.llms);
controller.set("settings", toolsModel.resultSetMeta.settings);
}
}
admin/assets/javascripts/discourse/templates/admin-plugins/show/discourse-ai-tools/edit.hbs
+2
View File
@@ -3,5 +3,7 @@
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
@llms={{this.llms}}
@settings={{this.settings}}
/>
</section>
admin/assets/javascripts/discourse/templates/admin-plugins/show/discourse-ai-tools/new.hbs
+2
View File
@@ -3,5 +3,7 @@
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
@llms={{this.llms}}
@settings={{this.settings}}
/>
</section>
app/controllers/discourse_ai/admin/ai_personas_controller.rb
+16 -6
View File
@@ -32,10 +32,19 @@ module DiscourseAi
}
end
llms =
DiscourseAi::Configuration::LlmEnumerator
.values(allowed_seeded_llms: SiteSetting.ai_bot_allowed_seeded_models)
.map { |hash| { id: hash[:value], name: hash[:name] } }
render json: { ai_personas: ai_personas, meta: { tools: tools, llms: llms } }
DiscourseAi::Configuration::LlmEnumerator.values_for_serialization(
allowed_seeded_llm_ids: SiteSetting.ai_bot_allowed_seeded_models_map,
)
render json: {
ai_personas: ai_personas,
meta: {
tools: tools,
llms: llms,
settings: {
rag_pdf_images_enabled: SiteSetting.ai_rag_pdf_images_enabled,
},
},
}
end
def new
@@ -187,7 +196,7 @@ module DiscourseAi
:priority,
:top_p,
:temperature,
:default_llm,
:default_llm_id,
:user_id,
:max_context_posts,
:vision_enabled,
@@ -195,7 +204,8 @@ module DiscourseAi
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_conversation_chunks,
:question_consolidator_llm,
:rag_llm_model_id,
:question_consolidator_llm_id,
:allow_chat_channel_mentions,
:allow_chat_direct_messages,
:allow_topic_mentions,
app/controllers/discourse_ai/admin/ai_tools_controller.rb
+1
View File
@@ -90,6 +90,7 @@ module DiscourseAi
:summary,
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_llm_model_id,
rag_uploads: [:id],
parameters: [:name, :type, :description, :required, enum: []],
)
app/controllers/discourse_ai/admin/rag_document_fragments_controller.rb
+1
View File
@@ -49,6 +49,7 @@ module DiscourseAi
def validate_extension!(filename)
extension = File.extname(filename)[1..-1] || ""
authorized_extensions = %w[txt md]
authorized_extensions.concat(%w[pdf png jpg jpeg]) if SiteSetting.ai_rag_pdf_images_enabled
if !authorized_extensions.include?(extension)
raise Discourse::InvalidParameters.new(
I18n.t(
app/jobs/regular/digest_rag_upload.rb
+33 -2
View File
@@ -28,7 +28,7 @@ module ::Jobs
# Check if this is the first time we process this upload.
if fragment_ids.empty?
document = get_uploaded_file(upload)
document = get_uploaded_file(upload: upload, target: target)
return if document.nil?
RagDocumentFragment.publish_status(upload, { total: 0, indexed: 0, left: 0 })
@@ -163,7 +163,38 @@ module ::Jobs
[buffer, split_char]
end
def get_uploaded_file(upload)
def get_uploaded_file(upload:, target:)
if %w[pdf png jpg jpeg].include?(upload.extension) && !SiteSetting.ai_rag_pdf_images_enabled
raise Discourse::InvalidAccess.new(
"The setting ai_rag_pdf_images_enabled is false, can not index images and pdfs.",
)
end
if upload.extension == "pdf"
pages =
DiscourseAi::Utils::PdfToImages.new(
upload: upload,
user: Discourse.system_user,
).uploaded_pages
return(
DiscourseAi::Utils::ImageToText.as_fake_file(
uploads: pages,
llm_model: target.rag_llm_model,
user: Discourse.system_user,
)
)
end
if %w[png jpg jpeg].include?(upload.extension)
return(
DiscourseAi::Utils::ImageToText.as_fake_file(
uploads: [upload],
llm_model: target.rag_llm_model,
user: Discourse.system_user,
)
)
end
store = Discourse.store
@file ||=
if store.external?
app/models/ai_persona.rb
+46 -42
View File
@@ -1,8 +1,8 @@
# frozen_string_literal: true
class AiPersona < ActiveRecord::Base
# TODO remove this line 01-1-2025
self.ignored_columns = %i[commands allow_chat mentionable]
# TODO remove this line 01-10-2025
self.ignored_columns = %i[default_llm question_consolidator_llm]
# places a hard limit, so per site we cache a maximum of 500 classes
MAX_PERSONAS_PER_SITE = 500
@@ -12,7 +12,7 @@ class AiPersona < ActiveRecord::Base
validates :system_prompt, presence: true, length: { maximum: 10_000_000 }
validate :system_persona_unchangeable, on: :update, if: :system
validate :chat_preconditions
validate :allowed_seeded_model, if: :default_llm
validate :allowed_seeded_model, if: :default_llm_id
validates :max_context_posts, numericality: { greater_than: 0 }, allow_nil: true
# leaves some room for growth but sets a maximum to avoid memory issues
# we may want to revisit this in the future
@@ -30,6 +30,10 @@ class AiPersona < ActiveRecord::Base
belongs_to :created_by, class_name: "User"
belongs_to :user
belongs_to :default_llm, class_name: "LlmModel"
belongs_to :question_consolidator_llm, class_name: "LlmModel"
belongs_to :rag_llm_model, class_name: "LlmModel"
has_many :upload_references, as: :target, dependent: :destroy
has_many :uploads, through: :upload_references
@@ -62,7 +66,7 @@ class AiPersona < ActiveRecord::Base
user_id: persona.user_id,
username: persona.user.username_lower,
allowed_group_ids: persona.allowed_group_ids,
default_llm: persona.default_llm,
default_llm_id: persona.default_llm_id,
force_default_llm: persona.force_default_llm,
allow_chat_channel_mentions: persona.allow_chat_channel_mentions,
allow_chat_direct_messages: persona.allow_chat_direct_messages,
@@ -157,12 +161,12 @@ class AiPersona < ActiveRecord::Base
user_id
system
mentionable
default_llm
default_llm_id
max_context_posts
vision_enabled
vision_max_pixels
rag_conversation_chunks
question_consolidator_llm
question_consolidator_llm_id
allow_chat_channel_mentions
allow_chat_direct_messages
allow_topic_mentions
@@ -302,7 +306,7 @@ class AiPersona < ActiveRecord::Base
if (
allow_chat_channel_mentions || allow_chat_direct_messages || allow_topic_mentions ||
force_default_llm
) && !default_llm
) && !default_llm_id
errors.add(:default_llm, I18n.t("discourse_ai.ai_bot.personas.default_llm_required"))
end
end
@@ -332,13 +336,12 @@ class AiPersona < ActiveRecord::Base
end
def allowed_seeded_model
return if default_llm.blank?
return if default_llm_id.blank?
llm = LlmModel.find_by(id: default_llm.split(":").last.to_i)
return if llm.nil?
return if !llm.seeded?
return if default_llm.nil?
return if !default_llm.seeded?
return if SiteSetting.ai_bot_allowed_seeded_models.include?(llm.id.to_s)
return if SiteSetting.ai_bot_allowed_seeded_models_map.include?(default_llm.id.to_s)
errors.add(:default_llm, I18n.t("discourse_ai.llm.configuration.invalid_seeded_model"))
end
@@ -348,36 +351,37 @@ end
#
# Table name: ai_personas
#
# id :bigint not null, primary key
# name :string(100) not null
# description :string(2000) not null
# system_prompt :string(10000000) not null
# allowed_group_ids :integer default([]), not null, is an Array
# created_by_id :integer
# enabled :boolean default(TRUE), not null
# created_at :datetime not null
# updated_at :datetime not null
# system :boolean default(FALSE), not null
# priority :boolean default(FALSE), not null
# temperature :float
# top_p :float
# user_id :integer
# default_llm :text
# max_context_posts :integer
# vision_enabled :boolean default(FALSE), not null
# vision_max_pixels :integer default(1048576), not null
# rag_chunk_tokens :integer default(374), not null
# rag_chunk_overlap_tokens :integer default(10), not null
# rag_conversation_chunks :integer default(10), not null
# question_consolidator_llm :text
# tool_details :boolean default(TRUE), not null
# tools :json not null
# forced_tool_count :integer default(-1), not null
# allow_chat_channel_mentions :boolean default(FALSE), not null
# allow_chat_direct_messages :boolean default(FALSE), not null
# allow_topic_mentions :boolean default(FALSE), not null
# allow_personal_messages :boolean default(TRUE), not null
# force_default_llm :boolean default(FALSE), not null
# id :bigint not null, primary key
# name :string(100) not null
# description :string(2000) not null
# system_prompt :string(10000000) not null
# allowed_group_ids :integer default([]), not null, is an Array
# created_by_id :integer
# enabled :boolean default(TRUE), not null
# created_at :datetime not null
# updated_at :datetime not null
# system :boolean default(FALSE), not null
# priority :boolean default(FALSE), not null
# temperature :float
# top_p :float
# user_id :integer
# max_context_posts :integer
# vision_enabled :boolean default(FALSE), not null
# vision_max_pixels :integer default(1048576), not null
# rag_chunk_tokens :integer default(374), not null
# rag_chunk_overlap_tokens :integer default(10), not null
# rag_conversation_chunks :integer default(10), not null
# tool_details :boolean default(TRUE), not null
# tools :json not null
# forced_tool_count :integer default(-1), not null
# allow_chat_channel_mentions :boolean default(FALSE), not null
# allow_chat_direct_messages :boolean default(FALSE), not null
# allow_topic_mentions :boolean default(FALSE), not null
# allow_personal_messages :boolean default(TRUE), not null
# force_default_llm :boolean default(FALSE), not null
# rag_llm_model_id :bigint
# default_llm_id :bigint
# question_consolidator_llm_id :bigint
#
# Indexes
#
app/models/ai_tool.rb
+2 -1
View File
@@ -8,6 +8,7 @@ class AiTool < ActiveRecord::Base
validates :script, presence: true, length: { maximum: 100_000 }
validates :created_by_id, presence: true
belongs_to :created_by, class_name: "User"
belongs_to :rag_llm_model, class_name: "LlmModel"
has_many :rag_document_fragments, dependent: :destroy, as: :target
has_many :upload_references, as: :target, dependent: :destroy
has_many :uploads, through: :upload_references
@@ -371,4 +372,4 @@ end
# rag_chunk_tokens :integer default(374), not null
# rag_chunk_overlap_tokens :integer default(10), not null
# tool_name :string(100) default(""), not null
#
# rag_llm_model_id :bigint
app/models/llm_model.rb
+1 -1
View File
@@ -70,7 +70,7 @@ class LlmModel < ActiveRecord::Base
end
def to_llm
DiscourseAi::Completions::Llm.proxy(identifier)
DiscourseAi::Completions::Llm.proxy(self)
end
def identifier
app/serializers/ai_custom_tool_list_serializer.rb
+7 -1
View File
@@ -6,7 +6,13 @@ class AiCustomToolListSerializer < ApplicationSerializer
has_many :ai_tools, serializer: AiCustomToolSerializer, embed: :objects
def meta
{ presets: AiTool.presets }
{
presets: AiTool.presets,
llms: DiscourseAi::Configuration::LlmEnumerator.values_for_serialization,
settings: {
rag_pdf_images_enabled: SiteSetting.ai_rag_pdf_images_enabled,
},
}
end
def ai_tools
app/serializers/ai_custom_tool_serializer.rb
+1
View File
@@ -10,6 +10,7 @@ class AiCustomToolSerializer < ApplicationSerializer
:script,
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_llm_model_id,
:created_by_id,
:created_at,
:updated_at
app/serializers/localized_ai_persona_serializer.rb
+3 -2
View File
@@ -14,7 +14,7 @@ class LocalizedAiPersonaSerializer < ApplicationSerializer
:allowed_group_ids,
:temperature,
:top_p,
:default_llm,
:default_llm_id,
:user_id,
:max_context_posts,
:vision_enabled,
@@ -22,7 +22,8 @@ class LocalizedAiPersonaSerializer < ApplicationSerializer
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_conversation_chunks,
:question_consolidator_llm,
:rag_llm_model_id,
:question_consolidator_llm_id,
:tool_details,
:forced_tool_count,
:allow_chat_channel_mentions,
assets/javascripts/discourse/admin/models/ai-persona.js
+6 -4
View File
@@ -15,7 +15,7 @@ const CREATE_ATTRIBUTES = [
"top_p",
"temperature",
"user_id",
"default_llm",
"default_llm_id",
"force_default_llm",
"user",
"max_context_posts",
@@ -25,7 +25,8 @@ const CREATE_ATTRIBUTES = [
"rag_chunk_tokens",
"rag_chunk_overlap_tokens",
"rag_conversation_chunks",
"question_consolidator_llm",
"rag_llm_model_id",
"question_consolidator_llm_id",
"allow_chat",
"tool_details",
"forced_tool_count",
@@ -43,7 +44,7 @@ const SYSTEM_ATTRIBUTES = [
"priority",
"tools",
"user_id",
"default_llm",
"default_llm_id",
"force_default_llm",
"user",
"max_context_posts",
@@ -53,7 +54,8 @@ const SYSTEM_ATTRIBUTES = [
"rag_chunk_tokens",
"rag_chunk_overlap_tokens",
"rag_conversation_chunks",
"question_consolidator_llm",
"rag_llm_model_id",
"question_consolidator_llm_id",
"tool_details",
"allow_personal_messages",
"allow_topic_mentions",
assets/javascripts/discourse/admin/models/ai-tool.js
+1
View File
@@ -12,6 +12,7 @@ const CREATE_ATTRIBUTES = [
"rag_uploads",
"rag_chunk_tokens",
"rag_chunk_overlap_tokens",
"rag_llm_model_id",
"enabled",
];
assets/javascripts/discourse/components/ai-persona-editor.gjs
+12 -7
View File
@@ -167,27 +167,27 @@ export default class PersonaEditor extends Component {
}
get mappedQuestionConsolidatorLlm() {
return this.editingModel?.question_consolidator_llm || "blank";
return this.editingModel?.question_consolidator_llm_id ?? "blank";
}
set mappedQuestionConsolidatorLlm(value) {
if (value === "blank") {
this.editingModel.question_consolidator_llm = null;
this.editingModel.question_consolidator_llm_id = null;
} else {
this.editingModel.question_consolidator_llm = value;
this.editingModel.question_consolidator_llm_id = value;
}
}
get mappedDefaultLlm() {
return this.editingModel?.default_llm || "blank";
return this.editingModel?.default_llm_id ?? "blank";
}
set mappedDefaultLlm(value) {
if (value === "blank") {
this.editingModel.default_llm = null;
this.editingModel.default_llm_id = null;
this.hasDefaultLlm = false;
} else {
this.editingModel.default_llm = value;
this.editingModel.default_llm_id = value;
this.hasDefaultLlm = true;
}
}
@@ -596,9 +596,14 @@ export default class PersonaEditor extends Component {
@target={{this.editingModel}}
@updateUploads={{this.updateUploads}}
@onRemove={{this.removeUpload}}
@allowPdfsAndImages={{@personas.resultSetMeta.settings.rag_pdf_images_enabled}}
/>
</div>
<RagOptions @model={{this.editingModel}}>
<RagOptions
@model={{this.editingModel}}
@llms={{@personas.resultSetMeta.llms}}
@allowPdfsAndImages={{@personas.resultSetMeta.settings.rag_pdf_images_enabled}}
>
<div class="control-group">
<label>{{i18n
"discourse_ai.ai_persona.rag_conversation_chunks"
assets/javascripts/discourse/components/ai-tool-editor.gjs
+8 -2
View File
@@ -90,7 +90,8 @@ export default class AiToolEditor extends Component {
"summary",
"rag_uploads",
"rag_chunk_tokens",
"rag_chunk_overlap_tokens"
"rag_chunk_overlap_tokens",
"rag_llm_model_id"
);
await this.args.model.save(data);
@@ -244,9 +245,14 @@ export default class AiToolEditor extends Component {
@target={{this.editingModel}}
@updateUploads={{this.updateUploads}}
@onRemove={{this.removeUpload}}
@allowPdfsAndImages={{@settings.rag_pdf_images_enabled}}
/>
</div>
<RagOptions @model={{this.editingModel}} />
<RagOptions
@model={{this.editingModel}}
@llms={{@llms}}
@allowPdfsAndImages={{@settings.rag_pdf_images_enabled}}
/>
{{/if}}
<div class="control-group ai-tool-editor__action_panel">
assets/javascripts/discourse/components/rag-options.gjs
+31
View File
@@ -5,6 +5,7 @@ import { on } from "@ember/modifier";
import { action } from "@ember/object";
import DTooltip from "discourse/components/d-tooltip";
import { i18n } from "discourse-i18n";
import AiLlmSelector from "./ai-llm-selector";
export default class RagOptions extends Component {
@tracked showIndexingOptions = false;
@@ -22,6 +23,22 @@ export default class RagOptions extends Component {
: i18n("discourse_ai.rag.options.show_indexing_options");
}
get visionLlms() {
return this.args.llms.filter((llm) => llm.vision_enabled);
}
get visionLlmId() {
return this.args.model.rag_llm_model_id ?? "blank";
}
set visionLlmId(value) {
if (value === "blank") {
this.args.model.rag_llm_model_id = null;
} else {
this.args.model.rag_llm_model_id = value;
}
}
<template>
{{#if @model.rag_uploads}}
<a
@@ -64,6 +81,20 @@ export default class RagOptions extends Component {
}}
/>
</div>
{{#if @allowPdfsAndImages}}
<div class="control-group">
<label>{{i18n "discourse_ai.rag.options.rag_llm_model"}}</label>
<AiLlmSelector
class="ai-persona-editor__llms"
@value={{this.visionLlmId}}
@llms={{this.visionLlms}}
/>
<DTooltip
@icon="circle-question"
@content={{i18n "discourse_ai.rag.options.rag_llm_model_help"}}
/>
</div>
{{/if}}
{{yield}}
{{/if}}
</template>
assets/javascripts/discourse/components/rag-uploader.gjs
+14 -2
View File
@@ -77,6 +77,14 @@ export default class RagUploader extends Component {
this.updateUploads(this.ragUploads);
}
get acceptedFileTypes() {
if (this.args?.allowPdfsAndImages) {
return ".txt,.md,.pdf,.png,.jpg,.jpeg";
} else {
return ".txt,.md";
}
}
@action
submitFiles() {
this.uppyUpload.openPicker();
@@ -119,7 +127,11 @@ export default class RagUploader extends Component {
<template>
<div class="rag-uploader">
<h3>{{i18n "discourse_ai.rag.uploads.title"}}</h3>
<p>{{i18n "discourse_ai.rag.uploads.description"}}</p>
{{#if @allowPdfsAndImages}}
<p>{{i18n "discourse_ai.rag.uploads.description_with_pdfs"}}</p>
{{else}}
<p>{{i18n "discourse_ai.rag.uploads.description"}}</p>
{{/if}}
{{#if this.ragUploads}}
<div class="rag-uploader__search-input-container">
@@ -187,7 +199,7 @@ export default class RagUploader extends Component {
disabled={{this.uploading}}
type="file"
multiple="multiple"
accept=".txt,.md"
accept={{this.acceptedFileTypes}}
/>
<DButton
@label="discourse_ai.rag.uploads.button"
config/locales/client.en.yml
+4 -1
View File
@@ -274,11 +274,14 @@ en:
rag_chunk_tokens_help: "The number of tokens to use for each chunk in the RAG model. Increase to increase the amount of context the AI can use. (changing will re-index all uploads)"
rag_chunk_overlap_tokens: "Upload chunk overlap tokens"
rag_chunk_overlap_tokens_help: "The number of tokens to overlap between chunks in the RAG model. (changing will re-index all uploads)"
rag_llm_model: "Indexing Language Model"
rag_llm_model_help: "The language model used for OCR during indexing of PDFs and images"
show_indexing_options: "Show upload options"
hide_indexing_options: "Hide upload options"
uploads:
title: "Uploads"
description: "Uploaded files should be formatted as plaintext (.txt) or markdown (.md)."
description: "Plaintext (.txt) or markdown (.md)"
description_with_pdfs: "Plaintext (.txt), markdown (.md), PDF (.pdf) or image (.png, .jpeg)"
button: "Add files"
filter: "Filter uploads"
indexed: "Indexed"
config/settings.yml
+4 -2
View File
@@ -347,12 +347,14 @@ discourse_ai:
ai_spam_detection_enabled:
default: false
hidden: true
ai_spam_detection_user_id:
default: ""
hidden: true
ai_spam_detection_model_allowed_seeded_models:
default: ""
hidden: true
type: list
ai_rag_pdf_images_enabled:
default: false
hidden: true
db/migrate/20250210024600_add_rag_llm_model.rb
+7
View File
@@ -0,0 +1,7 @@
# frozen_string_literal: true
class AddRagLlmModel < ActiveRecord::Migration[7.2]
def change
add_column :ai_personas, :rag_llm_model_id, :bigint
add_column :ai_tools, :rag_llm_model_id, :bigint
end
end
db/migrate/20250210032345_migrate_persona_to_llm_model_id.rb
+19
View File
@@ -0,0 +1,19 @@
# frozen_string_literal: true
class MigratePersonaToLlmModelId < ActiveRecord::Migration[7.2]
def up
add_column :ai_personas, :default_llm_id, :bigint
add_column :ai_personas, :question_consolidator_llm_id, :bigint
# personas are seeded, we do not mark stuff as readonline
execute <<~SQL
UPDATE ai_personas
set
default_llm_id = (select id from llm_models where ('custom:' || id) = default_llm),
question_consolidator_llm_id = (select id from llm_models where ('custom:' || id) = question_consolidator_llm)
SQL
end
def down
raise ActiveRecord::IrreversibleMigration
end
end
db/post_migrate/20250210032351_post_migrate_persona_to_llm_model_id.rb
+11
View File
@@ -0,0 +1,11 @@
# frozen_string_literal: true
class PostMigratePersonaToLlmModelId < ActiveRecord::Migration[7.2]
def up
remove_column :ai_personas, :default_llm
remove_column :ai_personas, :question_consolidator_llm
end
def down
raise ActiveRecord::IrreversibleMigration
end
end
evals/lib/boot.rb
+36
View File
@@ -0,0 +1,36 @@
# frozen_string_literal: true
# got to ensure evals are here
# rubocop:disable Discourse/Plugins/NamespaceConstants
EVAL_PATH = File.join(__dir__, "../cases")
# rubocop:enable Discourse/Plugins/NamespaceConstants
#
if !Dir.exist?(EVAL_PATH)
puts "Evals are missing, cloning from discourse/discourse-ai-evals"
success =
system("git clone [email protected]:discourse/discourse-ai-evals.git '#{EVAL_PATH}' 2>/dev/null")
# Fall back to HTTPS if SSH fails
if !success
puts "SSH clone failed, falling back to HTTPS..."
success = system("git clone https://github.com/discourse/discourse-ai-evals.git '#{EVAL_PATH}'")
end
if success
puts "Successfully cloned evals repository"
else
abort "Failed to clone evals repository"
end
end
discourse_path = File.expand_path(File.join(__dir__, "../../../.."))
# rubocop:disable Discourse/NoChdir
Dir.chdir(discourse_path)
# rubocop:enable Discourse/NoChdir
require "/home/sam/Source/discourse/config/environment"
ENV["DISCOURSE_AI_NO_DEBUG"] = "1"
module DiscourseAi::Evals
end
evals/lib/cli.rb
+47
View File
@@ -0,0 +1,47 @@
# frozen_string_literal: true
require "optparse"
class DiscourseAi::Evals::Cli
class Options
attr_accessor :eval_name, :model, :list, :list_models
def initialize(eval_name: nil, model: nil, list: false, list_models: false)
@eval_name = eval_name
@model = model
@list = list
@list_models = list_models
end
end
def self.parse_options!
options = Options.new
parser =
OptionParser.new do |opts|
opts.banner = "Usage: evals/run [options]"
opts.on("-e", "--eval NAME", "Name of the evaluation to run") do |eval_name|
options.eval_name = eval_name
end
opts.on("--list-models", "List models") { |model| options.list_models = true }
opts.on(
"-m",
"--model NAME",
"Model to evaluate (will eval all models if not specified)",
) { |model| options.model = model }
opts.on("-l", "--list", "List evals") { |model| options.list = true }
end
show_help = ARGV.empty?
parser.parse!
if show_help
puts parser
exit 0
end
options
end
end
evals/lib/eval.rb
+136
View File
@@ -0,0 +1,136 @@
#frozen_string_literal: true
class DiscourseAi::Evals::Eval
attr_reader :type,
:path,
:name,
:description,
:id,
:args,
:vision,
:expected_output,
:expected_output_regex
def initialize(path:)
@yaml = YAML.load_file(path).symbolize_keys
@path = path
@name = @yaml[:name]
@id = @yaml[:id]
@description = @yaml[:description]
@vision = @yaml[:vision]
@args = @yaml[:args]&.symbolize_keys
@type = @yaml[:type]
@expected_output = @yaml[:expected_output]
@expected_output_regex = @yaml[:expected_output_regex]
@expected_output_regex =
Regexp.new(@expected_output_regex, Regexp::MULTILINE) if @expected_output_regex
@args[:path] = File.expand_path(File.join(File.dirname(path), @args[:path])) if @args&.key?(
:path,
)
end
def run(llm:)
result =
case type
when "helper"
helper(llm, **args)
when "pdf_to_text"
pdf_to_text(llm, **args)
when "image_to_text"
image_to_text(llm, **args)
end
if expected_output
if result == expected_output
{ result: :pass }
else
{ result: :fail, expected_output: expected_output, actual_output: result }
end
elsif expected_output_regex
if result.match?(expected_output_regex)
{ result: :pass }
else
{ result: :fail, expected_output: expected_output_regex, actual_output: result }
end
else
{ result: :unknown, actual_output: result }
end
end
def print
puts "#{id}: #{description}"
end
def to_json
{
type: @type,
path: @path,
name: @name,
description: @description,
id: @id,
args: @args,
vision: @vision,
expected_output: @expected_output,
expected_output_regex: @expected_output_regex,
}.compact
end
private
def helper(llm, input:, name:)
completion_prompt = CompletionPrompt.find_by(name: name)
helper = DiscourseAi::AiHelper::Assistant.new(helper_llm: llm.llm_proxy)
result =
helper.generate_and_send_prompt(
completion_prompt,
input,
current_user = Discourse.system_user,
_force_default_locale = false,
)
result[:suggestions].first
end
def image_to_text(llm, path:)
upload =
UploadCreator.new(File.open(path), File.basename(path)).create_for(Discourse.system_user.id)
text = +""
DiscourseAi::Utils::ImageToText
.new(upload: upload, llm_model: llm.llm_model, user: Discourse.system_user)
.extract_text do |chunk, error|
text << chunk if chunk
text << "\n\n" if chunk
end
text
ensure
upload.destroy if upload
end
def pdf_to_text(llm, path:)
upload =
UploadCreator.new(File.open(path), File.basename(path)).create_for(Discourse.system_user.id)
uploads =
DiscourseAi::Utils::PdfToImages.new(
upload: upload,
user: Discourse.system_user,
).uploaded_pages
text = +""
uploads.each do |page_upload|
DiscourseAi::Utils::ImageToText
.new(upload: page_upload, llm_model: llm.llm_model, user: Discourse.system_user)
.extract_text do |chunk, error|
text << chunk if chunk
text << "\n\n" if chunk
end
upload.destroy
end
text
ensure
upload.destroy if upload
end
end
evals/lib/llm.rb
+129
View File
@@ -0,0 +1,129 @@
# frozen_string_literal: true
class DiscourseAi::Evals::Llm
CONFIGS = {
"gpt-4o" => {
display_name: "GPT-4o",
name: "gpt-4o",
tokenizer: "DiscourseAi::Tokenizer::OpenAiTokenizer",
api_key_env: "OPENAI_API_KEY",
provider: "open_ai",
url: "https://api.openai.com/v1/chat/completions",
max_prompt_tokens: 131_072,
vision_enabled: true,
},
"gpt-4o-mini" => {
display_name: "GPT-4o-mini",
name: "gpt-4o-mini",
tokenizer: "DiscourseAi::Tokenizer::OpenAiTokenizer",
api_key_env: "OPENAI_API_KEY",
provider: "open_ai",
url: "https://api.openai.com/v1/chat/completions",
max_prompt_tokens: 131_072,
vision_enabled: true,
},
"claude-3.5-haiku" => {
display_name: "Claude 3.5 Haiku",
name: "claude-3-5-haiku-latest",
tokenizer: "DiscourseAi::Tokenizer::AnthropicTokenizer",
api_key_env: "ANTHROPIC_API_KEY",
provider: "anthropic",
url: "https://api.anthropic.com/v1/messages",
max_prompt_tokens: 200_000,
vision_enabled: false,
},
"claude-3.5-sonnet" => {
display_name: "Claude 3.5 Sonnet",
name: "claude-3-5-sonnet-latest",
tokenizer: "DiscourseAi::Tokenizer::AnthropicTokenizer",
api_key_env: "ANTHROPIC_API_KEY",
provider: "anthropic",
url: "https://api.anthropic.com/v1/messages",
max_prompt_tokens: 200_000,
vision_enabled: true,
},
"gemini-2.0-flash" => {
display_name: "Gemini 2.0 Flash",
name: "gemini-2-0-flash",
tokenizer: "DiscourseAi::Tokenizer::GeminiTokenizer",
api_key_env: "GEMINI_API_KEY",
provider: "google",
url: "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash",
max_prompt_tokens: 1_000_000,
vision_enabled: true,
},
"gemini-2.0-pro-exp" => {
display_name: "Gemini 2.0 pro",
name: "gemini-2-0-pro-exp",
tokenizer: "DiscourseAi::Tokenizer::GeminiTokenizer",
api_key_env: "GEMINI_API_KEY",
provider: "google",
url: "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-pro-exp",
max_prompt_tokens: 1_000_000,
vision_enabled: true,
},
}
def self.print
CONFIGS
.keys
.map do |config_name|
begin
new(config_name)
rescue StandardError
nil
end
end
.compact
.each { |llm| puts "#{llm.config_name}: #{llm.name} (#{llm.provider})" }
end
def self.choose(config_name)
if CONFIGS[config_name].nil?
CONFIGS
.keys
.map do |config_name|
begin
new(config_name)
rescue => e
puts "Error initializing #{config_name}: #{e}"
nil
end
end
.compact
elsif !CONFIGS.include?(config_name)
raise "Invalid llm"
else
[new(config_name)]
end
end
attr_reader :llm_model
attr_reader :llm_proxy
attr_reader :config_name
def initialize(config_name)
config = CONFIGS[config_name].dup
api_key_env = config.delete(:api_key_env)
if !ENV[api_key_env]
raise "Missing API key for #{config_name}, should be set via #{api_key_env}"
end
config[:api_key] = ENV[api_key_env]
@llm_model = LlmModel.new(config)
@llm_proxy = DiscourseAi::Completions::Llm.proxy(@llm_model)
@config_name = config_name
end
def provider
@llm_model.provider
end
def name
@llm_model.display_name
end
def vision?
@llm_model.vision_enabled
end
end

Some files were not shown because too many files have changed in this diff Show More