Discourse/discourse-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FEATURE: flexible image handling within messages (#1214)

Browse Source 
* DEV: refactor bot internals

This introduces a proper object for bot context, this makes
it simpler to improve context management as we go cause we
have a nice object to work with

Starts refactoring allowing for a single message to have
multiple uploads throughout

* transplant method to message builder

* chipping away at inline uploads

* image support is improved but not fully fixed yet

partially working in anthropic, still got quite a few dialects to go

* open ai and claude are now working

* Gemini is now working as well

* fix nova

* more dialects...

* fix ollama

* fix specs

* update artifact fixed

* more tests

* spam scanner

* pass more specs

* bunch of specs improved

* more bug fixes.

* all the rest of the tests are working

* improve tests coverage and ensure custom tools are aware of new context object

* tests are working, but we need more tests

* resolve merge conflict

* new preamble and expanded specs on ai tool

* remove concept of "standalone tools"

This is no longer needed, we can set custom raw, tool details are injected into tool calls
This commit is contained in:
Sam
2025-03-31 12:39:07 -03:00
committed by GitHub
parent f3e78f0d80
commit 5b6d39a206
53 changed files with 1381 additions and 723 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/controllers/discourse_ai/admin/ai_tools_controller.rb
+1 -1
View File
@@ -55,7 +55,7 @@ module DiscourseAi
# we need an llm so we have a tokenizer
# but will do without if none is available
llm = LlmModel.first&.to_llm
runner = @ai_tool.runner(parameters, llm: llm, bot_user: current_user, context: {})
runner = @ai_tool.runner(parameters, llm: llm, bot_user: current_user)
result = runner.invoke
if result.is_a?(Hash) && result[:error]
app/jobs/regular/stream_discover_reply.rb
+7 -3
View File
@@ -30,9 +30,13 @@ module Jobs
base = { query: query, model_used: llm_model.display_name }
bot.reply(
{ conversation_context: [{ type: :user, content: query }], skip_tool_details: true },
) do |partial|
context =
DiscourseAi::AiBot::BotContext.new(
messages: [{ type: :user, content: query }],
skip_tool_details: true,
)
bot.reply(context) do |partial|
streamed_reply << partial
# Throttle updates.
app/models/ai_tool.rb
+161 -81
View File
@@ -35,7 +35,7 @@ class AiTool < ActiveRecord::Base
tool_name.presence || name
end
def runner(parameters, llm:, bot_user:, context: {})
def runner(parameters, llm:, bot_user:, context: nil)
DiscourseAi::AiBot::ToolRunner.new(
parameters: parameters,
llm: llm,
@@ -59,86 +59,166 @@ class AiTool < ActiveRecord::Base
def self.preamble
<<~JS
/**
* Tool API Quick Reference
*
* Entry Functions
*
* invoke(parameters): Main function. Receives parameters (Object). Must return a JSON-serializable value.
* Example:
* function invoke(parameters) { return "result"; }
*
* details(): Optional. Returns a string describing the tool.
* Example:
* function details() { return "Tool description."; }
*
* Provided Objects
*
* 1. http
* http.get(url, options?): Performs an HTTP GET request.
* Parameters:
* url (string): The request URL.
* options (Object, optional):
* headers (Object): Request headers.
* Returns:
* { status: number, body: string }
*
* http.post(url, options?): Performs an HTTP POST request.
* Parameters:
* url (string): The request URL.
* options (Object, optional):
* headers (Object): Request headers.
* body (string): Request body.
* Returns:
* { status: number, body: string }
*
* (also available: http.put, http.patch, http.delete)
*
* Note: Max 20 HTTP requests per execution.
*
* 2. llm
* llm.truncate(text, length): Truncates text to a specified token length.
* Parameters:
* text (string): Text to truncate.
* length (number): Max tokens.
* Returns:
* Truncated string.
*
* 3. index
* index.search(query, options?): Searches indexed documents.
* Parameters:
* query (string): Search query.
* options (Object, optional):
* filenames (Array): Limit search to specific files.
* limit (number): Max fragments (up to 200).
* Returns:
* Array of { fragment: string, metadata: string }
*
* 4. upload
* upload.create(filename, base_64_content): Uploads a file.
* Parameters:
* filename (string): Name of the file.
* base_64_content (string): Base64 encoded file content.
* Returns:
* { id: number, short_url: string }
*
* 5. chain
* chain.setCustomRaw(raw): Sets the body of the post and exist chain.
* Parameters:
* raw (string): raw content to add to post.
*
* Constraints
*
* Execution Time: ≤ 2000ms
* Memory: ≤ 10MB
* HTTP Requests: ≤ 20 per execution
* Exceeding limits will result in errors or termination.
*
* Security
*
* Sandboxed Environment: No access to system or global objects.
* No File System Access: Cannot read or write files.
*/
/**
* Tool API Quick Reference
*
* Entry Functions
*
* invoke(parameters): Main function. Receives parameters defined in the tool's signature (Object).
* Must return a JSON-serializable value (e.g., string, number, object, array).
* Example:
* function invoke(parameters) { return { result: "Data processed", input: parameters.query }; }
*
* details(): Optional function. Returns a string (can include basic HTML) describing
* the tool's action after invocation, often using data from the invocation.
* This is displayed in the chat interface.
* Example:
* let lastUrl;
* function invoke(parameters) {
* lastUrl = parameters.url;
* // ... perform action ...
* return { success: true, content: "..." };
* }
* function details() {
* return `Browsed: <a href="${lastUrl}">${lastUrl}</a>`;
* }
*
* Provided Objects & Functions
*
* 1. http
* Performs HTTP requests. Max 20 requests per execution.
*
* http.get(url, options?): Performs GET request.
* Parameters:
* url (string): The request URL.
* options (Object, optional):
* headers (Object): Request headers (e.g., { "Authorization": "Bearer key" }).
* Returns: { status: number, body: string }
*
* http.post(url, options?): Performs POST request.
* Parameters:
* url (string): The request URL.
* options (Object, optional):
* headers (Object): Request headers.
* body (string | Object): Request body. If an object, it's stringified as JSON.
* Returns: { status: number, body: string }
*
* http.put(url, options?): Performs PUT request (similar to POST).
* http.patch(url, options?): Performs PATCH request (similar to POST).
* http.delete(url, options?): Performs DELETE request (similar to GET/POST).
*
* 2. llm
* Interacts with the Language Model.
*
* llm.truncate(text, length): Truncates text to a specified token length based on the configured LLM's tokenizer.
* Parameters:
* text (string): Text to truncate.
* length (number): Maximum number of tokens.
* Returns: string (truncated text)
*
* llm.generate(prompt): Generates text using the configured LLM associated with the tool runner.
* Parameters:
* prompt (string | Object): The prompt. Can be a simple string or an object
* like { messages: [{ type: "system", content: "..." }, { type: "user", content: "..." }] }.
* Returns: string (generated text)
*
* 3. index
* Searches attached RAG (Retrieval-Augmented Generation) documents linked to this tool.
*
* index.search(query, options?): Searches indexed document fragments.
* Parameters:
* query (string): The search query used for semantic search.
* options (Object, optional):
* filenames (Array<string>): Filter search to fragments from specific uploaded filenames.
* limit (number): Maximum number of fragments to return (default: 10, max: 200).
* Returns: Array<{ fragment: string, metadata: string | null }> - Ordered by relevance.
*
* 4. upload
* Handles file uploads within Discourse.
*
* upload.create(filename, base_64_content): Uploads a file created by the tool, making it available in Discourse.
* Parameters:
* filename (string): The desired name for the file (basename is used for security).
* base_64_content (string): Base64 encoded content of the file.
* Returns: { id: number, url: string, short_url: string } - Details of the created upload record.
*
* 5. chain
* Controls the execution flow.
*
* chain.setCustomRaw(raw): Sets the final raw content of the bot's post and immediately
* stops the tool execution chain. Useful for tools that directly
* generate the full response content (e.g., image generation tools attaching the image markdown).
* Parameters:
* raw (string): The raw Markdown content for the post.
* Returns: void
*
* 6. discourse
* Interacts with Discourse specific features. Access is generally performed as the SystemUser.
*
* discourse.search(params): Performs a Discourse search.
* Parameters:
* params (Object): Search parameters (e.g., { search_query: "keyword", with_private: true, max_results: 10 }).
* `with_private: true` searches across all posts visible to the SystemUser. `result_style: 'detailed'` is used by default.
* Returns: Object (Discourse search results structure, includes posts, topics, users etc.)
*
* discourse.getPost(post_id): Retrieves details for a specific post.
* Parameters:
* post_id (number): The ID of the post.
* Returns: Object (Post details including `raw`, nested `topic` object with ListableTopicSerializer structure) or null if not found/accessible.
*
* discourse.getTopic(topic_id): Retrieves details for a specific topic.
* Parameters:
* topic_id (number): The ID of the topic.
* Returns: Object (Topic details using ListableTopicSerializer structure) or null if not found/accessible.
*
* discourse.getUser(user_id_or_username): Retrieves details for a specific user.
* Parameters:
* user_id_or_username (number | string): The ID or username of the user.
* Returns: Object (User details using UserSerializer structure) or null if not found.
*
* discourse.getPersona(name): Gets an object representing another AI Persona configured on the site.
* Parameters:
* name (string): The name of the target persona.
* Returns: Object { respondTo: function(params) } or null if persona not found.
* respondTo(params): Instructs the target persona to generate a response within the current context (e.g., replying to the same post or chat message).
* Parameters:
* params (Object, optional): { instructions: string, whisper: boolean }
* Returns: { success: boolean, post_id?: number, post_number?: number, message_id?: number } or { error: string }
*
* discourse.createChatMessage(params): Creates a new message in a Discourse Chat channel.
* Parameters:
* params (Object): { channel_name: string, username: string, message: string }
* `channel_name` can be the channel name or slug.
* `username` specifies the user who should appear as the sender. The user must exist.
* The sending user must have permission to post in the channel.
* Returns: { success: boolean, message_id?: number, message?: string, created_at?: string } or { error: string }
*
* 7. context
* An object containing information about the environment where the tool is being run.
* Available properties depend on the invocation context, but may include:
* post_id (number): ID of the post triggering the tool (if in a Post context).
* topic_id (number): ID of the topic (if in a Post context).
* private_message (boolean): Whether the context is a private message (in Post context).
* message_id (number): ID of the chat message triggering the tool (if in Chat context).
* channel_id (number): ID of the chat channel (if in Chat context).
* user (Object): Details of the user invoking the tool/persona (structure may vary, often null or SystemUser details unless explicitly passed).
* participants (string): Comma-separated list of usernames in a PM (if applicable).
* // ... other potential context-specific properties added by the calling environment.
*
* Constraints
*
* Execution Time: ≤ 2000ms (default timeout in milliseconds) - This timer *pauses* during external HTTP requests or LLM calls initiated via `http.*` or `llm.generate`, but applies to the script's own processing time.
* Memory: ≤ 10MB (V8 heap limit)
* Stack Depth: ≤ 20 (Marshal stack depth limit for Ruby interop)
* HTTP Requests: ≤ 20 per execution
* Exceeding limits will result in errors or termination (e.g., timeout error, out-of-memory error, TooManyRequestsError).
*
* Security
*
* Sandboxed Environment: The script runs in a restricted V8 JavaScript environment (via MiniRacer).
* No direct access to browser or environment, browser globals (like `window` or `document`), or the host system's file system.
* Network requests are proxied through the Discourse backend, not made directly from the sandbox.
*/
JS
end
lib/ai_bot/bot.rb
+10 -19
View File
@@ -75,9 +75,9 @@ module DiscourseAi
def force_tool_if_needed(prompt, context)
return if prompt.tool_choice == :none
context[:chosen_tools] ||= []
context.chosen_tools ||= []
forced_tools = persona.force_tool_use.map { |tool| tool.name }
force_tool = forced_tools.find { |name| !context[:chosen_tools].include?(name) }
force_tool = forced_tools.find { |name| !context.chosen_tools.include?(name) }
if force_tool && persona.forced_tool_count > 0
user_turns = prompt.messages.select { |m| m[:type] == :user }.length
@@ -85,7 +85,7 @@ module DiscourseAi
end
if force_tool
context[:chosen_tools] << force_tool
context.chosen_tools << force_tool
prompt.tool_choice = force_tool
else
prompt.tool_choice = nil
@@ -93,6 +93,9 @@ module DiscourseAi
end
def reply(context, &update_blk)
unless context.is_a?(BotContext)
raise ArgumentError, "context must be an instance of BotContext"
end
llm = DiscourseAi::Completions::Llm.proxy(model)
prompt = persona.craft_prompt(context, llm: llm)
@@ -100,7 +103,7 @@ module DiscourseAi
ongoing_chain = true
raw_context = []
user = context[:user]
user = context.user
llm_kwargs = { user: user }
llm_kwargs[:temperature] = persona.temperature if persona.temperature
@@ -277,27 +280,15 @@ module DiscourseAi
name: tool.name,
}
if tool.standalone?
standalone_context =
context.dup.merge(
conversation_context: [
context[:conversation_context].last,
tool_call_message,
tool_message,
],
)
prompt = persona.craft_prompt(standalone_context)
else
prompt.push(**tool_call_message)
prompt.push(**tool_message)
end
prompt.push(**tool_call_message)
prompt.push(**tool_message)
raw_context << [tool_call_message[:content], tool_call_id, "tool_call", tool.name]
raw_context << [invocation_result_json, tool_call_id, "tool", tool.name]
end
def invoke_tool(tool, llm, cancel, context, &update_blk)
show_placeholder = !context[:skip_tool_details] && !tool.class.allow_partial_tool_calls?
show_placeholder = !context.skip_tool_details && !tool.class.allow_partial_tool_calls?
update_blk.call("", cancel, build_placeholder(tool.summary, "")) if show_placeholder
lib/ai_bot/bot_context.rb
+107
View File
@@ -0,0 +1,107 @@
# frozen_string_literal: true
module DiscourseAi
module AiBot
class BotContext
attr_accessor :messages,
:topic_id,
:post_id,
:private_message,
:custom_instructions,
:user,
:skip_tool_details,
:participants,
:chosen_tools,
:message_id,
:channel_id,
:context_post_ids
def initialize(
post: nil,
participants: nil,
user: nil,
skip_tool_details: nil,
messages: [],
custom_instructions: nil,
site_url: nil,
site_title: nil,
site_description: nil,
time: nil,
message_id: nil,
channel_id: nil,
context_post_ids: nil
)
@participants = participants
@user = user
@skip_tool_details = skip_tool_details
@messages = messages
@custom_instructions = custom_instructions
@message_id = message_id
@channel_id = channel_id
@context_post_ids = context_post_ids
@site_url = site_url
@site_title = site_title
@site_description = site_description
@time = time
if post
@post_id = post.id
@topic_id = post.topic_id
@private_message = post.topic.private_message?
@participants ||= post.topic.allowed_users.map(&:username).join(", ") if @private_message
@user = post.user
end
end
# these are strings that can be safely interpolated into templates
TEMPLATE_PARAMS = %w[time site_url site_title site_description participants]
def lookup_template_param(key)
public_send(key.to_sym) if TEMPLATE_PARAMS.include?(key)
end
def time
@time ||= Time.zone.now
end
def site_url
@site_url ||= Discourse.base_url
end
def site_title
@site_title ||= SiteSetting.title
end
def site_description
@site_description ||= SiteSetting.site_description
end
def private_message?
@private_message
end
def to_json
{
messages: @messages,
topic_id: @topic_id,
post_id: @post_id,
private_message: @private_message,
custom_instructions: @custom_instructions,
username: @user&.username,
user_id: @user&.id,
participants: @participants,
chosen_tools: @chosen_tools,
message_id: @message_id,
channel_id: @channel_id,
context_post_ids: @context_post_ids,
site_url: @site_url,
site_title: @site_title,
site_description: @site_description,
skip_tool_details: @skip_tool_details,
}
end
end
end
end
lib/ai_bot/personas/persona.rb
+8 -8
View File
@@ -163,7 +163,7 @@ module DiscourseAi
def craft_prompt(context, llm: nil)
system_insts =
system_prompt.gsub(/\{(\w+)\}/) do |match|
found = context[match[1..-2].to_sym]
found = context.lookup_template_param(match[1..-2])
found.nil? ? match : found.to_s
end
@@ -180,16 +180,16 @@ module DiscourseAi
)
end
if context[:custom_instructions].present?
if context.custom_instructions.present?
prompt_insts << "\n"
prompt_insts << context[:custom_instructions]
prompt_insts << context.custom_instructions
end
fragments_guidance =
rag_fragments_prompt(
context[:conversation_context].to_a,
context.messages,
llm: question_consolidator_llm,
user: context[:user],
user: context.user,
)&.strip
prompt_insts << fragments_guidance if fragments_guidance.present?
@@ -197,9 +197,9 @@ module DiscourseAi
prompt =
DiscourseAi::Completions::Prompt.new(
prompt_insts,
messages: context[:conversation_context].to_a,
topic_id: context[:topic_id],
post_id: context[:post_id],
messages: context.messages,
topic_id: context.topic_id,
post_id: context.post_id,
)
prompt.max_pixels = self.class.vision_max_pixels if self.class.vision_enabled
lib/ai_bot/playground.rb
+51 -186
View File
@@ -227,90 +227,17 @@ module DiscourseAi
schedule_bot_reply(post) if can_attach?(post)
end
def conversation_context(post, style: nil)
# Pay attention to the `post_number <= ?` here.
# We want to inject the last post as context because they are translated differently.
# also setting default to 40, allowing huge contexts costs lots of tokens
max_posts = 40
if bot.persona.class.respond_to?(:max_context_posts)
max_posts = bot.persona.class.max_context_posts || 40
end
post_types = [Post.types[:regular]]
post_types << Post.types[:whisper] if post.post_type == Post.types[:whisper]
context =
post
.topic
.posts
.joins(:user)
.joins("LEFT JOIN post_custom_prompts ON post_custom_prompts.post_id = posts.id")
.where("post_number <= ?", post.post_number)
.order("post_number desc")
.where("post_type in (?)", post_types)
.limit(max_posts)
.pluck(
"posts.raw",
"users.username",
"post_custom_prompts.custom_prompt",
"(
SELECT array_agg(ref.upload_id)
FROM upload_references ref
WHERE ref.target_type = 'Post' AND ref.target_id = posts.id
) as upload_ids",
)
builder = DiscourseAi::Completions::PromptMessagesBuilder.new
builder.topic = post.topic
context.reverse_each do |raw, username, custom_prompt, upload_ids|
custom_prompt_translation =
Proc.new do |message|
# We can't keep backwards-compatibility for stored functions.
# Tool syntax requires a tool_call_id which we don't have.
if message[2] != "function"
custom_context = {
content: message[0],
type: message[2].present? ? message[2].to_sym : :model,
}
custom_context[:id] = message[1] if custom_context[:type] != :model
custom_context[:name] = message[3] if message[3]
thinking = message[4]
custom_context[:thinking] = thinking if thinking
builder.push(**custom_context)
end
end
if custom_prompt.present?
custom_prompt.each(&custom_prompt_translation)
else
context = {
content: raw,
type: (available_bot_usernames.include?(username) ? :model : :user),
}
context[:id] = username if context[:type] == :user
if upload_ids.present? && context[:type] == :user && bot.persona.class.vision_enabled
context[:upload_ids] = upload_ids.compact
end
builder.push(**context)
end
end
builder.to_a(style: style || (post.topic.private_message? ? :bot : :topic))
end
def title_playground(post, user)
context = conversation_context(post)
messages =
DiscourseAi::Completions::PromptMessagesBuilder.messages_from_post(
post,
max_posts: 5,
bot_usernames: available_bot_usernames,
include_uploads: bot.persona.class.vision_enabled,
)
bot
.get_updated_title(context, post, user)
.get_updated_title(messages, post, user)
.tap do |new_title|
PostRevisor.new(post.topic.first_post, post.topic).revise!(
bot.bot_user,
@@ -326,83 +253,6 @@ module DiscourseAi
)
end
def chat_context(message, channel, persona_user, context_post_ids)
has_vision = bot.persona.class.vision_enabled
include_thread_titles = !channel.direct_message_channel? && !message.thread_id
current_id = message.id
if !channel.direct_message_channel?
# we are interacting via mentions ... strip mention
instruction_message = message.message.gsub(/@#{bot.bot_user.username}/i, "").strip
end
messages = nil
max_messages = 40
if bot.persona.class.respond_to?(:max_context_posts)
max_messages = bot.persona.class.max_context_posts || 40
end
if !message.thread_id && channel.direct_message_channel?
messages = [message]
elsif !channel.direct_message_channel? && !message.thread_id
messages =
Chat::Message
.joins("left join chat_threads on chat_threads.id = chat_messages.thread_id")
.where(chat_channel_id: channel.id)
.where(
"chat_messages.thread_id IS NULL OR chat_threads.original_message_id = chat_messages.id",
)
.order(id: :desc)
.limit(max_messages)
.to_a
.reverse
end
messages ||=
ChatSDK::Thread.last_messages(
thread_id: message.thread_id,
guardian: Discourse.system_user.guardian,
page_size: max_messages,
)
builder = DiscourseAi::Completions::PromptMessagesBuilder.new
guardian = Guardian.new(message.user)
if context_post_ids
builder.set_chat_context_posts(context_post_ids, guardian, include_uploads: has_vision)
end
messages.each do |m|
# restore stripped message
m.message = instruction_message if m.id == current_id && instruction_message
if available_bot_user_ids.include?(m.user_id)
builder.push(type: :model, content: m.message)
else
upload_ids = nil
upload_ids = m.uploads.map(&:id) if has_vision && m.uploads.present?
mapped_message = m.message
thread_title = nil
thread_title = m.thread&.title if include_thread_titles && m.thread_id
mapped_message = "(#{thread_title})\n#{m.message}" if thread_title
builder.push(
type: :user,
content: mapped_message,
name: m.user.username,
upload_ids: upload_ids,
)
end
end
builder.to_a(
limit: max_messages,
style: channel.direct_message_channel? ? :chat_with_context : :chat,
)
end
def reply_to_chat_message(message, channel, context_post_ids)
persona_user = User.find(bot.persona.class.user_id)
@@ -410,10 +260,32 @@ module DiscourseAi
context_post_ids = nil if !channel.direct_message_channel?
max_chat_messages = 40
if bot.persona.class.respond_to?(:max_context_posts)
max_chat_messages = bot.persona.class.max_context_posts || 40
end
if !channel.direct_message_channel?
# we are interacting via mentions ... strip mention
instruction_message = message.message.gsub(/@#{bot.bot_user.username}/i, "").strip
end
context =
get_context(
participants: participants.join(", "),
conversation_context: chat_context(message, channel, persona_user, context_post_ids),
BotContext.new(
participants: participants,
message_id: message.id,
channel_id: channel.id,
context_post_ids: context_post_ids,
messages:
DiscourseAi::Completions::PromptMessagesBuilder.messages_from_chat(
message,
channel: channel,
context_post_ids: context_post_ids,
include_uploads: bot.persona.class.vision_enabled,
max_messages: max_chat_messages,
bot_user_ids: available_bot_user_ids,
instruction_message: instruction_message,
),
user: message.user,
skip_tool_details: true,
)
@@ -460,22 +332,6 @@ module DiscourseAi
reply
end
def get_context(participants:, conversation_context:, user:, skip_tool_details: nil)
result = {
site_url: Discourse.base_url,
site_title: SiteSetting.title,
site_description: SiteSetting.site_description,
time: Time.zone.now,
participants: participants,
conversation_context: conversation_context,
user: user,
}
result[:skip_tool_details] = true if skip_tool_details
result
end
def reply_to(
post,
custom_instructions: nil,
@@ -509,16 +365,25 @@ module DiscourseAi
end
)
# safeguard
max_context_posts = 40
if bot.persona.class.respond_to?(:max_context_posts)
max_context_posts = bot.persona.class.max_context_posts || 40
end
context =
get_context(
participants: post.topic.allowed_users.map(&:username).join(", "),
conversation_context: conversation_context(post, style: context_style),
user: post.user,
BotContext.new(
post: post,
custom_instructions: custom_instructions,
messages:
DiscourseAi::Completions::PromptMessagesBuilder.messages_from_post(
post,
style: context_style,
max_posts: max_context_posts,
include_uploads: bot.persona.class.vision_enabled,
bot_usernames: available_bot_usernames,
),
)
context[:post_id] = post.id
context[:topic_id] = post.topic_id
context[:private_message] = post.topic.private_message?
context[:custom_instructions] = custom_instructions
reply_user = bot.bot_user
if bot.persona.class.respond_to?(:user_id)
@@ -562,7 +427,7 @@ module DiscourseAi
Discourse.redis.setex(redis_stream_key, 60, 1)
end
context[:skip_tool_details] ||= !bot.persona.class.tool_details
context.skip_tool_details ||= !bot.persona.class.tool_details
post_streamer = PostStreamer.new(delay: Rails.env.test? ? 0 : 0.5) if stream_reply
lib/ai_bot/tool_runner.rb
+17 -11
View File
@@ -13,7 +13,13 @@ module DiscourseAi
MARSHAL_STACK_DEPTH = 20
MAX_HTTP_REQUESTS = 20
def initialize(parameters:, llm:, bot_user:, context: {}, tool:, timeout: nil)
def initialize(parameters:, llm:, bot_user:, context: nil, tool:, timeout: nil)
if context && !context.is_a?(DiscourseAi::AiBot::BotContext)
raise ArgumentError, "context must be a BotContext object"
end
context ||= DiscourseAi::AiBot::BotContext.new
@parameters = parameters
@llm = llm
@bot_user = bot_user
@@ -99,7 +105,7 @@ module DiscourseAi
},
};
const context = #{JSON.generate(@context)};
const context = #{JSON.generate(@context.to_json)};
function details() { return ""; };
JS
@@ -240,13 +246,13 @@ module DiscourseAi
def llm_user
@llm_user ||=
begin
@context[:llm_user] || post&.user || @bot_user
post&.user || @bot_user
end
end
def post
return @post if defined?(@post)
post_id = @context[:post_id]
post_id = @context.post_id
@post = post_id && Post.find_by(id: post_id)
end
@@ -336,8 +342,8 @@ module DiscourseAi
bot = DiscourseAi::AiBot::Bot.as(@bot_user || persona.user, persona: persona)
playground = DiscourseAi::AiBot::Playground.new(bot)
if @context[:post_id]
post = Post.find_by(id: @context[:post_id])
if @context.post_id
post = Post.find_by(id: @context.post_id)
return { error: "Post not found" } if post.nil?
reply_post =
@@ -354,13 +360,13 @@ module DiscourseAi
else
return { error: "Failed to create reply" }
end
elsif @context[:message_id] && @context[:channel_id]
message = Chat::Message.find_by(id: @context[:message_id])
channel = Chat::Channel.find_by(id: @context[:channel_id])
elsif @context.message_id && @context.channel_id
message = Chat::Message.find_by(id: @context.message_id)
channel = Chat::Channel.find_by(id: @context.channel_id)
return { error: "Message or channel not found" } if message.nil? || channel.nil?
reply =
playground.reply_to_chat_message(message, channel, @context[:context_post_ids])
playground.reply_to_chat_message(message, channel, @context.context_post_ids)
if reply
return { success: true, message_id: reply.id }
@@ -457,7 +463,7 @@ module DiscourseAi
UploadCreator.new(
file,
filename,
for_private_message: @context[:private_message],
for_private_message: @context.private_message,
).create_for(@bot_user.id)
{ id: upload.id, short_url: upload.short_url, url: upload.url }
lib/ai_bot/tools/create_artifact.rb
+1 -1
View File
@@ -108,7 +108,7 @@ module DiscourseAi
end
def invoke
post = Post.find_by(id: context[:post_id])
post = Post.find_by(id: context.post_id)
return error_response("No post context found") unless post
partial_response = +""
lib/ai_bot/tools/dall_e.rb
+1 -1
View File
@@ -111,7 +111,7 @@ module DiscourseAi
UploadCreator.new(
file,
"image.png",
for_private_message: context[:private_message],
for_private_message: context.private_message?,
).create_for(bot_user.id),
}
end
lib/ai_bot/tools/image.rb
+1 -1
View File
@@ -131,7 +131,7 @@ module DiscourseAi
UploadCreator.new(
file,
"image.png",
for_private_message: context[:private_message],
for_private_message: context.private_message,
).create_for(bot_user.id),
seed: image[:seed],
}
lib/ai_bot/tools/read.rb
+1 -1
View File
@@ -48,7 +48,7 @@ module DiscourseAi
def invoke
not_found = { topic_id: topic_id, description: "Topic not found" }
guardian = Guardian.new(context[:user]) if options[:read_private] && context[:user]
guardian = Guardian.new(context.user) if options[:read_private] && context.user
guardian ||= Guardian.new
@title = ""
lib/ai_bot/tools/read_artifact.rb
+1 -1
View File
@@ -59,7 +59,7 @@ module DiscourseAi
end
def post
@post ||= Post.find_by(id: context[:post_id])
@post ||= Post.find_by(id: context.post_id)
end
def handle_discourse_artifact(uri)
lib/ai_bot/tools/search.rb
+1 -1
View File
@@ -130,7 +130,7 @@ module DiscourseAi
after: parameters[:after],
status: parameters[:status],
max_results: max_results,
current_user: options[:search_private] ? context[:user] : nil,
current_user: options[:search_private] ? context.user : nil,
)
@last_num_results = results[:rows]&.length || 0
lib/ai_bot/tools/summarize.rb
-4
View File
@@ -40,10 +40,6 @@ module DiscourseAi
false
end
def standalone?
true
end
def custom_raw
@last_summary || I18n.t("discourse_ai.ai_bot.topic_not_found")
end
lib/ai_bot/tools/tool.rb
+5 -6
View File
@@ -56,14 +56,17 @@ module DiscourseAi
persona_options: {},
bot_user:,
llm:,
context: {}
context: nil
)
@parameters = parameters
@tool_call_id = tool_call_id
@persona_options = persona_options
@bot_user = bot_user
@llm = llm
@context = context
@context = context.nil? ? DiscourseAi::AiBot::BotContext.new(messages: []) : context
if [email protected]_a?(DiscourseAi::AiBot::BotContext)
raise ArgumentError, "context must be a DiscourseAi::AiBot::Context"
end
end
def name
@@ -108,10 +111,6 @@ module DiscourseAi
true
end
def standalone?
false
end
protected
def fetch_default_branch(repo)
lib/ai_bot/tools/update_artifact.rb
+2 -2
View File
@@ -39,7 +39,7 @@ module DiscourseAi
def self.inject_prompt(prompt:, context:, persona:)
return if persona.options["do_not_echo_artifact"].to_s == "true"
# we inject the current artifact content into the last user message
if topic_id = context[:topic_id]
if topic_id = context.topic_id
posts = Post.where(topic_id: topic_id)
artifact = AiArtifact.order("id desc").where(post: posts).first
if artifact
@@ -113,7 +113,7 @@ module DiscourseAi
end
def invoke
post = Post.find_by(id: context[:post_id])
post = Post.find_by(id: context.post_id)
return error_response("No post context found") unless post
artifact = AiArtifact.find_by(id: parameters[:artifact_id])
lib/ai_helper/assistant.rb
+3 -2
View File
@@ -188,9 +188,10 @@ module DiscourseAi
messages: [
{
type: :user,
content:
content: [
"Describe this image in a single sentence#{custom_locale_instructions(user)}",
upload_ids: [upload.id],
{ upload_id: upload.id },
],
},
],
)
lib/ai_moderation/spam_scanner.rb
+4 -1
View File
@@ -187,7 +187,10 @@ module DiscourseAi
prompt = DiscourseAi::Completions::Prompt.new(system_prompt)
args = { type: :user, content: context }
upload_ids = post.upload_ids
args[:upload_ids] = upload_ids.take(3) if upload_ids.present?
if upload_ids.present?
args[:content] = [args[:content]]
upload_ids.take(3).each { |upload_id| args[:content] << { upload_id: upload_id } }
end
prompt.push(**args)
prompt
end
lib/automation/llm_tool_triage.rb
+1 -5
View File
@@ -7,11 +7,7 @@ module DiscourseAi
return if !tool
return if !tool.parameters.blank?
context = {
post_id: post.id,
automation_id: automation&.id,
automation_name: automation&.name,
}
context = DiscourseAi::AiBot::BotContext.new(post: post)
runner = tool.runner({}, llm: nil, bot_user: Discourse.system_user, context: context)
runner.invoke
lib/automation/llm_triage.rb
+6 -1
View File
@@ -42,7 +42,12 @@ module DiscourseAi
content = llm.tokenizer.truncate(content, max_post_tokens) if max_post_tokens.present?
prompt.push(type: :user, content: content, upload_ids: post.upload_ids)
if post.upload_ids.present?
content = [content]
content.concat(post.upload_ids.map { |upload_id| { upload_id: upload_id } })
end
prompt.push(type: :user, content: content)
result = nil
lib/completions/dialects/chat_gpt.rb
+33 -21
View File
@@ -17,13 +17,13 @@ module DiscourseAi
llm_model.provider == "open_ai" || llm_model.provider == "azure"
end
def translate
def embed_user_ids?
return @embed_user_ids if defined?(@embed_user_ids)
@embed_user_ids =
prompt.messages.any? do |m|
m[:id] && m[:type] == :user && !m[:id].to_s.match?(VALID_ID_REGEX)
end
super
end
def max_prompt_tokens
@@ -102,35 +102,47 @@ module DiscourseAi
end
def user_msg(msg)
user_message = { role: "user", content: msg[:content] }
content_array = []
user_message = { role: "user" }
if msg[:id]
if @embed_user_ids
user_message[:content] = "#{msg[:id]}: #{msg[:content]}"
if embed_user_ids?
content_array << "#{msg[:id]}: "
else
user_message[:name] = msg[:id]
end
end
user_message[:content] = inline_images(user_message[:content], msg) if vision_support?
content_array << msg[:content]
content_array =
to_encoded_content_array(
content: content_array.flatten,
image_encoder: ->(details) { image_node(details) },
text_encoder: ->(text) { { type: "text", text: text } },
allow_vision: vision_support?,
)
user_message[:content] = no_array_if_only_text(content_array)
user_message
end
def inline_images(content, message)
encoded_uploads = prompt.encoded_uploads(message)
return content if encoded_uploads.blank?
def no_array_if_only_text(content_array)
if content_array.size == 1 && content_array.first[:type] == "text"
content_array.first[:text]
else
content_array
end
end
content_w_imgs =
encoded_uploads.reduce([]) do |memo, details|
memo << {
type: "image_url",
image_url: {
url: "data:#{details[:mime_type]};base64,#{details[:base64]}",
},
}
end
content_w_imgs << { type: "text", text: message[:content] }
def image_node(details)
{
type: "image_url",
image_url: {
url: "data:#{details[:mime_type]};base64,#{details[:base64]}",
},
}
end
def per_message_overhead
lib/completions/dialects/claude.rb
+43 -29
View File
@@ -87,9 +87,9 @@ module DiscourseAi
end
def model_msg(msg)
if msg[:thinking] || msg[:redacted_thinking_signature]
content_array = []
content_array = []
if msg[:thinking] || msg[:redacted_thinking_signature]
if msg[:thinking]
content_array << {
type: "thinking",
@@ -104,13 +104,19 @@ module DiscourseAi
data: msg[:redacted_thinking_signature],
}
end
content_array << { type: "text", text: msg[:content] }
{ role: "assistant", content: content_array }
else
{ role: "assistant", content: msg[:content] }
end
# other encoder is used to pass through thinking
content_array =
to_encoded_content_array(
content: [content_array, msg[:content]].flatten,
image_encoder: ->(details) {},
text_encoder: ->(text) { { type: "text", text: text } },
other_encoder: ->(details) { details },
allow_vision: false,
)
{ role: "assistant", content: no_array_if_only_text(content_array) }
end
def system_msg(msg)
@@ -124,31 +130,39 @@ module DiscourseAi
end
def user_msg(msg)
content = +""
content << "#{msg[:id]}: " if msg[:id]
content << msg[:content]
content = inline_images(content, msg) if vision_support?
content_array = []
content_array << "#{msg[:id]}: " if msg[:id]
content_array.concat([msg[:content]].flatten)
{ role: "user", content: content }
content_array =
to_encoded_content_array(
content: content_array,
image_encoder: ->(details) { image_node(details) },
text_encoder: ->(text) { { type: "text", text: text } },
allow_vision: vision_support?,
)
{ role: "user", content: no_array_if_only_text(content_array) }
end
def inline_images(content, message)
encoded_uploads = prompt.encoded_uploads(message)
return content if encoded_uploads.blank?
# keeping our payload as backward compatible as possible
def no_array_if_only_text(content_array)
if content_array.length == 1 && content_array.first[:type] == "text"
content_array.first[:text]
else
content_array
end
end
content_w_imgs =
encoded_uploads.reduce([]) do |memo, details|
memo << {
source: {
type: "base64",
data: details[:base64],
media_type: details[:mime_type],
},
type: "image",
}
end
content_w_imgs << { type: "text", text: content }
def image_node(details)
{
source: {
type: "base64",
data: details[:base64],
media_type: details[:mime_type],
},
type: "image",
}
end
end
end
lib/completions/dialects/command.rb
+3 -3
View File
@@ -110,9 +110,9 @@ module DiscourseAi
end
def user_msg(msg)
user_message = { role: "USER", message: msg[:content] }
user_message[:message] = "#{msg[:id]}: #{msg[:content]}" if msg[:id]
content = prompt.text_only(msg)
user_message = { role: "USER", message: content }
user_message[:message] = "#{msg[:id]}: #{content}" if msg[:id]
user_message
end
end
lib/completions/dialects/dialect.rb
+32
View File
@@ -227,6 +227,38 @@ module DiscourseAi
msg = msg.merge(content: new_content)
user_msg(msg)
end
def to_encoded_content_array(
content:,
image_encoder:,
text_encoder:,
other_encoder: nil,
allow_vision:
)
content = [content] if !content.is_a?(Array)
current_string = +""
result = []
content.each do |c|
if c.is_a?(String)
current_string << c
elsif c.is_a?(Hash) && c.key?(:upload_id) && allow_vision
if !current_string.empty?
result << text_encoder.call(current_string)
current_string = +""
end
encoded = prompt.encode_upload(c[:upload_id])
result << image_encoder.call(encoded) if encoded
elsif other_encoder
encoded = other_encoder.call(c)
result << encoded if encoded
end
end
result << text_encoder.call(current_string) if !current_string.empty?
result
end
end
end
end
lib/completions/dialects/gemini.rb
+18 -17
View File
@@ -106,28 +106,29 @@ module DiscourseAi
end
def user_msg(msg)
if beta_api?
# support new format with multiple parts
result = { role: "user", parts: [{ text: msg[:content] }] }
return result unless vision_support?
content_array = []
content_array << "#{msg[:id]}: " if msg[:id]
upload_parts = uploaded_parts(msg)
result[:parts].concat(upload_parts) if upload_parts.present?
result
content_array << msg[:content]
content_array.flatten!
content_array =
to_encoded_content_array(
content: content_array,
image_encoder: ->(details) { image_node(details) },
text_encoder: ->(text) { { text: text } },
allow_vision: vision_support? && beta_api?,
)
if beta_api?
{ role: "user", parts: content_array }
else
{ role: "user", parts: { text: msg[:content] } }
{ role: "user", parts: content_array.first }
end
end
def uploaded_parts(message)
encoded_uploads = prompt.encoded_uploads(message)
result = []
if encoded_uploads.present?
encoded_uploads.each do |details|
result << { inlineData: { mimeType: details[:mime_type], data: details[:base64] } }
end
end
result
def image_node(details)
{ inlineData: { mimeType: details[:mime_type], data: details[:base64] } }
end
def tool_call_msg(msg)
lib/completions/dialects/nova.rb
+1 -1
View File
@@ -155,7 +155,7 @@ module DiscourseAi
end
end
{ role: "user", content: msg[:content], images: images }
{ role: "user", content: prompt.text_only(msg), images: images }
end
def model_msg(msg)
lib/completions/dialects/ollama.rb
+1 -1
View File
@@ -69,7 +69,7 @@ module DiscourseAi
end
def user_msg(msg)
user_message = { role: "user", content: msg[:content] }
user_message = { role: "user", content: prompt.text_only(msg) }
encoded_uploads = prompt.encoded_uploads(msg)
if encoded_uploads.present?
lib/completions/dialects/open_ai_compatible.rb
+1 -53
View File
@@ -3,7 +3,7 @@
module DiscourseAi
module Completions
module Dialects
class OpenAiCompatible < Dialect
class OpenAiCompatible < ChatGpt
class << self
def can_translate?(_llm_model)
# fallback dialect
@@ -43,58 +43,6 @@ module DiscourseAi
translated.unshift(user_msg)
end
private
def system_msg(msg)
msg = { role: "system", content: msg[:content] }
if tools_dialect.instructions.present?
msg[:content] = msg[:content].dup << "\n\n#{tools_dialect.instructions}"
end
msg
end
def model_msg(msg)
{ role: "assistant", content: msg[:content] }
end
def tool_call_msg(msg)
translated = tools_dialect.from_raw_tool_call(msg)
{ role: "assistant", content: translated }
end
def tool_msg(msg)
translated = tools_dialect.from_raw_tool(msg)
{ role: "user", content: translated }
end
def user_msg(msg)
content = +""
content << "#{msg[:id]}: " if msg[:id]
content << msg[:content]
message = { role: "user", content: content }
message[:content] = inline_images(message[:content], msg) if vision_support?
message
end
def inline_images(content, message)
encoded_uploads = prompt.encoded_uploads(message)
return content if encoded_uploads.blank?
encoded_uploads.reduce([{ type: "text", text: message[:content] }]) do |memo, details|
memo << {
type: "image_url",
image_url: {
url: "data:#{details[:mime_type]};base64,#{details[:base64]}",
},
}
end
end
end
end
end
lib/completions/prompt.rb
+47 -15
View File
@@ -89,7 +89,6 @@ module DiscourseAi
content:,
id: nil,
name: nil,
upload_ids: nil,
thinking: nil,
thinking_signature: nil,
redacted_thinking_signature: nil
@@ -98,7 +97,6 @@ module DiscourseAi
new_message = { type: type, content: content }
new_message[:name] = name.to_s if name
new_message[:id] = id.to_s if id
new_message[:upload_ids] = upload_ids if upload_ids
new_message[:thinking] = thinking if thinking
new_message[:thinking_signature] = thinking_signature if thinking_signature
new_message[
@@ -115,11 +113,44 @@ module DiscourseAi
tools.present?
end
# helper method to get base64 encoded uploads
# at the correct dimentions
def encoded_uploads(message)
return [] if message[:upload_ids].blank?
UploadEncoder.encode(upload_ids: message[:upload_ids], max_pixels: max_pixels)
if message[:content].is_a?(Array)
upload_ids =
message[:content]
.map do |content|
content[:upload_id] if content.is_a?(Hash) && content.key?(:upload_id)
end
.compact
if !upload_ids.empty?
return UploadEncoder.encode(upload_ids: upload_ids, max_pixels: max_pixels)
end
end
[]
end
def text_only(message)
if message[:content].is_a?(Array)
message[:content].map { |element| element if element.is_a?(String) }.compact.join
else
message[:content]
end
end
def encode_upload(upload_id)
UploadEncoder.encode(upload_ids: [upload_id], max_pixels: max_pixels).first
end
def content_with_encoded_uploads(content)
return [content] unless content.is_a?(Array)
content.map do |c|
if c.is_a?(Hash) && c.key?(:upload_id)
encode_upload(c[:upload_id])
else
c
end
end
end
def ==(other)
@@ -150,7 +181,6 @@ module DiscourseAi
content
id
name
upload_ids
thinking
thinking_signature
redacted_thinking_signature
@@ -159,15 +189,17 @@ module DiscourseAi
raise ArgumentError, "message contains invalid keys: #{invalid_keys}"
end
if message[:type] == :upload_ids && !message[:upload_ids].is_a?(Array)
raise ArgumentError, "upload_ids must be an array of ids"
if message[:content].is_a?(Array)
message[:content].each do |content|
if !content.is_a?(String) && !(content.is_a?(Hash) && content.keys == [:upload_id])
raise ArgumentError, "Array message content must be a string or {upload_id: ...} "
end
end
else
if !message[:content].is_a?(String)
raise ArgumentError, "Message content must be a string or an array"
end
end
if message[:upload_ids].present? && message[:type] != :user
raise ArgumentError, "upload_ids are only supported for users"
end
raise ArgumentError, "message content must be a string" if !message[:content].is_a?(String)
end
def validate_turn(last_turn, new_turn)

Some files were not shown because too many files have changed in this diff Show More