Compare commits

...
Author SHA1 Message Date
rngus2344 3dfa16ad0f spec updated 2025-03-05 16:39:54 -06:00
rngus2344 e94385927b spec added 2025-03-05 16:21:51 -06:00
rngus2344 20701cb48a abstract logic into separate method 2025-03-05 14:18:17 -06:00
rngus2344 8ce7331870 settings update 2025-03-05 13:55:10 -06:00
rngus2344 1aa727743d topic summary with ai persona siteSettings 2025-03-05 13:00:06 -06:00
rngus2344 64ad45e28d DEV: Add Ai summary persona siteSettings 2025-03-05 11:27:39 -06:00
Discourse Translator Bot 2c0a8e7f5c Update translations (#1131) 2025-02-18 14:51:56 +01:00
Martin Brennan 0582e75205 DEV: Skip PDF tests (#1129)
CI=1 is not set on our internal build system,
we need to fix that but for now let's unblock the build
2025-02-18 10:17:11 +10:00
SamandAlan Guo Xiang Tan ce79a18790 FEATURE: Native PDF support (#1127)
* FEATURE: Native PDF support

This amends it so we use PDF Reader gem to extract text from PDFs

* This means that our simple pdf eval passes at last

* fix spec

* skip test in CI

* test file support

* Update lib/utils/image_to_text.rb

Co-authored-by: Alan Guo Xiang Tan <[email protected]>

* address pr comments

---------

Co-authored-by: Alan Guo Xiang Tan <[email protected]>
2025-02-18 09:22:57 +11:00
Sam 9a6aec2cf6 DEV: eval support for tool calls (#1128)
Also fixes anthropic with no params, streaming calls
2025-02-18 07:58:54 +11:00
Sam 5e80f93e4c FEATURE: PDF support for rag pipeline (#1118)
This PR introduces several enhancements and refactorings to the AI Persona and RAG (Retrieval-Augmented Generation) functionalities within the discourse-ai plugin. Here's a breakdown of the changes:

**1. LLM Model Association for RAG and Personas:**

-   **New Database Columns:** Adds `rag_llm_model_id` to both `ai_personas` and `ai_tools` tables. This allows specifying a dedicated LLM for RAG indexing, separate from the persona's primary LLM.  Adds `default_llm_id` and `question_consolidator_llm_id` to `ai_personas`.
-   **Migration:**  Includes a migration (`20250210032345_migrate_persona_to_llm_model_id.rb`) to populate the new `default_llm_id` and `question_consolidator_llm_id` columns in `ai_personas` based on the existing `default_llm` and `question_consolidator_llm` string columns, and a post migration to remove the latter.
-   **Model Changes:**  The `AiPersona` and `AiTool` models now `belong_to` an `LlmModel` via `rag_llm_model_id`. The `LlmModel.proxy` method now accepts an `LlmModel` instance instead of just an identifier.  `AiPersona` now has `default_llm_id` and `question_consolidator_llm_id` attributes.
-   **UI Updates:**  The AI Persona and AI Tool editors in the admin panel now allow selecting an LLM for RAG indexing (if PDF/image support is enabled).  The RAG options component displays an LLM selector.
-   **Serialization:** The serializers (`AiCustomToolSerializer`, `AiCustomToolListSerializer`, `LocalizedAiPersonaSerializer`) have been updated to include the new `rag_llm_model_id`, `default_llm_id` and `question_consolidator_llm_id` attributes.

**2. PDF and Image Support for RAG:**

-   **Site Setting:** Introduces a new hidden site setting, `ai_rag_pdf_images_enabled`, to control whether PDF and image files can be indexed for RAG. This defaults to `false`.
-   **File Upload Validation:** The `RagDocumentFragmentsController` now checks the `ai_rag_pdf_images_enabled` setting and allows PDF, PNG, JPG, and JPEG files if enabled.  Error handling is included for cases where PDF/image indexing is attempted with the setting disabled.
-   **PDF Processing:** Adds a new utility class, `DiscourseAi::Utils::PdfToImages`, which uses ImageMagick (`magick`) to convert PDF pages into individual PNG images. A maximum PDF size and conversion timeout are enforced.
-   **Image Processing:** A new utility class, `DiscourseAi::Utils::ImageToText`, is included to handle OCR for the images and PDFs.
-   **RAG Digestion Job:** The `DigestRagUpload` job now handles PDF and image uploads. It uses `PdfToImages` and `ImageToText` to extract text and create document fragments.
-   **UI Updates:**  The RAG uploader component now accepts PDF and image file types if `ai_rag_pdf_images_enabled` is true. The UI text is adjusted to indicate supported file types.

**3. Refactoring and Improvements:**

-   **LLM Enumeration:** The `DiscourseAi::Configuration::LlmEnumerator` now provides a `values_for_serialization` method, which returns a simplified array of LLM data (id, name, vision_enabled) suitable for use in serializers. This avoids exposing unnecessary details to the frontend.
-   **AI Helper:** The `AiHelper::Assistant` now takes optional `helper_llm` and `image_caption_llm` parameters in its constructor, allowing for greater flexibility.
-   **Bot and Persona Updates:** Several updates were made across the codebase, changing the string based association to a LLM to the new model based.
-   **Audit Logs:** The `DiscourseAi::Completions::Endpoints::Base` now formats raw request payloads as pretty JSON for easier auditing.
- **Eval Script:** An evaluation script is included.

**4. Testing:**

-    The PR introduces a new eval system for LLMs, this allows us to test how functionality works across various LLM providers. This lives in `/evals`
2025-02-14 12:15:07 +11:00
Joffrey JAFFEUX e2afbc26d3 FIX: correctly handle provider edit (#1125)
Prior to this commit, editing the provider wouldn't recompute the provider params. It would also not correctly recompute the "canEditURL" property.

To make possible this commit has:
- made a fix in core: https://github.com/discourse/discourse/pull/31329
- ensures the provider params are recomputed when provider is changed
- made the check on `canEditURL` based on form state and not initial model value

Tests have been added to confirm the expected behavior.
2025-02-13 12:03:13 +01:00
dependabot[bot] 35f15629fb Build(deps-dev): Bump rack from 3.1.6 to 3.1.10 (#1124)
Bumps [rack](https://github.com/rack/rack) from 3.1.6 to 3.1.10.
- [Release notes](https://github.com/rack/rack/releases)
- [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rack/rack/compare/v3.1.6...v3.1.10)

---
updated-dependencies:
- dependency-name: rack
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-13 12:09:04 +11:00
David Battersby 1bfad17b9a FIX: update draft key for new PM with AI bot (#1123)
* FIX: update draft key for new PM with AI bot

* allow multiple drafts with ai bot

* fix linting
2025-02-13 12:08:31 +11:00
Rafael dos Santos Silva 77c6543c5d FIX: Embeddings backfill job compat when transitioning models (#1122)
When you already have embeddings for a model stored and change models,
our backfill script was failing to backfill the newly configured model.

Regression introduced most likely in 1686a8a
2025-02-12 10:37:45 -03:00
Rafael dos Santos Silva 708a3bd2b8 UX: Better tooltips for embeddings task instructions prefixes (#1121) 2025-02-11 15:01:29 -03:00
Discourse Translator Bot 8c653fdb25 Update translations (#1120) 2025-02-11 16:43:19 +01:00
Martin BrennanandTed Johansson 7b1bdbde6d FIX: Check post action creator result when flagging spam (#1119)
Currently in core re-flagging something that is already flagged as spam
is not supported, long term we may want to support this but in the meantime
we should not be silencing/hiding if the PostActionCreator fails
when flagging things as spam.

---------

Co-authored-by: Ted Johansson <[email protected]>
2025-02-11 13:29:27 +10:00
Hoa Nguyen b60926c6e6 FEATURE: Tool name validation (#842)
* FEATURE: Tool name validation

- Add unique index to the name column of the ai_tools table
- correct our tests for AiToolController
- tool_name field which will be used to represent to LLM
- Add tool_name to Tools's presets
- Add duplicate tools validation for AiPersona
- Add unique constraint to the name column of the ai_tools table

* DEV: Validate duplicate tool_name between builin tools and custom tools

* lint

* chore: fix linting

* fix conlict mistakes

* chore: correct icon class

* chore: fix failed specs

* Add max_length to tool_name

* chore: correct the option name

* lintings

* fix lintings
2025-02-07 14:34:47 +11:00
David Taylor 551f674c43 DEV: Bump dependencies and fix linting (#1115) 2025-02-06 17:42:32 +01:00
Roman Rizzi 90bcb8b503 DEV: Build sentiment clients outside of promises (#1117) 2025-02-06 13:11:10 -03:00
Roman Rizzi e52045ebdc DEV: Robust check for embeddings enabled (#1116) 2025-02-06 12:18:55 -03:00
Kris 83c7919856 UX: clarify embeddings description (#1113) 2025-02-06 08:50:01 -05:00
David Taylor a996aa45bc DEV: Pin version for Discourse <3.5.0.beta1-dev (#1114) 2025-02-05 19:57:52 +01:00
Discourse Translator Bot bdef136080 Update translations (#1112) 2025-02-04 15:18:08 +01:00
Roman Rizzi 1b1b44353b FEATURE: Changes to summaries' outdated logic. (#1108)
Before this change, a summary was only outdated when new content appeared, for topics with "best replies", when the query returned different results. The intent behind this change is to detect when a summary is outdated as a result of an edit.

Additionally, we are changing the backfill candidates query to compare "ai_summary_backfill_topic_max_age_days" against "last_posted_at" instead of "created_at", to catch long-lived, active topics. This was discussed here: https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/14?u=roman_rizzi
2025-02-04 09:31:11 -03:00
Joffrey JAFFEUX d3b93f984d UX: include none false for provider params (#1111)
This change has been forgotten in https://github.com/discourse/discourse-ai/commit/40e996b174332030762b29c4fb80557aa9c9151c
2025-02-04 12:38:22 +01:00
Joffrey JAFFEUX 40e996b174 DEV: converts llm admin page to use form kit (#1099)
This also converts the quota editor, and the quota modal.
2025-02-04 11:51:01 +01:00
Sam 43c56d7c92 FIX: need to be able to search replace within lines (#1110)
(this is needed for very simple diffs and HTML)
2025-02-04 18:16:52 +11:00
Sam a7d032fa28 DEV: artifact system update (#1096)
### Why

This pull request fundamentally restructures how AI bots create and update web artifacts to address critical limitations in the previous approach:

1.  **Improved Artifact Context for LLMs**: Previously, artifact creation and update tools included the *entire* artifact source code directly in the tool arguments. This overloaded the Language Model (LLM) with raw code, making it difficult for the LLM to maintain a clear understanding of the artifact's current state when applying changes. The LLM would struggle to differentiate between the base artifact and the requested modifications, leading to confusion and less effective updates.
2.  **Reduced Token Usage and History Bloat**: Including the full artifact source code in every tool interaction was extremely token-inefficient.  As conversations progressed, this redundant code in the history consumed a significant number of tokens unnecessarily. This not only increased costs but also diluted the context for the LLM with less relevant historical information.
3.  **Enabling Updates for Large Artifacts**: The lack of a practical diff or targeted update mechanism made it nearly impossible to efficiently update larger web artifacts.  Sending the entire source code for every minor change was both computationally expensive and prone to errors, effectively blocking the use of AI bots for meaningful modifications of complex artifacts.

**This pull request addresses these core issues by**:

*   Introducing methods for the AI bot to explicitly *read* and understand the current state of an artifact.
*   Implementing efficient update strategies that send *targeted* changes rather than the entire artifact source code.
*   Providing options to control the level of artifact context included in LLM prompts, optimizing token usage.

### What

The main changes implemented in this PR to resolve the above issues are:

1.  **`Read Artifact` Tool for Contextual Awareness**:
    - A new `read_artifact` tool is introduced, enabling AI bots to fetch and process the current content of a web artifact from a given URL (local or external).
    - This provides the LLM with a clear and up-to-date representation of the artifact's HTML, CSS, and JavaScript, improving its understanding of the base to be modified.
    - By cloning local artifacts, it allows the bot to work with a fresh copy, further enhancing context and control.

2.  **Refactored `Update Artifact` Tool with Efficient Strategies**:
    - The `update_artifact` tool is redesigned to employ more efficient update strategies, minimizing token usage and improving update precision:
        - **`diff` strategy**:  Utilizes a search-and-replace diff algorithm to apply only the necessary, targeted changes to the artifact's code. This significantly reduces the amount of code sent to the LLM and focuses its attention on the specific modifications.
        - **`full` strategy**:  Provides the option to replace the entire content sections (HTML, CSS, JavaScript) when a complete rewrite is required.
    - Tool options enhance the control over the update process:
        - `editor_llm`:  Allows selection of a specific LLM for artifact updates, potentially optimizing for code editing tasks.
        - `update_algorithm`: Enables choosing between `diff` and `full` update strategies based on the nature of the required changes.
        - `do_not_echo_artifact`:  Defaults to true, and by *not* echoing the artifact in prompts, it further reduces token consumption in scenarios where the LLM might not need the full artifact context for every update step (though effectiveness might be slightly reduced in certain update scenarios).

3.  **System and General Persona Tool Option Visibility and Customization**:
    - Tool options, including those for system personas, are made visible and editable in the admin UI. This allows administrators to fine-tune the behavior of all personas and their tools, including setting specific LLMs or update algorithms. This was previously limited or hidden for system personas.

4.  **Centralized and Improved Content Security Policy (CSP) Management**:
    - The CSP for AI artifacts is consolidated and made more maintainable through the `ALLOWED_CDN_SOURCES` constant. This improves code organization and future updates to the allowed CDN list, while maintaining the existing security posture.

5.  **Codebase Improvements**:
    - Refactoring of diff utilities, introduction of strategy classes, enhanced error handling, new locales, and comprehensive testing all contribute to a more robust, efficient, and maintainable artifact management system.

By addressing the issues of LLM context confusion, token inefficiency, and the limitations of updating large artifacts, this pull request significantly improves the practicality and effectiveness of AI bots in managing web artifacts within Discourse.
2025-02-04 16:27:27 +11:00
Roman Rizzi 7ad922331b FIX: Make sure DiscoursePrometheus is installed when collecting metrics (#1107) 2025-02-03 11:04:25 -03:00
Sam cf86d274a0 FEATURE: improve o3-mini support (#1106)
* DEV: raise timeout for reasoning LLMs

* FIX: use id to identify llms, not model_name

model_name is not unique, in the case of reasoning models
you may configure the same llm multiple times using different
reasoning levels.
2025-02-03 08:45:56 +11:00
Sam 381a2715c8 FEATURE: o3-mini supports (#1105)
1. Adds o3-mini presets
2. Adds support for reasoning effort
3. properly use "developer" messages for reasoning models
2025-02-01 14:08:34 +11:00
Rafael dos Santos Silva 8c22540e61 FEATURE: Use persona default LLM for Discord integration (#1104) 2025-01-31 18:11:20 -03:00
Keegan George 13f9a1908a DEV: Only permit config of allowed seeded models (#1103)
This update resolves a regression that was introduced in https://github.com/discourse/discourse-ai/pull/1036/files. Previously, only seeded models that were allowed could be configured for model settings. However, in our attempts to prevent unreachable LLM errors from not allowing settings to persist, it also unknowingly allowed seeded models that were not allowed to be configured. This update resolves this issue, while maintaining the ability to still set unreachable LLMs.
2025-01-31 13:01:15 -08:00
Discourse Translator Bot 980f17a338 Update translations (#1097) 2025-01-31 10:48:55 +01:00
Penar Musaraj d1149ac1b8 FIX: Android tablet composer menu z-index (#1102)
An interim fix for ensuring the AI helper menus are properly shown.
2025-01-30 10:12:37 -08:00
Roman Rizzi 1572068735 DEV: Improve embedding configs validations (#1101)
Before this change, we let you set the embeddings selected model back to " " even with embeddings enabled. This will leave the site in a broken state.

Additionally, it adds a fail-safe for these scenarios to avoid errors on the topics page.
2025-01-30 14:16:56 -03:00
Rafael dos Santos Silva 8f0756fbca FEATURE: Block seeded models for being a persona default (#1100) 2025-01-29 17:13:19 -03:00
Martin Brennan c49b455fd9 DEV: Use new core plugin API to register report mode (#1094)
Introduced in https://github.com/discourse/discourse/pull/31011
2025-01-29 10:33:55 +10:00
Roman Rizzi ffb3011c06 FIX: OpenAI api key still needed for Dall-E. Unhide it. (#1098) 2025-01-29 09:38:22 +11:00
Kris d5bf9521f0 DEV: add missing class to summarize button (#1093) 2025-01-28 08:49:29 -05:00
Ted Johansson 45eabedf47 DEV: Ween off deprecated plugin nav mode option (#1095)
We have removed this flag in core. All plugins now use the "top mode" for their navigation. A backwards-compatible change has been made in core while we remove the usage from plugins.
2025-01-28 12:04:32 +08:00
Roman Rizzi a53719ab8e FIX: Open AI embeddings config migration & Seeded indexes cleanup & (#1092)
This change fixes two different problems.

First, we add a data migration to migrate the configuration of sites using Open AI's embedding model. There was a window between the embedding config changes and #1087, where sites could end up in a broken state due to an unconfigured selected model setting, as reported on https://meta.discourse.org/t/-/348964

The second fix drops pre-seeded search indexes of the models we didn't migrate and corrects the ones where the dimensions don't match. Since the index uses the model ID, new embedding configs could use one of these ones even when the dimensions no longer match.
2025-01-27 15:24:43 -03:00
Natalie Tay fe44d78156 DEV: Expose AI spam scanning metrics (#1077)
This should give us a better idea on how our scanner is faring across sites.

```
# HELP discourse_discourse_ai_spam_detection AI spam scanning statistics
# TYPE discourse_discourse_ai_spam_detection counter
discourse_discourse_ai_spam_detection{db="default",type="scanned"} 16
discourse_discourse_ai_spam_detection{db="default",type="is_spam"} 7
discourse_discourse_ai_spam_detection{db="default",type="false_positive"} 1
discourse_discourse_ai_spam_detection{db="default",type="false_negative"} 2
```
2025-01-27 11:57:01 +08:00
Roman Rizzi ad7bb9bd31 DEV: Promote historical post-deploy migrations (#1091) 2025-01-24 11:49:15 -03:00
Kris 99e73f09ff UX: improve embeddings config styles (#1085)
* WIP: improve embeddings config styles

* switch to textarea, fix back button

* remove log, update button, fix tests

* stree

* fix spec

* spec fix

* remove comment
2025-01-24 16:24:59 +11:00
Martin Brennan 952e0a51d6 UX: Update usage "Learn more..." link (#1090)
There is a new Meta topic for this:

https://meta.discourse.org/t/discourse-ai-ai-usage/348677
2025-01-24 14:18:18 +10:00
Kris 956efba8cb UX: set usage as first AI admin tab (#1089) 2025-01-24 10:14:52 +11:00
Rafael dos Santos Silva 67a1257b89 FEATURE: Gemini Tokenizer (#1088) 2025-01-23 18:20:35 -03:00
Roman Rizzi 5a97752117 FIX: Always raise the single exception/Open AI models migration (#1087) 2025-01-23 15:30:06 -03:00
Penar Musaraj d5cf53e8e0 UX: Fix composer helper z-index (#1086)
Followup to https://github.com/discourse/discourse-ai/pull/1064

That commits adds a higher z-index due to core changes, we no longer need
an iPad-specific z-index.
2025-01-23 10:07:27 -05:00
Sam 8bf350206e FEATURE: track duration of AI calls (#1082)
* FEATURE: track duration of AI calls

* annotate
2025-01-23 11:32:12 +11:00
Roman Rizzi faa8e6e873 FIX: Embeddings backfill rake task was using old code (#1084) 2025-01-22 14:00:26 -03:00
Roman Rizzi e2e753d73c FEATURE: Formalize support for matryoshka dimensions. (#1083)
We have a flag to signal we are shortening the embeddings of a model.
Only used in Open AI's text-embedding-3-*, but we plan to use it for other services.
2025-01-22 11:26:46 -03:00
我秦始皇 654f90f1cd FIX: convert provider_params hash to json before db insert (#1081)
* FIX: convert provider_params hash to json before db insert

* FIX: lint issues in config migration

* FIX: simplify provider_params json conversion
2025-01-22 09:55:41 -03:00
Roman Rizzi a5e5ae72a8 FIX: Open AI embedding shortening is only available for some models (#1080) 2025-01-21 17:50:40 -03:00
Roman Rizzi 3b66fb3e87 FIX: Restore the accidentally deleted query prefix. (#1079)
Additionally, we add a prefix for embedding generation.
Both are stored in the definitions table.
2025-01-21 14:10:31 -03:00
Roman Rizzi f5cf1019fb FEATURE: configurable embeddings (#1049)
* Use AR model for embeddings features

* endpoints

* Embeddings CRUD UI

* Add presets. Hide a couple more settings

* system specs

* Seed embedding definition from old settings

* Generate search bit index on the fly. cleanup orphaned data

* support for seeded models

* Fix run test for new embedding

* fix selected model not set correctly
2025-01-21 12:23:19 -03:00
Discourse Translator Bot fad4b65d4f Update translations (#1078) 2025-01-21 15:55:40 +01:00
Sam 2c609e165b FEATURE: Add user location info to spam scanner context (#1076)
This adds registration and last known IP information and email to scanning context.

This provides another hint for spam scanner about possible malicious users.

For example registered in India, replying from Australia or
email is clearly a throwaway email address.
2025-01-21 17:51:21 +11:00
Kelv 7957796e56 DEV: update all suffix alt icon names (#1075) 2025-01-20 17:33:25 +08:00
David Taylor 890b85bff3 DEV: Update icons for FA6 (#1074) 2025-01-17 10:12:57 +00:00
Roman Rizzi 4784e7fe43 FIX: Set default for existing records. (#1073)
We'll later copy the correct value from content_range. 1 should be the min highest post number a topic has.
2025-01-16 10:38:53 -03:00
Roman Rizzi 46fcdb6ba5 FIX: Make summaries backfill job more resilient. (#1071)
To quickly select backfill candidates without comparing SHAs, we compare the last summarized post to the topic's highest_post_number. However, hiding or deleting a post and adding a small action will update this column, causing the job to stall and re-generate the same summary repeatedly until someone posts a regular reply. On top of this, this is not always true for topics with `best_replies`, as this last reply isn't necessarily included.

Since this is not evident at first glance and each summarization strategy picks its targets differently, I'm opting to simplify the backfill logic and how we track potential candidates.

The first step is dropping `content_range`, which serves no purpose and it's there because summary caching was supposed to work differently at the beginning. So instead, I'm replacing it with a column called `highest_target_number`, which tracks `highest_post_number` for topics and could track other things like channel's `message_count` in the future.

Now that we have this column when selecting every potential backfill candidate, we'll check if the summary is truly outdated by comparing the SHAs, and if it's not, we just update the column and move on
2025-01-16 09:42:53 -03:00
Rafael dos Santos Silva f9aa2de413 FIX: AWS Bedrock non-streaming calls response log (#1072) 2025-01-15 18:51:25 -03:00
Sam 81b952d56e FIX: only hide posts detected explicitly as spam (#1070)
When enabling spam scanner it there may be old unscanned posts
this can create a risky situation where spam scanner operates
on legit posts during false positives

To keep this a lot safer we no longer try to hide old stuff by
the spammers.
2025-01-15 16:50:41 +11:00
Natalie Tay c881f8b361 DEV: Add rake task to send topics or posts to spam scanner (#1059) 2025-01-15 11:48:57 +08:00
Rafael dos Santos Silva 92f122c54d SECURITY: Fix XSS on Shared AI Conversations local Onebox (#1069) 2025-01-14 18:05:37 -03:00
Roman Rizzi cd03874b4d FIX: Missing table check in post_migration (#1068) 2025-01-14 17:33:01 -03:00
Roman Rizzi 65456c8b30 DEV: Migration to remove old embeddings tables~ (#1067)
* DEV: Migration to remove old embeddings tables~

* Check for table existence
2025-01-14 17:13:34 -03:00
Roman Rizzi c4d2b7de1d PERF: Optimize backfill query to prevent statement timeouts (#1066) 2025-01-14 15:39:19 -03:00
Roman Rizzi 6721c6751d FIX: Do batches for backfilling huge embeddings tables (#1065) 2025-01-14 14:42:40 -03:00
Keegan George bbae790c2b FIX: Composer helper not appearing on tablets (#1064)
This update fixes an issue when the composer helper menu was not being shown on tablets in desktop mode. Updating the `z-index` to use the modal-dialog case is more appropriate here.
2025-01-14 09:35:31 -08:00
Roman Rizzi 356ea77201 FIX: Split backfill into separate migrations to use independent transactions (#1063) 2025-01-14 13:30:52 -03:00
Roman Rizzi 09ca123757 FIX: Split statements to avoid timeout (#1062) 2025-01-14 12:54:18 -03:00
Discourse Translator Bot c029bc8979 Update translations (#1060) 2025-01-14 16:20:00 +01:00
Roman Rizzi 65bbcd71fc DEV: Embedding tables' model_id has to be a bigint (#1058)
* DEV: Embedding tables' model_id has to be a bigint

* Drop old search_bit indexes

* copy rag fragment embeddings created during deploy window
2025-01-14 10:53:06 -03:00
SamandKeegan George d07cf51653 FEATURE: llm quotas (#1047)
Adds a comprehensive quota management system for LLM models that allows:

- Setting per-group (applied per user in the group) token and usage limits with configurable durations
- Tracking and enforcing token/usage limits across user groups
- Quota reset periods (hourly, daily, weekly, or custom)
-  Admin UI for managing quotas with real-time updates

This system provides granular control over LLM API usage by allowing admins
to define limits on both total tokens and number of requests per group.
Supports multiple concurrent quotas per model and automatically handles
quota resets.


Co-authored-by: Keegan George <[email protected]>
2025-01-14 15:54:09 +11:00
Sam 20612fde52 FEATURE: add the ability to disable streaming on an Open AI LLM
Disabling streaming is required for models such o1 that do not have streaming
enabled yet

It is good to carry this feature around in case various apis decide not to support streaming endpoints and Discourse AI can continue to work just as it did before. 

Also: fixes issue where sharing artifacts would miss viewport leading to tiny artifacts on mobile
2025-01-13 17:01:01 +11:00
Jarek Radosz 7e9c0dc076 FIX: Invalid locale yaml (#1057) 2025-01-11 23:31:33 +01:00
Mark VanLandingham 2d1ce01320 DEV: Add no-results className to full page search toggle (#1055) 2025-01-10 10:59:18 -06:00
Keegan George b24669c810 DEV: Add structure for errors in spam (#1054)
This update adds some structure for handling errors in the spam config while also handling a specific error related to the spam scanning user not being an admin account.
2025-01-09 09:17:06 -08:00
Keegan George 24b69bf840 FIX: Update spam controller action should consider seeded LLM properly (#1053)
The seeded LLM setting: `SiteSetting.ai_spam_detection_model_allowed_seeded_models` returns a _string_ with IDs separated by pipes. running `_map` on it will return an array with strings. We were previously checking for the id with custom prefix identifier, but instead we should be checking the stringified ID.
2025-01-08 13:41:25 -08:00
Guhyoun Nam 404092a68c DEV: Add appEvents trigger when Ai search results toggled (#1052)
This PR adds appEvents triggers when Ai search results are toggled.
2025-01-08 12:17:25 -06:00
Mark VanLandingham 327adbde29 UX: Full page search -- always show tooltip & add msg (#1051) 2025-01-08 09:05:30 -06:00
Kris 749af40fad UX: close summary modal on click outside (#1050) 2025-01-07 11:24:27 -05:00
Discourse Translator Bot 61758ff8a6 Update translations (#1040) 2025-01-03 14:01:41 +01:00
Mark VanLandingham b6cefd10fa DEV: Move semantic search from connector to component (#1048) 2025-01-02 12:32:49 -06:00
Sam 11d0f60f1e FEATURE: smart date support for AI helper (#1044)
* FEATURE: smart date support for AI helper

This feature allows conversion of human typed in dates and times
to smart "Discourse" timezone friendly dates.

* fix specs and lint

* lint

* address feedback

* add specs
2024-12-31 08:04:25 +11:00
Sam f9f89adac5 FIX: keep track of silence reason when spam detection flags user (#1046)
Previously reason was blank for silencing user
2024-12-27 17:47:16 +11:00
Keegan George b480f13a0f FIX: Prevent LLM enumerator from erroring when spam enabled (#1045)
This PR fixes an issue where LLM enumerator would error out when `SiteSetting.ai_spam_detection = true` but there was no `AiModerationSetting.spam` present.

Typically, we add an `LlmDependencyValidator` for the setting itself, however, since Spam is unique in that it has it's model set in `AiModerationSetting` instead of a `SiteSetting`, we'll add a simple check here to prevent erroring out.
2024-12-27 09:12:29 +11:00
Sam 47ecf86aa1 FIX: embedding validation (#1043) 2024-12-24 09:37:23 +11:00
Rafael dos Santos Silva 792df58fbc FIX: AI Helper category / tag suggestion when user does not categories muted (#1042) 2024-12-23 15:58:26 -03:00
Roman Rizzi ceac6e5efb FIX: Embeddings validator test needs to use the new Vector class. (#1041) 2024-12-23 14:19:22 -03:00
Keegan George bdb8f1d5e0 FIX: Custom prefix causing allowed seeded LLMs not to be shown (#1039)
* FIX: Custom prefix causing allowed seeded LLMs not to be shown

* DEV: update spec

* not `_map` so should be string not array
2024-12-23 16:42:26 +11:00
Kris d15876025f UX: disabled preseeded edit button, add description (#1038) 2024-12-20 19:33:45 -05:00
Rafael dos Santos Silva 7607477ff9 FIX: Cloudflare Workers AI embeddings (#1037)
Regressed on 534b0df
2024-12-20 17:45:27 -03:00
Keegan George 059b3fabb8 DEV: Unreachable LLM error shouldn't prevent setting (#1036)
Previously we had the behaviour for model settings so that when you try and set a model, it runs a test and returns an error if it can't run the test successfully. The error then prevents you from setting the site setting.

This results in some issues when we try and automate things. This PR updates that so that the test runs and discreetly logs the changes, but doesn't prevent the setting from being set. Instead we rely on "run test" in the LLM config along with ProblemChecks to catch issues.
2024-12-20 11:52:11 -08:00
Sam 6a7a45fd4f FIX: properly spin down unused streamer threads (#1035)
Previous version was prone to the bug:

https://github.com/ruby-concurrency/concurrent-ruby/issues/1075

This is particularly bad cause we could have a DB connection
attached to the thread and we never clear it up, so after N hours
this could start exhibiting weird connection issues.
2024-12-20 12:09:42 +11:00
Kris ac705b694b UX: minor improvements to LLM page and admin tables (#1034) 2024-12-19 18:14:22 -05:00
Discourse Translator Bot a4033e2af9 Update translations (#1032) 2024-12-18 15:19:47 +01:00
Martin Brennan f35db8068b DEV: Change to use DPageSubheader (#1033)
Previously was AdminPageSubheader until https://github.com/discourse/discourse/pull/30146
2024-12-18 17:39:31 +10:00
Sam fae2d5ff2c FEATURE: link correctly to filters to assist in debugging spam (#1031)
- Add spam_score_type to AiSpamSerializer for better integration with reviewables.
- Introduce a custom filter for detecting AI spam false negatives in moderation workflows.
- Refactor spam report generation to improve identification of false negatives.
- Add tests to verify the custom filter and its behavior.
- Introduce links for all spam counts in report
2024-12-17 11:02:18 +11:00
Keegan George 90ce942108 FEATURE: Add periodic problem checks for each LLM in use (#1020)
This feature adds a periodic problem check which periodically checks for issues with LLMs that are in use. Periodically, we will run a test to see if the in use LLMs are still operational. If it is not, the LLM with the problem is surfaced to the admin so they can easily go and update the configuration.
2024-12-16 15:00:05 -08:00
Mark VanLandingham 24b107881a FEATURE: Unavailable state for semantic search when sort is not Relevant (#1030)
This commit adds an "unavailable" state for the AI semantic search toggle. Currently the AI toggle disappears when the sort by is anything but Relevance which makes the UI confusing for users looking for AI results. This should help!
2024-12-16 14:30:11 -06:00
Roman Rizzi 534b0df391 REFACTOR: Separation of concerns for embedding generation. (#1027)
In a previous refactor, we moved the responsibility of querying and storing embeddings into the `Schema` class. Now, it's time for embedding generation.

The motivation behind these changes is to isolate vector characteristics in simple objects to later replace them with a DB-backed version, similar to what we did with LLM configs.
2024-12-16 09:55:39 -03:00
Martin Brennan 222e2cf4f9 UX: Use new DStatTiles reusable component from core (#1025)
For the Spam and Usage tabs in admin
2024-12-16 16:48:46 +10:00
Roman RizziandKeegan George 94b85ece80 FIX: Make sure gists are atleast five minutes old before updating them (#1029)
* FIX: Make sure gists are atleast five minutes old before updating them

* Update app/jobs/regular/fast_track_topic_gist.rb

Co-authored-by: Keegan George <[email protected]>

---------

Co-authored-by: Keegan George <[email protected]>
2024-12-13 19:36:34 -03:00
Roman Rizzi 1c40a698ca FIX: get strategy version through vector_rep (#1028) 2024-12-13 18:49:18 -03:00
David Taylor fae1fbc796 DEV: Correct i18n lint violation (#1026) 2024-12-13 16:01:04 +00:00
Roman Rizzi eae527f99d REFACTOR: A Simpler way of interacting with embeddings tables. (#1023)
* REFACTOR: A Simpler way of interacting with embeddings' tables.

This change adds a new abstraction called `Schema`, which acts as a repository that supports the same DB features `VectorRepresentation::Base` has, with the exception that removes the need to have duplicated methods per embeddings table.

It is also a bit more flexible when performing a similarity search because you can pass it a block that gives you access to the builder, allowing you to add multiple joins/where conditions.
2024-12-13 10:15:21 -03:00
Roman Rizzi 97ec2c5ff4 FEATURE: Show gists everywhere except suggested/related (#995) 2024-12-12 12:29:35 -03:00
Krzysztof Kotlarek 04c4ff8cf0 UX: No admin header for edit personas tools or llms (#1021)
In this PR, we added functionality to hide the admin header for edit/new actions - https://github.com/discourse/discourse/pull/30175

To make it work properly, we have to rename `show` to `edit` which is also a more accurate name.
2024-12-12 10:48:58 +11:00
Sam 47c1ea337e FIX: allow scanning of trashed posts and deleted users for test (#1024)
When a post is trashed we should still be allowed to scan it
post.topic will be nil for a trashed topic even if post is trashed
2024-12-12 10:26:05 +11:00
47f5da7e42 FEATURE: Add AI-powered spam detection for new user posts (#1004)
This introduces a comprehensive spam detection system that uses LLM models
to automatically identify and flag potential spam posts. The system is
designed to be both powerful and configurable while preventing false positives.

Key Features:
* Automatically scans first 3 posts from new users (TL0/TL1)
* Creates dedicated AI flagging user to distinguish from system flags
* Tracks false positives/negatives for quality monitoring
* Supports custom instructions to fine-tune detection
* Includes test interface for trying detection on any post

Technical Implementation:
* New database tables:
  - ai_spam_logs: Stores scan history and results
  - ai_moderation_settings: Stores LLM config and custom instructions
* Rate limiting and safeguards:
  - Minimum 10-minute delay between rescans
  - Only scans significant edits (>10 char difference)
  - Maximum 3 scans per post
  - 24-hour maximum age for scannable posts
* Admin UI features:
  - Real-time testing capabilities
  - 7-day statistics dashboard
  - Configurable LLM model selection
  - Custom instruction support

Security and Performance:
* Respects trust levels - only scans TL0/TL1 users
* Skips private messages entirely
* Stops scanning users after 3 successful public posts
* Includes comprehensive test coverage
* Maintains audit log of all scan attempts


---------

Co-authored-by: Keegan George <[email protected]>
Co-authored-by: Martin Brennan <[email protected]>
2024-12-12 09:17:25 +11:00
Martin Brennan ae80494448 UX: Improve rough edges of AI usage page (#1014)
* UX: Improve rough edges of AI usage page

* Ensure all text uses I18n
* Change from <button> usage to <DButton>
* Use <AdminConfigAreaCard> in place of custom card styles
* Format numbers nicely using our number format helper,
  show full values on hover using title attr
* Ensure 0 is always shown for counters, instead of being blank

* FEATURE: Load usage data after page load

Use ConditionalLoadingSpinner to hide load of usage
data, this prevents us hanging on page load with a white
screen.

* UX: Split users table, and add empty placeholders and page subheader

* DEV: Test fix
2024-12-12 08:55:24 +11:00
Keegan George a4440c507b UX: Make sentiment trends more readable (#1018)
Instead of a stacked chart showing a separate series for positive and negative, this PR introduces a simplification to the overall sentiment dashboard. It comprises the sentiment into a single series of the difference between `positive - negative` instead. This should allow for the data to be more easy to scan and look for trends
2024-12-11 09:13:18 -08:00
Roman Rizzi 5fc7a730ef FIX: Triage rule should append selected tags instead of replacing them (#1022) 2024-12-11 11:19:44 -03:00
Discourse Translator Bot 34f43f398d Update translations (#1015) 2024-12-10 20:16:44 +01:00
Roman Rizzi 6da35d8e66 FIX: Gemini inference client was missing #instance (#1019) 2024-12-10 15:42:31 -03:00
Keegan George 700e9de073 Revert "UX: Make sentiment trends more readable in time series data (#1013)" (#1016)
This reverts commit 375dd702b2.
2024-12-10 08:15:27 -08:00
Keegan George 375dd702b2 UX: Make sentiment trends more readable in time series data (#1013)
Instead of a stacked chart showing a separate series for positive and negative, this PR introduces a simplification to the overall sentiment dashboard. It comprises the sentiment into a single series of the difference between `positive - negative` instead. This should allow for the data to be more easy to scan and look for trends.
2024-12-10 07:22:41 -08:00
Sam 7ca21cc329 FEATURE: first class support for OpenRouter (#1011)
* FEATURE: first class support for OpenRouter

This new implementation supports picking quantization and provider pref

Also:

- Improve logging for summary generation
- Improve error message when contacting LLMs fails

* Better support for full screen artifacts on iPad

Support back button to close full screen
2024-12-10 05:59:19 +11:00
Kris 51e0a96e51 UX: ensure composer popup is always above AI icons (#1012) 2024-12-09 13:03:27 -05:00
Discourse Translator Bot 6ce14a778b Update translations (#1009) 2024-12-06 18:05:02 +01:00
Roman Rizzi 085dde7042 FEATURE: Select stop sequences from triage script (#1010) 2024-12-06 11:13:47 -03:00
Roman Rizzi 7ebbcd2de3 FIX: Make sure prompt uploads get included in the prompt when triaging (#1008) 2024-12-05 21:04:35 -03:00
Sam a55216773a FEATURE: Amazon Nova support via bedrock (#997)
Refactor dialect selection and add Nova API support

    Change dialect selection to use llm_model object instead of just provider name
    Add support for Amazon Bedrock's Nova API with native tools
    Implement Nova-specific message processing and formatting
    Update specs for Nova and AWS Bedrock endpoints
    Enhance AWS Bedrock support to handle Nova models
    Fix Gemini beta API detection logic
2024-12-06 07:45:58 +11:00
Kelv 5e87a50202 DEV: Update more deprecated Font Awesome icon names (#1005)
* DEV: Update more deprecated Font Awesome icon names

* update to trash-can
2024-12-06 07:45:05 +11:00
Kris 70fae2b699 FIX: close link in shared conversation model (#1007) 2024-12-05 15:19:36 -05:00
Rafael dos Santos Silva 5ba274f7a5 FIX: Change how we filter date for emotion on /filter (#1006) 2024-12-05 14:10:12 -03:00
Roman Rizzi 4ba74511c2 FIX: Make sure limits are updated and applied on each step (#1002) 2024-12-05 10:31:39 -03:00
Kris 50f61925eb FIX: typo in topic gist for escapedExcerpt (#1003) 2024-12-05 08:03:08 -05:00
Kris 625fd06744 UX: improve layout of inline title editing buttons (#1000) 2024-12-04 17:04:26 -05:00
Roman Rizzi b32b1cf241 FIX: Add a digest check to avoid repeteadly generating embeddings (bulk) (#1001) 2024-12-04 17:47:28 -03:00
Keegan George d6beac48f8 DEV: Improve explain suggestion footnote replacement (#999)
Previously, when clicking add footnote on an explain suggestion it would replace the selected word by finding the first occurrence of the word. This results in issues when there are more than one occurrences of a word in a post. This is not trivial to solve, so this PR instead prevents incorrect text replacements by only allowing the replacement if it's unique. We use the same logic here that we use to determine if something can be fast edited.

In this PR we also update tests for post helper explain suggestions. For a while, we haven't had tests here due to streaming/timing issues, we've been skipping our system specs. In this PR, we add acceptance tests to handle this which gives us improved ability to publish message bus updates in the testing environment so that it can be better tested without issues.
2024-12-04 11:41:34 -08:00
Rafael dos Santos Silva 938d4c018c FIX: More resilient sentiment backfill query (#998) 2024-12-04 12:10:31 -03:00
Discourse Translator Bot e7c2cd861a Update translations (#994) 2024-12-04 11:59:37 +01:00
Kelv f8a81a0beb DEV: Update deprecated Font Awesome icon names (#996) 2024-12-04 03:18:08 +01:00
KrisandKeegan George 8203bdfbc9 UX: move topic summary from DMenu to DModal (#992)
Co-authored-by: Keegan George <[email protected]>
2024-12-03 13:30:15 -05:00
Roman Rizzi ce6a2eca21 FEATURE: Backfill posts sentiment. (#982)
* FEATURE: Backfill posts sentiment.

It adds a scheduled job to backfill posts' sentiment, similar to our existing rake task, but with two settings to control the batch size and posts' max-age.

* Make sure model_name order is consistent.
2024-12-03 10:27:03 -03:00
Sam 7c65dd171f FIX: regression, no longer sending examples to AI helper (#993)
For a while now we have not been sending the examples to AI
helper, which can lead to inconsistent results.

Note: this also means that in non English we did not send
English results, so this may end up reducing performance

That said first thing we need to do is fix the regression.
2024-12-03 16:03:46 +11:00
Rafael dos Santos Silva e3f5e86dc5 FIX: AI Automation scripts were broken when using seeded models (#991) 2024-12-02 19:07:05 -03:00
Tobias Eigen fcf2350652 changed "contact this model to "use this model" server.en.yml (#989)
translators gave feedback that the phrase "contact this model" is confusing. changed it to "use this model".
2024-12-03 07:26:54 +11:00
Keegan George fc88bb08ab FIX: Tag suggester is suggesting already assigned tags (#990)
This PR fixes an issue where the tag suggester for edit title topic area was suggesting tags that are already assigned on a post. It also updates the amount of suggested tags to 7 so that there is still a decent amount of tags suggested when tags are already assigned.
2024-12-03 07:25:04 +11:00
Sam 117c06220e FEATURE: allow artifacts to be updated (#980)
Add support for versioned artifacts with improved diff handling

* Add versioned artifacts support allowing artifacts to be updated and tracked
  - New `ai_artifact_versions` table to store version history
  - Support for updating artifacts through a new `UpdateArtifact` tool
  - Add version-aware artifact rendering in posts
  - Include change descriptions for version tracking

* Enhance artifact rendering and security
  - Add support for module-type scripts and external JS dependencies
  - Expand CSP to allow trusted CDN sources (unpkg, cdnjs, jsdelivr, googleapis)
  - Improve JavaScript handling in artifacts

* Implement robust diff handling system (this is dormant but ready to use once LLMs catch up)
  - Add new DiffUtils module for applying changes to artifacts
  - Support for unified diff format with multiple hunks
  - Intelligent handling of whitespace and line endings
  - Comprehensive error handling for diff operations

* Update routes and UI components
  - Add versioned artifact routes
  - Update markdown processing for versioned artifacts

Also

- Tweaks summary prompt
- Improves upload support in custom tool to also provide urls
2024-12-03 07:23:31 +11:00
Rafael dos Santos Silva 0ac18d157b FEATURE: Adjustments to gist summaries (#988)
- makes visible to everyone by default
- backfills gists before full summaries
- adds configurable max age setting to backfill job
2024-12-02 15:22:35 -03:00
Keegan George 50ad5415ff UX: no-text style being incorrectly added to tag suggester (#987)
In the tag suggester menu we use `DButton` as a wrapper element and use the `discourseTag` helper to render the text inside the element. So visually there is text content inside the button. However, since `DButton` assumes that no `label`/`translatedLabel` inside an element means `.no-text` CSS style should be applied to the button's element, it was resulting in some incorrect styling being applied to this menu. This PR resolves that by programmatically adding the tag as a `translatedLabel` and then visually hiding it with CSS.
2024-12-02 10:11:04 -08:00
Tobias Eigen c53363406d changed "automatic auto" to "auto" in client.en.yml (#986)
The description of the automatic captions dialog prompt included "Would you like to enable automatic auto captions on image uploads." I removed the second "auto" at the suggestion of translators.
2024-12-02 09:32:08 -08:00
Rafael dos Santos Silva 3828370679 DEV: Cleanup deprecations (#952) 2024-12-02 14:18:03 -03:00
508 changed files with 863588 additions and 6563 deletions
+3
View File
@@ -1,3 +1,6 @@
< 3.5.0.beta1-dev: bdef136080074a993e7c4f5ca562edc31a8ba756
< 3.4.0.beta4-dev: a53719ab8eb071459f215227421b3ea4987e5f87
< 3.4.0.beta4-dev: 20612fde52d3f740cad64823ef8aadb0748b567f
< 3.4.0.beta3-dev: decf1bb49d737ea15308400f22f89d1d1e71d13d
< 3.4.0.beta1-dev: 9d887ad4ace8e33c3fe7dbb39237e882c08b4f0b
< 3.3.0.beta5-dev: 4d8090002f6dcd8e34d41033606bf131fa221475
+2
View File
@@ -2,3 +2,5 @@ node_modules
/gems
/auto_generated
.env
evals/log
evals/cases
+3
View File
@@ -2,3 +2,6 @@ inherit_gem:
rubocop-discourse: stree-compat.yml
RSpec/NamedSubject:
Enabled: false
Style/GlobalVars:
AllowedVariables: [$prometheus_client]
+1 -1
View File
@@ -29,7 +29,7 @@ GEM
racc
prettier_print (1.2.1)
racc (1.8.0)
rack (3.1.6)
rack (3.1.10)
rainbow (3.1.1)
regexp_parser (2.9.2)
rexml (3.3.9)
+7
View File
@@ -0,0 +1,7 @@
{
"tests": {
"requiredPlugins": [
"https://github.com/discourse/discourse-prometheus"
]
}
}
@@ -0,0 +1,21 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class AdminPluginsShowDiscourseAiEmbeddingsEdit extends DiscourseRoute {
async model(params) {
const allEmbeddings = this.modelFor(
"adminPlugins.show.discourse-ai-embeddings"
);
const id = parseInt(params.id, 10);
const record = allEmbeddings.findBy("id", id);
record.provider_params = record.provider_params || {};
return record;
}
setupController(controller, model) {
super.setupController(controller, model);
controller.set(
"allEmbeddings",
this.modelFor("adminPlugins.show.discourse-ai-embeddings")
);
}
}
@@ -0,0 +1,17 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class AdminPluginsShowDiscourseAiEmbeddingsNew extends DiscourseRoute {
async model() {
const record = this.store.createRecord("ai-embedding");
record.provider_params = {};
return record;
}
setupController(controller, model) {
super.setupController(controller, model);
controller.set(
"allEmbeddings",
this.modelFor("adminPlugins.show.discourse-ai-embeddings")
);
}
}
@@ -0,0 +1,7 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiAiEmbeddingsRoute extends DiscourseRoute {
model() {
return this.store.findAll("ai-embedding");
}
}
@@ -1,6 +1,6 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class AdminPluginsShowDiscourseAiLlmsShow extends DiscourseRoute {
export default class AdminPluginsShowDiscourseAiLlmsEdit extends DiscourseRoute {
async model(params) {
const allLlms = this.modelFor("adminPlugins.show.discourse-ai-llms");
const id = parseInt(params.id, 10);
@@ -1,6 +1,6 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class AdminPluginsShowDiscourseAiPersonasShow extends DiscourseRoute {
export default class AdminPluginsShowDiscourseAiPersonasEdit extends DiscourseRoute {
async model(params) {
const allPersonas = this.modelFor(
"adminPlugins.show.discourse-ai-personas"
@@ -0,0 +1,11 @@
import { service } from "@ember/service";
import { ajax } from "discourse/lib/ajax";
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiSpamRoute extends DiscourseRoute {
@service store;
model() {
return ajax("/admin/plugins/discourse-ai/ai-spam.json");
}
}
@@ -1,6 +1,6 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiToolsShowRoute extends DiscourseRoute {
export default class DiscourseAiToolsEditRoute extends DiscourseRoute {
async model(params) {
const allTools = this.modelFor("adminPlugins.show.discourse-ai-tools");
const id = parseInt(params.id, 10);
@@ -14,5 +14,7 @@ export default class DiscourseAiToolsShowRoute extends DiscourseRoute {
controller.set("allTools", toolsModel);
controller.set("presets", toolsModel.resultSetMeta.presets);
controller.set("llms", toolsModel.resultSetMeta.llms);
controller.set("settings", toolsModel.resultSetMeta.settings);
}
}
@@ -11,5 +11,7 @@ export default class DiscourseAiToolsNewRoute extends DiscourseRoute {
controller.set("allTools", toolsModel);
controller.set("presets", toolsModel.resultSetMeta.presets);
controller.set("llms", toolsModel.resultSetMeta.llms);
controller.set("settings", toolsModel.resultSetMeta.settings);
}
}
@@ -0,0 +1,4 @@
<AiEmbeddingsListEditor
@embeddings={{this.allEmbeddings}}
@currentEmbedding={{this.model}}
/>
@@ -0,0 +1 @@
<AiEmbeddingsListEditor @embeddings={{this.model}} />
@@ -0,0 +1,4 @@
<AiEmbeddingsListEditor
@embeddings={{this.allEmbeddings}}
@currentEmbedding={{this.model}}
/>
@@ -0,0 +1 @@
<AiSpam @model={{this.model}} />
@@ -3,5 +3,7 @@
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
@llms={{this.llms}}
@settings={{this.settings}}
/>
</section>
@@ -3,5 +3,7 @@
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
@llms={{this.llms}}
@settings={{this.settings}}
/>
</section>
@@ -0,0 +1,133 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiEmbeddingsController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
def index
embedding_defs = EmbeddingDefinition.all.order(:display_name)
render json: {
ai_embeddings:
ActiveModel::ArraySerializer.new(
embedding_defs,
each_serializer: AiEmbeddingDefinitionSerializer,
root: false,
).as_json,
meta: {
provider_params: EmbeddingDefinition.provider_params,
providers: EmbeddingDefinition.provider_names,
distance_functions: EmbeddingDefinition.distance_functions,
tokenizers:
EmbeddingDefinition.tokenizer_names.map { |tn|
{ id: tn, name: tn.split("::").last }
},
presets: EmbeddingDefinition.presets,
},
}
end
def new
end
def edit
embedding_def = EmbeddingDefinition.find(params[:id])
render json: AiEmbeddingDefinitionSerializer.new(embedding_def)
end
def create
embedding_def = EmbeddingDefinition.new(ai_embeddings_params)
if embedding_def.save
render json: AiEmbeddingDefinitionSerializer.new(embedding_def), status: :created
else
render_json_error embedding_def
end
end
def update
embedding_def = EmbeddingDefinition.find(params[:id])
if embedding_def.seeded?
return(
render_json_error(I18n.t("discourse_ai.embeddings.cannot_edit_builtin"), status: 403)
)
end
if embedding_def.update(ai_embeddings_params.except(:dimensions))
render json: AiEmbeddingDefinitionSerializer.new(embedding_def)
else
render_json_error embedding_def
end
end
def destroy
embedding_def = EmbeddingDefinition.find(params[:id])
if embedding_def.seeded?
return(
render_json_error(I18n.t("discourse_ai.embeddings.cannot_edit_builtin"), status: 403)
)
end
if embedding_def.id == SiteSetting.ai_embeddings_selected_model.to_i
return render_json_error(I18n.t("discourse_ai.embeddings.delete_failed"), status: 409)
end
if embedding_def.destroy
head :no_content
else
render_json_error embedding_def
end
end
def test
RateLimiter.new(
current_user,
"ai_embeddings_test_#{current_user.id}",
3,
1.minute,
).performed!
embedding_def = EmbeddingDefinition.new(ai_embeddings_params)
DiscourseAi::Embeddings::Vector.new(embedding_def).vector_from("this is a test")
render json: { success: true }
rescue Net::HTTPBadResponse => e
render json: { success: false, error: e.message }
end
private
def ai_embeddings_params
permitted =
params.require(:ai_embedding).permit(
:display_name,
:dimensions,
:max_sequence_length,
:pg_function,
:provider,
:url,
:api_key,
:tokenizer_class,
:embed_prompt,
:search_prompt,
:matryoshka_dimensions,
)
extra_field_names = EmbeddingDefinition.provider_params.dig(permitted[:provider]&.to_sym)
if extra_field_names.present?
received_prov_params =
params.dig(:ai_embedding, :provider_params)&.slice(*extra_field_names.keys)
if received_prov_params.present?
permitted[:provider_params] = received_prov_params.permit!
end
end
permitted
end
end
end
end
@@ -0,0 +1,59 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiLlmQuotasController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
def index
quotas = LlmQuota.includes(:group)
render json: {
quotas:
ActiveModel::ArraySerializer.new(quotas, each_serializer: LlmQuotaSerializer),
}
end
def create
quota = LlmQuota.new(quota_params)
if quota.save
render json: LlmQuotaSerializer.new(quota), status: :created
else
render_json_error quota
end
end
def update
quota = LlmQuota.find(params[:id])
if quota.update(quota_params)
render json: LlmQuotaSerializer.new(quota)
else
render_json_error quota
end
end
def destroy
quota = LlmQuota.find(params[:id])
quota.destroy!
head :no_content
rescue ActiveRecord::RecordNotFound
render json: { error: I18n.t("not_found") }, status: 404
end
private
def quota_params
params.require(:quota).permit(
:group_id,
:llm_model_id,
:max_tokens,
:max_usages,
:duration_seconds,
)
end
end
end
end
@@ -6,7 +6,7 @@ module DiscourseAi
requires_plugin ::DiscourseAi::PLUGIN_NAME
def index
llms = LlmModel.all.order(:display_name)
llms = LlmModel.all.includes(:llm_quotas).order(:display_name)
render json: {
ai_llms:
@@ -30,13 +30,21 @@ module DiscourseAi
}
end
def show
def new
end
def edit
llm_model = LlmModel.find(params[:id])
render json: LlmModelSerializer.new(llm_model)
end
def create
llm_model = LlmModel.new(ai_llm_params)
# we could do nested attributes but the mechanics are not ideal leading
# to lots of complex debugging, this is simpler
quota_params.each { |quota| llm_model.llm_quotas.build(quota) } if quota_params
if llm_model.save
llm_model.toggle_companion_user
render json: LlmModelSerializer.new(llm_model), status: :created
@@ -48,6 +56,25 @@ module DiscourseAi
def update
llm_model = LlmModel.find(params[:id])
if params[:ai_llm].key?(:llm_quotas)
if quota_params
existing_quota_group_ids = llm_model.llm_quotas.pluck(:group_id)
new_quota_group_ids = quota_params.map { |q| q[:group_id] }
llm_model
.llm_quotas
.where(group_id: existing_quota_group_ids - new_quota_group_ids)
.destroy_all
quota_params.each do |quota_param|
quota = llm_model.llm_quotas.find_or_initialize_by(group_id: quota_param[:group_id])
quota.update!(quota_param)
end
else
llm_model.llm_quotas.destroy_all
end
end
if llm_model.seeded?
return render_json_error(I18n.t("discourse_ai.llm.cannot_edit_builtin"), status: 403)
end
@@ -107,6 +134,19 @@ module DiscourseAi
private
def quota_params
if params[:ai_llm][:llm_quotas].present?
params[:ai_llm][:llm_quotas].map do |quota|
mapped = {}
mapped[:group_id] = quota[:group_id].to_i
mapped[:max_tokens] = quota[:max_tokens].to_i if quota[:max_tokens].present?
mapped[:max_usages] = quota[:max_usages].to_i if quota[:max_usages].present?
mapped[:duration_seconds] = quota[:duration_seconds].to_i
mapped
end
end
end
def ai_llm_params(updating: nil)
return {} if params[:ai_llm].blank?
@@ -5,7 +5,7 @@ module DiscourseAi
class AiPersonasController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
before_action :find_ai_persona, only: %i[show update destroy create_user]
before_action :find_ai_persona, only: %i[edit update destroy create_user]
def index
ai_personas =
@@ -23,17 +23,34 @@ module DiscourseAi
.each do |tool|
tools << {
id: "custom-#{tool.id}",
name: I18n.t("discourse_ai.tools.custom_name", name: tool.name.capitalize),
name:
I18n.t(
"discourse_ai.tools.custom_name",
name: tool.name.capitalize,
tool_name: tool.tool_name,
),
}
end
llms =
DiscourseAi::Configuration::LlmEnumerator.values.map do |hash|
{ id: hash[:value], name: hash[:name] }
end
render json: { ai_personas: ai_personas, meta: { tools: tools, llms: llms } }
DiscourseAi::Configuration::LlmEnumerator.values_for_serialization(
allowed_seeded_llm_ids: SiteSetting.ai_bot_allowed_seeded_models_map,
)
render json: {
ai_personas: ai_personas,
meta: {
tools: tools,
llms: llms,
settings: {
rag_images_enabled: SiteSetting.ai_rag_images_enabled,
},
},
}
end
def show
def new
end
def edit
render json: LocalizedAiPersonaSerializer.new(@ai_persona)
end
@@ -179,7 +196,7 @@ module DiscourseAi
:priority,
:top_p,
:temperature,
:default_llm,
:default_llm_id,
:user_id,
:max_context_posts,
:vision_enabled,
@@ -187,7 +204,8 @@ module DiscourseAi
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_conversation_chunks,
:question_consolidator_llm,
:rag_llm_model_id,
:question_consolidator_llm_id,
:allow_chat_channel_mentions,
:allow_chat_direct_messages,
:allow_topic_mentions,
@@ -0,0 +1,137 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiSpamController < ::Admin::AdminController
requires_plugin "discourse-ai"
def show
render json: AiSpamSerializer.new(spam_config, root: false)
end
def update
updated_params = {}
if allowed_params.key?(:llm_model_id)
llm_model_id = updated_params[:llm_model_id] = allowed_params[:llm_model_id]
if llm_model_id.to_i < 0 &&
!SiteSetting.ai_spam_detection_model_allowed_seeded_models_map.include?(
llm_model_id.to_s,
)
return(
render_json_error(
I18n.t("discourse_ai.llm.configuration.invalid_seeded_model"),
status: 422,
)
)
end
end
updated_params[:data] = {
custom_instructions: allowed_params[:custom_instructions],
} if allowed_params.key?(:custom_instructions)
if updated_params.present?
# not using upsert cause we will not get the correct validation errors
if AiModerationSetting.spam
AiModerationSetting.spam.update!(updated_params)
else
AiModerationSetting.create!(updated_params.merge(setting_type: :spam))
end
end
is_enabled = ActiveModel::Type::Boolean.new.cast(allowed_params[:is_enabled])
if allowed_params.key?(:is_enabled)
if is_enabled && !AiModerationSetting.spam&.llm_model_id
return(
render_json_error(
I18n.t("discourse_ai.llm.configuration.must_select_model"),
status: 422,
)
)
end
SiteSetting.ai_spam_detection_enabled = is_enabled
end
render json: AiSpamSerializer.new(spam_config, root: false)
end
def test
url = params[:post_url].to_s
post = nil
if url.match?(/^\d+$/)
post_id = url.to_i
post = Post.find_by(id: post_id)
end
route = UrlHelper.rails_route_from_url(url) if !post
if route
if route[:controller] == "topics"
post_number = route[:post_number] || 1
post = Post.with_deleted.find_by(post_number: post_number, topic_id: route[:topic_id])
end
end
raise Discourse::NotFound if !post
result =
DiscourseAi::AiModeration::SpamScanner.test_post(
post,
custom_instructions: params[:custom_instructions],
llm_id: params[:llm_id],
)
render json: result
end
def fix_errors
case params[:error]
when "spam_scanner_not_admin"
begin
DiscourseAi::AiModeration::SpamScanner.fix_spam_scanner_not_admin
render json: success_json
rescue ActiveRecord::RecordInvalid
render_json_error(
I18n.t("discourse_ai.spam_detection.bot_user_update_failed"),
status: :unprocessable_entity,
)
rescue StandardError
render_json_error(
I18n.t("discourse_ai.spam_detection.unexpected"),
status: :internal_server_error,
)
end
else
render_json_error(
I18n.t("discourse_ai.spam_detection.invalid_error_type"),
status: :bad_request,
)
end
end
private
def allowed_params
params.permit(:is_enabled, :llm_model_id, :custom_instructions)
end
def spam_config
spam_config = {
enabled: SiteSetting.ai_spam_detection_enabled,
settings: AiModerationSetting.spam,
}
spam_config[:stats] = DiscourseAi::AiModeration::SpamReport.generate(min_date: 1.week.ago)
if spam_config[:stats].scanned_count > 0
spam_config[
:flagging_username
] = DiscourseAi::AiModeration::SpamScanner.flagging_user&.username
end
spam_config
end
end
end
end
@@ -5,14 +5,17 @@ module DiscourseAi
class AiToolsController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
before_action :find_ai_tool, only: %i[test show update destroy]
before_action :find_ai_tool, only: %i[test edit update destroy]
def index
ai_tools = AiTool.all
render_serialized({ ai_tools: ai_tools }, AiCustomToolListSerializer, root: false)
end
def show
def new
end
def edit
render_serialized(@ai_tool, AiCustomToolSerializer)
end
@@ -81,11 +84,13 @@ module DiscourseAi
.require(:ai_tool)
.permit(
:name,
:tool_name,
:description,
:script,
:summary,
:rag_chunk_tokens,
:rag_chunk_overlap_tokens,
:rag_llm_model_id,
rag_uploads: [:id],
parameters: [:name, :type, :description, :required, enum: []],
)
@@ -6,6 +6,9 @@ module DiscourseAi
requires_plugin "discourse-ai"
def show
end
def report
render json: AiUsageSerializer.new(create_report, root: false)
end
@@ -20,7 +20,7 @@ module DiscourseAi
def upload_file
file = params[:file] || params[:files].first
if !SiteSetting.ai_embeddings_enabled?
if !DiscourseAi::Embeddings.enabled?
raise Discourse::InvalidAccess.new("Embeddings not enabled")
end
@@ -48,7 +48,8 @@ module DiscourseAi
def validate_extension!(filename)
extension = File.extname(filename)[1..-1] || ""
authorized_extensions = %w[txt md]
authorized_extensions = %w[txt md pdf]
authorized_extensions.concat(%w[png jpg jpeg]) if SiteSetting.ai_rag_images_enabled
if !authorized_extensions.include?(extension)
raise Discourse::InvalidParameters.new(
I18n.t(
@@ -12,29 +12,40 @@ module DiscourseAi
artifact = AiArtifact.find(params[:id])
post = Post.find_by(id: artifact.post_id)
if artifact.metadata&.dig("public")
if artifact.public?
# no guardian needed
else
raise Discourse::NotFound if !post&.topic&.private_message?
raise Discourse::NotFound if !guardian.can_see?(post)
end
name = artifact.name
if params[:version].present?
artifact = artifact.versions.find_by(version_number: params[:version])
raise Discourse::NotFound if !artifact
end
js = artifact.js || ""
if !js.match?(%r{\A\s*<script.*</script>}mi)
mod = ""
mod = " type=\"module\"" if js.match?(/\A\s*import.*/)
js = "<script#{mod}>\n#{js}\n</script>"
end
# Prepare the inner (untrusted) HTML document
untrusted_html = <<~HTML
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>#{ERB::Util.html_escape(artifact.name)}</title>
<title>#{ERB::Util.html_escape(name)}</title>
<style>
#{artifact.css}
</style>
</head>
<body>
#{artifact.html}
<script>
#{artifact.js}
</script>
#{js}
</body>
</html>
HTML
@@ -45,7 +56,8 @@ module DiscourseAi
<html>
<head>
<meta charset="UTF-8">
<title>#{ERB::Util.html_escape(artifact.name)}</title>
<title>#{ERB::Util.html_escape(name)}</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover, interactive-widget=resizes-content">
<style>
html, body, iframe {
margin: 0;
@@ -67,7 +79,9 @@ module DiscourseAi
HTML
response.headers.delete("X-Frame-Options")
response.headers["Content-Security-Policy"] = "script-src 'unsafe-inline';"
response.headers[
"Content-Security-Policy"
] = "script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval' #{AiArtifact::ALLOWED_CDN_SOURCES.join(" ")};"
response.headers["X-Robots-Tag"] = "noindex"
# Render the content

Some files were not shown because too many files have changed in this diff Show More