Compare commits

...
Author SHA1 Message Date
Roman Rizzi f1598b4679 FIX: Use COPY to avoid migration statement timeout 2025-01-14 12:23:49 -03:00
Roman Rizzi 65bbcd71fc DEV: Embedding tables' model_id has to be a bigint (#1058)
* DEV: Embedding tables' model_id has to be a bigint

* Drop old search_bit indexes

* copy rag fragment embeddings created during deploy window
2025-01-14 10:53:06 -03:00
SamandKeegan George d07cf51653 FEATURE: llm quotas (#1047)
Adds a comprehensive quota management system for LLM models that allows:

- Setting per-group (applied per user in the group) token and usage limits with configurable durations
- Tracking and enforcing token/usage limits across user groups
- Quota reset periods (hourly, daily, weekly, or custom)
-  Admin UI for managing quotas with real-time updates

This system provides granular control over LLM API usage by allowing admins
to define limits on both total tokens and number of requests per group.
Supports multiple concurrent quotas per model and automatically handles
quota resets.


Co-authored-by: Keegan George <[email protected]>
2025-01-14 15:54:09 +11:00
Sam 20612fde52 FEATURE: add the ability to disable streaming on an Open AI LLM
Disabling streaming is required for models such o1 that do not have streaming
enabled yet

It is good to carry this feature around in case various apis decide not to support streaming endpoints and Discourse AI can continue to work just as it did before. 

Also: fixes issue where sharing artifacts would miss viewport leading to tiny artifacts on mobile
2025-01-13 17:01:01 +11:00
Jarek Radosz 7e9c0dc076 FIX: Invalid locale yaml (#1057) 2025-01-11 23:31:33 +01:00
Mark VanLandingham 2d1ce01320 DEV: Add no-results className to full page search toggle (#1055) 2025-01-10 10:59:18 -06:00
Keegan George b24669c810 DEV: Add structure for errors in spam (#1054)
This update adds some structure for handling errors in the spam config while also handling a specific error related to the spam scanning user not being an admin account.
2025-01-09 09:17:06 -08:00
Keegan George 24b69bf840 FIX: Update spam controller action should consider seeded LLM properly (#1053)
The seeded LLM setting: `SiteSetting.ai_spam_detection_model_allowed_seeded_models` returns a _string_ with IDs separated by pipes. running `_map` on it will return an array with strings. We were previously checking for the id with custom prefix identifier, but instead we should be checking the stringified ID.
2025-01-08 13:41:25 -08:00
Guhyoun Nam 404092a68c DEV: Add appEvents trigger when Ai search results toggled (#1052)
This PR adds appEvents triggers when Ai search results are toggled.
2025-01-08 12:17:25 -06:00
Mark VanLandingham 327adbde29 UX: Full page search -- always show tooltip & add msg (#1051) 2025-01-08 09:05:30 -06:00
Kris 749af40fad UX: close summary modal on click outside (#1050) 2025-01-07 11:24:27 -05:00
Discourse Translator Bot 61758ff8a6 Update translations (#1040) 2025-01-03 14:01:41 +01:00
Mark VanLandingham b6cefd10fa DEV: Move semantic search from connector to component (#1048) 2025-01-02 12:32:49 -06:00
Sam 11d0f60f1e FEATURE: smart date support for AI helper (#1044)
* FEATURE: smart date support for AI helper

This feature allows conversion of human typed in dates and times
to smart "Discourse" timezone friendly dates.

* fix specs and lint

* lint

* address feedback

* add specs
2024-12-31 08:04:25 +11:00
Sam f9f89adac5 FIX: keep track of silence reason when spam detection flags user (#1046)
Previously reason was blank for silencing user
2024-12-27 17:47:16 +11:00
Keegan George b480f13a0f FIX: Prevent LLM enumerator from erroring when spam enabled (#1045)
This PR fixes an issue where LLM enumerator would error out when `SiteSetting.ai_spam_detection = true` but there was no `AiModerationSetting.spam` present.

Typically, we add an `LlmDependencyValidator` for the setting itself, however, since Spam is unique in that it has it's model set in `AiModerationSetting` instead of a `SiteSetting`, we'll add a simple check here to prevent erroring out.
2024-12-27 09:12:29 +11:00
Sam 47ecf86aa1 FIX: embedding validation (#1043) 2024-12-24 09:37:23 +11:00
Rafael dos Santos Silva 792df58fbc FIX: AI Helper category / tag suggestion when user does not categories muted (#1042) 2024-12-23 15:58:26 -03:00
Roman Rizzi ceac6e5efb FIX: Embeddings validator test needs to use the new Vector class. (#1041) 2024-12-23 14:19:22 -03:00
Keegan George bdb8f1d5e0 FIX: Custom prefix causing allowed seeded LLMs not to be shown (#1039)
* FIX: Custom prefix causing allowed seeded LLMs not to be shown

* DEV: update spec

* not `_map` so should be string not array
2024-12-23 16:42:26 +11:00
Kris d15876025f UX: disabled preseeded edit button, add description (#1038) 2024-12-20 19:33:45 -05:00
Rafael dos Santos Silva 7607477ff9 FIX: Cloudflare Workers AI embeddings (#1037)
Regressed on 534b0df
2024-12-20 17:45:27 -03:00
Keegan George 059b3fabb8 DEV: Unreachable LLM error shouldn't prevent setting (#1036)
Previously we had the behaviour for model settings so that when you try and set a model, it runs a test and returns an error if it can't run the test successfully. The error then prevents you from setting the site setting.

This results in some issues when we try and automate things. This PR updates that so that the test runs and discreetly logs the changes, but doesn't prevent the setting from being set. Instead we rely on "run test" in the LLM config along with ProblemChecks to catch issues.
2024-12-20 11:52:11 -08:00
Sam 6a7a45fd4f FIX: properly spin down unused streamer threads (#1035)
Previous version was prone to the bug:

https://github.com/ruby-concurrency/concurrent-ruby/issues/1075

This is particularly bad cause we could have a DB connection
attached to the thread and we never clear it up, so after N hours
this could start exhibiting weird connection issues.
2024-12-20 12:09:42 +11:00
Kris ac705b694b UX: minor improvements to LLM page and admin tables (#1034) 2024-12-19 18:14:22 -05:00
Discourse Translator Bot a4033e2af9 Update translations (#1032) 2024-12-18 15:19:47 +01:00
Martin Brennan f35db8068b DEV: Change to use DPageSubheader (#1033)
Previously was AdminPageSubheader until https://github.com/discourse/discourse/pull/30146
2024-12-18 17:39:31 +10:00
Sam fae2d5ff2c FEATURE: link correctly to filters to assist in debugging spam (#1031)
- Add spam_score_type to AiSpamSerializer for better integration with reviewables.
- Introduce a custom filter for detecting AI spam false negatives in moderation workflows.
- Refactor spam report generation to improve identification of false negatives.
- Add tests to verify the custom filter and its behavior.
- Introduce links for all spam counts in report
2024-12-17 11:02:18 +11:00
Keegan George 90ce942108 FEATURE: Add periodic problem checks for each LLM in use (#1020)
This feature adds a periodic problem check which periodically checks for issues with LLMs that are in use. Periodically, we will run a test to see if the in use LLMs are still operational. If it is not, the LLM with the problem is surfaced to the admin so they can easily go and update the configuration.
2024-12-16 15:00:05 -08:00
Mark VanLandingham 24b107881a FEATURE: Unavailable state for semantic search when sort is not Relevant (#1030)
This commit adds an "unavailable" state for the AI semantic search toggle. Currently the AI toggle disappears when the sort by is anything but Relevance which makes the UI confusing for users looking for AI results. This should help!
2024-12-16 14:30:11 -06:00
Roman Rizzi 534b0df391 REFACTOR: Separation of concerns for embedding generation. (#1027)
In a previous refactor, we moved the responsibility of querying and storing embeddings into the `Schema` class. Now, it's time for embedding generation.

The motivation behind these changes is to isolate vector characteristics in simple objects to later replace them with a DB-backed version, similar to what we did with LLM configs.
2024-12-16 09:55:39 -03:00
Martin Brennan 222e2cf4f9 UX: Use new DStatTiles reusable component from core (#1025)
For the Spam and Usage tabs in admin
2024-12-16 16:48:46 +10:00
Roman RizziandKeegan George 94b85ece80 FIX: Make sure gists are atleast five minutes old before updating them (#1029)
* FIX: Make sure gists are atleast five minutes old before updating them

* Update app/jobs/regular/fast_track_topic_gist.rb

Co-authored-by: Keegan George <[email protected]>

---------

Co-authored-by: Keegan George <[email protected]>
2024-12-13 19:36:34 -03:00
Roman Rizzi 1c40a698ca FIX: get strategy version through vector_rep (#1028) 2024-12-13 18:49:18 -03:00
David Taylor fae1fbc796 DEV: Correct i18n lint violation (#1026) 2024-12-13 16:01:04 +00:00
Roman Rizzi eae527f99d REFACTOR: A Simpler way of interacting with embeddings tables. (#1023)
* REFACTOR: A Simpler way of interacting with embeddings' tables.

This change adds a new abstraction called `Schema`, which acts as a repository that supports the same DB features `VectorRepresentation::Base` has, with the exception that removes the need to have duplicated methods per embeddings table.

It is also a bit more flexible when performing a similarity search because you can pass it a block that gives you access to the builder, allowing you to add multiple joins/where conditions.
2024-12-13 10:15:21 -03:00
Roman Rizzi 97ec2c5ff4 FEATURE: Show gists everywhere except suggested/related (#995) 2024-12-12 12:29:35 -03:00
Krzysztof Kotlarek 04c4ff8cf0 UX: No admin header for edit personas tools or llms (#1021)
In this PR, we added functionality to hide the admin header for edit/new actions - https://github.com/discourse/discourse/pull/30175

To make it work properly, we have to rename `show` to `edit` which is also a more accurate name.
2024-12-12 10:48:58 +11:00
Sam 47c1ea337e FIX: allow scanning of trashed posts and deleted users for test (#1024)
When a post is trashed we should still be allowed to scan it
post.topic will be nil for a trashed topic even if post is trashed
2024-12-12 10:26:05 +11:00
47f5da7e42 FEATURE: Add AI-powered spam detection for new user posts (#1004)
This introduces a comprehensive spam detection system that uses LLM models
to automatically identify and flag potential spam posts. The system is
designed to be both powerful and configurable while preventing false positives.

Key Features:
* Automatically scans first 3 posts from new users (TL0/TL1)
* Creates dedicated AI flagging user to distinguish from system flags
* Tracks false positives/negatives for quality monitoring
* Supports custom instructions to fine-tune detection
* Includes test interface for trying detection on any post

Technical Implementation:
* New database tables:
  - ai_spam_logs: Stores scan history and results
  - ai_moderation_settings: Stores LLM config and custom instructions
* Rate limiting and safeguards:
  - Minimum 10-minute delay between rescans
  - Only scans significant edits (>10 char difference)
  - Maximum 3 scans per post
  - 24-hour maximum age for scannable posts
* Admin UI features:
  - Real-time testing capabilities
  - 7-day statistics dashboard
  - Configurable LLM model selection
  - Custom instruction support

Security and Performance:
* Respects trust levels - only scans TL0/TL1 users
* Skips private messages entirely
* Stops scanning users after 3 successful public posts
* Includes comprehensive test coverage
* Maintains audit log of all scan attempts


---------

Co-authored-by: Keegan George <[email protected]>
Co-authored-by: Martin Brennan <[email protected]>
2024-12-12 09:17:25 +11:00
Martin Brennan ae80494448 UX: Improve rough edges of AI usage page (#1014)
* UX: Improve rough edges of AI usage page

* Ensure all text uses I18n
* Change from <button> usage to <DButton>
* Use <AdminConfigAreaCard> in place of custom card styles
* Format numbers nicely using our number format helper,
  show full values on hover using title attr
* Ensure 0 is always shown for counters, instead of being blank

* FEATURE: Load usage data after page load

Use ConditionalLoadingSpinner to hide load of usage
data, this prevents us hanging on page load with a white
screen.

* UX: Split users table, and add empty placeholders and page subheader

* DEV: Test fix
2024-12-12 08:55:24 +11:00
Keegan George a4440c507b UX: Make sentiment trends more readable (#1018)
Instead of a stacked chart showing a separate series for positive and negative, this PR introduces a simplification to the overall sentiment dashboard. It comprises the sentiment into a single series of the difference between `positive - negative` instead. This should allow for the data to be more easy to scan and look for trends
2024-12-11 09:13:18 -08:00
Roman Rizzi 5fc7a730ef FIX: Triage rule should append selected tags instead of replacing them (#1022) 2024-12-11 11:19:44 -03:00
Discourse Translator Bot 34f43f398d Update translations (#1015) 2024-12-10 20:16:44 +01:00
Roman Rizzi 6da35d8e66 FIX: Gemini inference client was missing #instance (#1019) 2024-12-10 15:42:31 -03:00
Keegan George 700e9de073 Revert "UX: Make sentiment trends more readable in time series data (#1013)" (#1016)
This reverts commit 375dd702b2.
2024-12-10 08:15:27 -08:00
Keegan George 375dd702b2 UX: Make sentiment trends more readable in time series data (#1013)
Instead of a stacked chart showing a separate series for positive and negative, this PR introduces a simplification to the overall sentiment dashboard. It comprises the sentiment into a single series of the difference between `positive - negative` instead. This should allow for the data to be more easy to scan and look for trends.
2024-12-10 07:22:41 -08:00
Sam 7ca21cc329 FEATURE: first class support for OpenRouter (#1011)
* FEATURE: first class support for OpenRouter

This new implementation supports picking quantization and provider pref

Also:

- Improve logging for summary generation
- Improve error message when contacting LLMs fails

* Better support for full screen artifacts on iPad

Support back button to close full screen
2024-12-10 05:59:19 +11:00
Kris 51e0a96e51 UX: ensure composer popup is always above AI icons (#1012) 2024-12-09 13:03:27 -05:00
Discourse Translator Bot 6ce14a778b Update translations (#1009) 2024-12-06 18:05:02 +01:00
Roman Rizzi 085dde7042 FEATURE: Select stop sequences from triage script (#1010) 2024-12-06 11:13:47 -03:00
Roman Rizzi 7ebbcd2de3 FIX: Make sure prompt uploads get included in the prompt when triaging (#1008) 2024-12-05 21:04:35 -03:00
Sam a55216773a FEATURE: Amazon Nova support via bedrock (#997)
Refactor dialect selection and add Nova API support

    Change dialect selection to use llm_model object instead of just provider name
    Add support for Amazon Bedrock's Nova API with native tools
    Implement Nova-specific message processing and formatting
    Update specs for Nova and AWS Bedrock endpoints
    Enhance AWS Bedrock support to handle Nova models
    Fix Gemini beta API detection logic
2024-12-06 07:45:58 +11:00
Kelv 5e87a50202 DEV: Update more deprecated Font Awesome icon names (#1005)
* DEV: Update more deprecated Font Awesome icon names

* update to trash-can
2024-12-06 07:45:05 +11:00
Kris 70fae2b699 FIX: close link in shared conversation model (#1007) 2024-12-05 15:19:36 -05:00
Rafael dos Santos Silva 5ba274f7a5 FIX: Change how we filter date for emotion on /filter (#1006) 2024-12-05 14:10:12 -03:00
Roman Rizzi 4ba74511c2 FIX: Make sure limits are updated and applied on each step (#1002) 2024-12-05 10:31:39 -03:00
Kris 50f61925eb FIX: typo in topic gist for escapedExcerpt (#1003) 2024-12-05 08:03:08 -05:00
Kris 625fd06744 UX: improve layout of inline title editing buttons (#1000) 2024-12-04 17:04:26 -05:00
Roman Rizzi b32b1cf241 FIX: Add a digest check to avoid repeteadly generating embeddings (bulk) (#1001) 2024-12-04 17:47:28 -03:00
Keegan George d6beac48f8 DEV: Improve explain suggestion footnote replacement (#999)
Previously, when clicking add footnote on an explain suggestion it would replace the selected word by finding the first occurrence of the word. This results in issues when there are more than one occurrences of a word in a post. This is not trivial to solve, so this PR instead prevents incorrect text replacements by only allowing the replacement if it's unique. We use the same logic here that we use to determine if something can be fast edited.

In this PR we also update tests for post helper explain suggestions. For a while, we haven't had tests here due to streaming/timing issues, we've been skipping our system specs. In this PR, we add acceptance tests to handle this which gives us improved ability to publish message bus updates in the testing environment so that it can be better tested without issues.
2024-12-04 11:41:34 -08:00
Rafael dos Santos Silva 938d4c018c FIX: More resilient sentiment backfill query (#998) 2024-12-04 12:10:31 -03:00
Discourse Translator Bot e7c2cd861a Update translations (#994) 2024-12-04 11:59:37 +01:00
Kelv f8a81a0beb DEV: Update deprecated Font Awesome icon names (#996) 2024-12-04 03:18:08 +01:00
KrisandKeegan George 8203bdfbc9 UX: move topic summary from DMenu to DModal (#992)
Co-authored-by: Keegan George <[email protected]>
2024-12-03 13:30:15 -05:00
Roman Rizzi ce6a2eca21 FEATURE: Backfill posts sentiment. (#982)
* FEATURE: Backfill posts sentiment.

It adds a scheduled job to backfill posts' sentiment, similar to our existing rake task, but with two settings to control the batch size and posts' max-age.

* Make sure model_name order is consistent.
2024-12-03 10:27:03 -03:00
Sam 7c65dd171f FIX: regression, no longer sending examples to AI helper (#993)
For a while now we have not been sending the examples to AI
helper, which can lead to inconsistent results.

Note: this also means that in non English we did not send
English results, so this may end up reducing performance

That said first thing we need to do is fix the regression.
2024-12-03 16:03:46 +11:00
Rafael dos Santos Silva e3f5e86dc5 FIX: AI Automation scripts were broken when using seeded models (#991) 2024-12-02 19:07:05 -03:00
Tobias Eigen fcf2350652 changed "contact this model to "use this model" server.en.yml (#989)
translators gave feedback that the phrase "contact this model" is confusing. changed it to "use this model".
2024-12-03 07:26:54 +11:00
Keegan George fc88bb08ab FIX: Tag suggester is suggesting already assigned tags (#990)
This PR fixes an issue where the tag suggester for edit title topic area was suggesting tags that are already assigned on a post. It also updates the amount of suggested tags to 7 so that there is still a decent amount of tags suggested when tags are already assigned.
2024-12-03 07:25:04 +11:00
Sam 117c06220e FEATURE: allow artifacts to be updated (#980)
Add support for versioned artifacts with improved diff handling

* Add versioned artifacts support allowing artifacts to be updated and tracked
  - New `ai_artifact_versions` table to store version history
  - Support for updating artifacts through a new `UpdateArtifact` tool
  - Add version-aware artifact rendering in posts
  - Include change descriptions for version tracking

* Enhance artifact rendering and security
  - Add support for module-type scripts and external JS dependencies
  - Expand CSP to allow trusted CDN sources (unpkg, cdnjs, jsdelivr, googleapis)
  - Improve JavaScript handling in artifacts

* Implement robust diff handling system (this is dormant but ready to use once LLMs catch up)
  - Add new DiffUtils module for applying changes to artifacts
  - Support for unified diff format with multiple hunks
  - Intelligent handling of whitespace and line endings
  - Comprehensive error handling for diff operations

* Update routes and UI components
  - Add versioned artifact routes
  - Update markdown processing for versioned artifacts

Also

- Tweaks summary prompt
- Improves upload support in custom tool to also provide urls
2024-12-03 07:23:31 +11:00
Rafael dos Santos Silva 0ac18d157b FEATURE: Adjustments to gist summaries (#988)
- makes visible to everyone by default
- backfills gists before full summaries
- adds configurable max age setting to backfill job
2024-12-02 15:22:35 -03:00
Keegan George 50ad5415ff UX: no-text style being incorrectly added to tag suggester (#987)
In the tag suggester menu we use `DButton` as a wrapper element and use the `discourseTag` helper to render the text inside the element. So visually there is text content inside the button. However, since `DButton` assumes that no `label`/`translatedLabel` inside an element means `.no-text` CSS style should be applied to the button's element, it was resulting in some incorrect styling being applied to this menu. This PR resolves that by programmatically adding the tag as a `translatedLabel` and then visually hiding it with CSS.
2024-12-02 10:11:04 -08:00
Tobias Eigen c53363406d changed "automatic auto" to "auto" in client.en.yml (#986)
The description of the automatic captions dialog prompt included "Would you like to enable automatic auto captions on image uploads." I removed the second "auto" at the suggestion of translators.
2024-12-02 09:32:08 -08:00
Rafael dos Santos Silva 3828370679 DEV: Cleanup deprecations (#952) 2024-12-02 14:18:03 -03:00
Roman Rizzi 0abd4b1244 FIX: Sentiment classification results needs to be transformed before saving (#983) 2024-11-29 17:31:56 -03:00
David Taylor 120a20c5cd DEV: Convert to native class syntax (#981) 2024-11-29 16:41:09 +01:00
Sam 0cb2c413ba FEATURE: exclude muted categories from category suggester (#979)
The logic here is that users do not particularly care about
topics in the category so we can exclude them from tag
and category suggestions
2024-11-29 12:17:28 +11:00
Rafael dos Santos Silva 80adefa1c1 FIX: Move count logic to the end for tag suggestions (#978) 2024-11-28 16:27:38 -03:00
Sam bc0657f478 FEATURE: AI Usage page (#964)
- Added a new admin interface to track AI usage metrics, including tokens, features, and models.
- Introduced a new route `/admin/plugins/discourse-ai/ai-usage` and supporting API endpoint in `AiUsageController`.
- Implemented `AiUsageSerializer` for structuring AI usage data.
- Integrated CSS stylings for charts and tables under `stylesheets/modules/llms/common/usage.scss`.
- Enhanced backend with `AiApiAuditLog` model changes: added `cached_tokens` column  (implemented with OpenAI for now) with relevant DB migration and indexing.
- Created `Report` module for efficient aggregation and filtering of AI usage metrics.
- Updated AI Bot title generation logic to log correctly to user vs bot
- Extended test coverage for the new tracking features, ensuring data consistency and access controls.
2024-11-29 06:26:48 +11:00
Roman Rizzi c980c34d77 REFACTOR: Simplify sentiment classification (#977)
This change adds a simpler class for sentiment classification, replacing the soon-to-be removed `Classificator` hierarchy. Additionally, it adds a method for classifying concurrently, speeding up the backfill rake task.
2024-11-28 15:38:23 -03:00
Rafael dos Santos Silva 6456a4f44a FIX: Keep related topics data cached when a user scrolls up (#976)
See https://meta.discourse.org/t/related-topics-missing-after-rereading-a-topic-with-more-than-20-posts/306147/6
2024-11-28 14:25:15 -03:00
Rafael dos Santos Silva eacc4ed945 UX: Fix styling of AI Helper on quick edit (#975) 2024-11-28 13:06:48 -03:00
David Taylor 4a2369356f DEV: Update linting (#974) 2024-11-28 16:07:42 +01:00
Rafael dos Santos Silva 4980a4b2f7 FIX: Multiple concurrent summaries could result in pg index errors (#973) 2024-11-28 11:53:04 -03:00
Keegan George 54ebbbe769 UX: Ensure suggestions are left aligned (#971)
This PR ensures that suggestions are aligned to the left. By default, they are center aligned which looks odd when there are long lines of text.
2024-11-27 20:53:30 -08:00
Keegan George 8b33ba3d19 FIX: Suggestion menus not showing in mobile (#970)
The suggestion menu z-index was not enough for mobile so it isn't being shown when it's used in `modalForMobile` bottom menus.
2024-11-27 19:51:03 -08:00
Keegan George f1c7ee8624 DEV: Better control what prompts can appear in post/composer (#969)
This PR updates the logic for the location map so it permits only the desired prompts through to the composer/post menu. Anything else won't be shown by default.

This PR also adds relevant tests to prevent regression.
2024-11-27 16:14:21 -08:00
Keegan George 4da033c667 FIX: Double render error with thumbnail suggestions (#968)
This PR fixes a bug where a double render error appears in the logs when thumbnails are suggested
2024-11-27 15:12:27 -08:00
Keegan George dabef02919 DEV: Prevent detect_text_locale from appearing in menus (#967)
### 🔍 Overview
With the recent changes to allow DiscourseAi in the translator plugin, `detect_text_locale` was needed as a CompletionPrompt. However, it is leaking into composer/post helper menus. This PR ensures we don't not show it in those menus.
2024-11-28 09:27:08 +11:00
Sam fbcc8e493a FEATURE: Skip PM scanning in LLM triage by default (#966)
Usually people do not want to scan personal messages. Sometimes
they may. In that case they can enable triage on personal messages.
2024-11-28 09:25:29 +11:00
Keegan George 6b7d7c1179 REFACTOR: Helper suggestions (#914)
This PR adds some updates to the Helper suggestions to improve it's functionality and modernize some of the codebase.
2024-11-27 12:21:03 -08:00
Rafael dos Santos SilvaandRoman Rizzi 0d3e6b2726 FIX: Fix ordering of random post embeddings backfill (#965)
* FIX: Fix ordering of random post embeddings backfill

* fix annotations

---------

Co-authored-by: Roman Rizzi <[email protected]>
2024-11-27 17:01:54 -03:00
Martin Brennan be05e28679 UX: Add table style for LLM list (#963)
Followup 2f7895bb91,
I forgot to commit this minor change
2024-11-27 13:45:01 +10:00
Martin Brennan 2f7895bb91 UX: Applying more admin UI guidelines (#956)
This commit applies further admin UI guidelines, now that they have been more
fleshed out in core, to the AI admin UI:

* Tools
* LLMs
* Personas

The changes include but are not limited to:

* Applying the table CSS classes, for desktop and mobile
* Adding a description and learn more link for each tab
* Adding an empty list placeholder with CTA using `AdminConfigAreaEmptyList`
* Replacing custom headings with `AdminPageSubheader`
2024-11-27 13:34:56 +10:00
Discourse Translator Bot 7a094cde18 Update translations (#962) 2024-11-26 23:15:10 +01:00
Roman Rizzi 251628bfa1 FIX: Shutdown embeddings thread pool after processing (#961) 2024-11-26 18:12:03 -03:00
Roman Rizzi ef07fcb308 FIX: Skip records without content to classify (#960) 2024-11-26 15:54:20 -03:00
Roman Rizzi ddf2bf7034 DEV: Backfill embeddings concurrently. (#941)
We are adding a new method for generating and storing embeddings in bulk, which relies on `Concurrent::Promises::Future`. Generating an embedding consists of three steps:

Prepare text
HTTP call to retrieve the vector
Save to DB.
Each one is independently executed on whatever thread the pool gives us.

We are bringing a custom thread pool instead of the global executor since we want control over how many threads we spawn to limit concurrency. We also avoid firing thousands of HTTP requests when working with large batches.
2024-11-26 14:12:32 -03:00
Rafael dos Santos Silva 23193ee6f2 FEATURE: Calculate gists from non hot topics too (#958)
Also renames some settings to remove 'hot' references.
2024-11-26 13:44:12 -03:00
Sam 54f2d34ccb DEV: skip flakey spec (#955)
This spec fails inconsistently with:

-fragment-n14
       +You are a helpful Discourse assistant.
       +You _understand_ and **generate** Discourse Markdown.
       +You live in a Discourse Forum Message.
       +
       +You live in the forum with the URL: http://test.localhost
       +The title of your site: test site title
       +The description is: test site description
       +The participants in this conversation are: joe, jane
       +The date now is: 2024-11-25 20:23:02 UTC, much has changed since you were trained.
       +
       +You were trained on OLD data, lean on search to get up to date information about this forum
       +When searching try to SIMPLIFY search terms
       +Discourse search joins all terms with AND. Reduce and simplify terms to find more results.<guidance>
       +The following texts will give you additional guidance for your response.
       +We included them because we believe they are relevant to this conversation topic.
       +
       +Texts:
       +
       +fragment-n10
       +fragment-n9
       +fragment-n8
       +fragment-n7
       +fragment-n6
       +fragment-n5
       +fragment-n4
       +fragment-n3
       +fragment-n2
       +fragment-n1
       +</guidance>
2024-11-26 07:49:33 +11:00
Sam 616b990894 FEATURE: LLM mentions and auto silence (#949)
* FEATURE: allow mentioning an LLM mid conversation to switch

This is a edgecase feature that allow you to start a conversation
in a PM with LLM1 and then use LLM2 to evaluation or continue
the conversation

* FEATURE: allow auto silencing of spam accounts

New rule can also allow for silencing an account automatically

This can prevent spammers from creating additional posts.
2024-11-26 07:19:56 +11:00
Rafael dos Santos Silva 6c25718a7f FEATURE: Add links to filtered emotion view on emotion dashboard table (#953) 2024-11-25 15:51:01 -03:00
Roman Rizzi 79021252e9 REFACTOR: Tidy-up embedding endpoints config. (#937)
Two changes worth mentioning:

`#instance` returns a fully configured embedding endpoint ready to use.
All endpoints respond to the same method and have the same signature - `perform!(text)`

This makes it easier to reuse them when generating embeddings in bulk.
2024-11-25 13:12:43 -03:00
Rafael dos Santos Silva aeba72441d FIX: Switch up start/end dates in emotion dashboard report (#951) 2024-11-25 12:25:48 -03:00
Roman Rizzi 95762723de PERF: Preload only gists when including summaries in topic list (#948)
* PERF: Preload only gists when including summaries in topic list

* Add unique index on summaries and dedup existing records

* Make hot topics batch size setting hidden
2024-11-25 12:24:02 -03:00
Rafael dos Santos Silva 5fb1177f7b FEATURE: Refinements to Emotion in dashboard (#947)
* FEATURE: Refinements to Emotion in dashboard

- Added descriptions to individual reports
- Made reports work with data older than 60 days
2024-11-25 11:31:51 -03:00
David Taylor ea24328415 FIX: Correctly register 'info' icon (#950)
This icon is not part of the core set. Previously, d-ai was relying on other installed plugins (e.g. data-explorer) to include the icon
2024-11-25 10:20:17 -03:00
Natalie Tay f8231d259b FEATURE: Add locale detection prompt from translator (#946) 2024-11-25 08:33:54 +11:00
Roman Rizzi e54f2da1a5 FIX: Unnecessary complex preloading accidentally filters some topics. (#945)
The `topic_query_create_list_topics` modifier we append was always meant to avoid an N+1 situation when serializing gists. However, I tried to be too smart and only preload these, which resulted in some topics with *only* regular summaries getting removed from the list. This issue became apparent now we are adding gists to other lists besides hot.

Let's simplify the preloading, which still solves the N+1 issue, and let the serializer get the needed summary.
2024-11-22 12:07:27 -03:00
Rafael dos Santos Silva 1921e4e90d UX: Add emojis to the admin dash emotion table (#944) 2024-11-22 12:02:18 -03:00
Joffrey JAFFEUX 2cc8115b48 FIX: disables temporarily ai_summaries filtering (#943) 2024-11-22 08:34:54 +01:00
Sam 86cf4ccba7 FIX: automatically bust cache for share ai assets (#942)
* FIX: automatically bust cache for share ai assets

CDNs can be configured to strip query params in Discourse
hosting. This is generally safe, but in this case we had
no way of busting the cache using the path.

New design properly caches and properly breaks busts the
cache if asset changes so we don't need to worry about versions

* one day I will set up conditional lint on save :)
2024-11-22 11:23:15 +11:00
Rafael dos Santos Silva 2c78961bed FIX: Properly capture period for /filter emotion ordering (#940) 2024-11-21 16:54:30 -03:00
Rafael dos Santos Silva 8e00e036e1 FEATURE: Make emotion /filter ordering match the dashboard table (#939)
* FEATURE: Make emotion /filter ordering match the dashboard table

This change makes the /filter endpoint use the same criteria we use
in the dashboard table for emotion, so it is not confusing for users.
It means that only posts made in the period with the emotion shall be
shown in the /filter, and the order is simply a count of posts that
match the emotion in the period.

It also uses a trick to extract the filter period, and apply it to
the CTE clause that calculates post emotion count on the period, making
it a bit more efficient. Downside is that /filter filters are evaluated
from left to right, so it will only get the speed-up if the emotion
order is last. As we do this on the dashboard table, it should cover
most uses of the ordering, kicking the need for materialized views
down the road.

* Remove zero score in filter

* add table tooltip

* lint
2024-11-21 15:18:31 -03:00
Sam d56ed53eb1 FIX: cancel functionality regressed (#938)
The cancel messaging was not floating correctly to the HTTP call leading to impossible to cancel completions 

This is now fully tested as well.
2024-11-21 17:51:45 +11:00
Kris d83248cf68 UX: force long words to break in topic list gists (#936) 2024-11-20 15:49:20 -05:00
David Taylor 872e43e849 DEV: Update linting (#935) 2024-11-20 18:50:50 +01:00
David Taylor 2900d2c87d DEV: Update linting (#933) 2024-11-20 15:43:28 +01:00
Kris b68a4ca718 UX: ensure gists occupy full row (#934) 2024-11-20 09:33:49 -05:00
David Taylor 1a10680818 DEV: Replace DSection with body-class helper (#924) 2024-11-20 14:11:15 +00:00
Sam 52c644798d DEV: improve artifact presentation (#932)
1. Keep source in a "details" block after rendered so it does
not overwhelm users

2. Ensure artifacts are never indexed by robots

3. Cache break our CSS that changed recently
2024-11-20 18:53:19 +11:00
Sam 2652716398 UX: improve artifact styling add direct share link (#930)
Also remove uneeded sandboxing give this is all handled by
artifacts directly
2024-11-20 13:13:03 +11:00
Sam a0aec48606 FIX: gists are not html safe (#931)
Also allow "Everyone" in ai_hot_topic_gists_allowed_groups
2024-11-20 10:54:49 +11:00
Discourse Translator Bot f09e74c05e Update translations (#929) 2024-11-20 00:21:34 +01:00
Rafael dos Santos Silva 2c8d81827f FIX: Misc fixes for sentiment in the admin dashboard (#928)
* FIX: Misc fixes for sentiment in the admin dashboard

- Fixes missing filters for the main graph

- Fixes previous 30 days trend in emotion table

Also moves links to individual cells in emotion table, so admins can
drill down to the specific time period on their reports.

* lints
2024-11-19 19:16:21 -03:00
Kris a9afa04329 UX: update gist toggle styles (#926) 2024-11-19 15:33:34 -05:00
Roman Rizzi 530a795d43 FIX: Instruct AR that we want to use ai_summaries for filtering. (#927)
We use `includes` instead of `joins` because we want to eager-load summaries, avoiding an extra query when summarizing. However, Rails will complain unless you explicitly inform them you plan to use that inside a `WHERE` clause.
2024-11-19 17:32:13 -03:00
Roman Rizzi dcde94a393 FIX: Reduce scope of topic gists inclusion. (#925)
The topic query is used differently, and we can't assume the modifier will always receive an AR relation. Let's scope it to `Discourse#filters` instead of most lists.
2024-11-19 15:37:19 -03:00
Roman Rizzi 3c91f374ac FIX: Skip gists from PM topic lists (#923) 2024-11-19 12:51:19 -03:00
Roman Rizzi fb80d776d8 FEATURE: Enable gists on all topic lists (#922) 2024-11-19 11:04:34 -03:00
Rafael dos Santos Silva 48d08dedd4 FEATURE: Emotion activity metrics table (#916) 2024-11-19 10:01:10 -03:00
David Taylor b10be23533 FIX: Ensure artifacts are sandboxed, even when visited directly (#921)
It's important that artifacts are never given 'same origin' access to the forum domain, so that they cannot access cookies, or make authenticated HTTP requests. So even when visiting the URL directly, we need to wrap them in a sandboxed iframe.
2024-11-19 11:44:17 +00:00
David Taylor 6b9c66054c DEV: Update eslint config (#917)
* DEV: Update eslint config

* fixup

* pnpm upgrade
2024-11-19 11:57:40 +01:00
Sam 3ae1e4eaf0 FIX: properly bypass CSP for artifacts (#920)
Was meant to be bypassed but was not implemented correctly
2024-11-19 20:25:07 +11:00
Sam 755b63f31f FEATURE: Add support for Mistral models (#919)
Adds support for mistral models (pixtral and mistral large now have presets)

Also corrects token accounting in AWS bedrock models
2024-11-19 17:28:09 +11:00
Sam 0d7f353284 FEATURE: AI artifacts (#898)
This is a significant PR that introduces AI Artifacts functionality to the discourse-ai plugin along with several other improvements. Here are the key changes:

1. AI Artifacts System:
   - Adds a new `AiArtifact` model and database migration
   - Allows creation of web artifacts with HTML, CSS, and JavaScript content
   - Introduces security settings (`strict`, `lax`, `disabled`) for controlling artifact execution
   - Implements artifact rendering in iframes with sandbox protection
   - New `CreateArtifact` tool for AI to generate interactive content

2. Tool System Improvements:
   - Adds support for partial tool calls, allowing incremental updates during generation
   - Better handling of tool call states and progress tracking
   - Improved XML tool processing with CDATA support
   - Fixes for tool parameter handling and duplicate invocations

3. LLM Provider Updates:
   - Updates for Anthropic Claude models with correct token limits
   - Adds support for native/XML tool modes in Gemini integration
   - Adds new model configurations including Llama 3.1 models
   - Improvements to streaming response handling

4. UI Enhancements:
   - New artifact viewer component with expand/collapse functionality
   - Security controls for artifact execution (click-to-run in strict mode)
   - Improved dialog and response handling
   - Better error management for tool execution

5. Security Improvements:
   - Sandbox controls for artifact execution
   - Public/private artifact sharing controls
   - Security settings to control artifact behavior
   - CSP and frame-options handling for artifacts

6. Technical Improvements:
   - Better post streaming implementation
   - Improved error handling in completions
   - Better memory management for partial tool calls
   - Enhanced testing coverage

7. Configuration:
   - New site settings for artifact security
   - Extended LLM model configurations
   - Additional tool configuration options

This PR significantly enhances the plugin's capabilities for generating and displaying interactive content while maintaining security and providing flexible configuration options for administrators.
2024-11-19 09:22:39 +11:00
464 changed files with 20313 additions and 6460 deletions
+1
View File
@@ -1,3 +1,4 @@
< 3.4.0.beta4-dev: 20612fde52d3f740cad64823ef8aadb0748b567f
< 3.4.0.beta3-dev: decf1bb49d737ea15308400f22f89d1d1e71d13d
< 3.4.0.beta1-dev: 9d887ad4ace8e33c3fe7dbb39237e882c08b4f0b
< 3.3.0.beta5-dev: 4d8090002f6dcd8e34d41033606bf131fa221475
-1
View File
@@ -1 +0,0 @@
module.exports = require("@discourse/lint-configs/eslint");
@@ -1,19 +1,19 @@
import DiscourseRoute from "discourse/routes/discourse";
export default DiscourseRoute.extend({
export default class AdminPluginsShowDiscourseAiLlmsEdit extends DiscourseRoute {
async model(params) {
const allLlms = this.modelFor("adminPlugins.show.discourse-ai-llms");
const id = parseInt(params.id, 10);
const record = allLlms.findBy("id", id);
record.provider_params = record.provider_params || {};
return record;
},
}
setupController(controller, model) {
this._super(controller, model);
super.setupController(controller, model);
controller.set(
"allLlms",
this.modelFor("adminPlugins.show.discourse-ai-llms")
);
},
});
}
}
@@ -1,18 +1,18 @@
import DiscourseRoute from "discourse/routes/discourse";
export default DiscourseRoute.extend({
queryParams: {
export default class AdminPluginsShowDiscourseAiLlmsNew extends DiscourseRoute {
queryParams = {
llmTemplate: { refreshModel: true },
},
};
async model() {
const record = this.store.createRecord("ai-llm");
record.provider_params = {};
return record;
},
}
setupController(controller, model) {
this._super(controller, model);
super.setupController(controller, model);
controller.set(
"allLlms",
this.modelFor("adminPlugins.show.discourse-ai-llms")
@@ -21,5 +21,5 @@ export default DiscourseRoute.extend({
"llmTemplate",
this.paramsFor(this.routeName).llmTemplate || null
);
},
});
}
}
@@ -1,19 +1,19 @@
import DiscourseRoute from "discourse/routes/discourse";
export default DiscourseRoute.extend({
export default class AdminPluginsShowDiscourseAiPersonasEdit extends DiscourseRoute {
async model(params) {
const allPersonas = this.modelFor(
"adminPlugins.show.discourse-ai-personas"
);
const id = parseInt(params.id, 10);
return allPersonas.findBy("id", id);
},
}
setupController(controller, model) {
this._super(controller, model);
super.setupController(controller, model);
controller.set(
"allPersonas",
this.modelFor("adminPlugins.show.discourse-ai-personas")
);
},
});
}
}
@@ -1,7 +1,7 @@
import { AUTO_GROUPS } from "discourse/lib/constants";
import DiscourseRoute from "discourse/routes/discourse";
export default DiscourseRoute.extend({
export default class AdminPluginsShowDiscourseAiPersonasNew extends DiscourseRoute {
async model() {
const record = this.store.createRecord("ai-persona");
record.set("allowed_group_ids", [AUTO_GROUPS.trust_level_0.id]);
@@ -13,13 +13,13 @@ export default DiscourseRoute.extend({
record.set("allow_personal_messages", true);
record.set("tool_details", false);
return record;
},
}
setupController(controller, model) {
this._super(controller, model);
super.setupController(controller, model);
controller.set(
"allPersonas",
this.modelFor("adminPlugins.show.discourse-ai-personas")
);
},
});
}
}
@@ -0,0 +1,11 @@
import { service } from "@ember/service";
import { ajax } from "discourse/lib/ajax";
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiSpamRoute extends DiscourseRoute {
@service store;
model() {
return ajax("/admin/plugins/discourse-ai/ai-spam.json");
}
}
@@ -1,6 +1,6 @@
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiToolsShowRoute extends DiscourseRoute {
export default class DiscourseAiToolsEditRoute extends DiscourseRoute {
async model(params) {
const allTools = this.modelFor("adminPlugins.show.discourse-ai-tools");
const id = parseInt(params.id, 10);
@@ -0,0 +1,11 @@
import { service } from "@ember/service";
import { ajax } from "discourse/lib/ajax";
import DiscourseRoute from "discourse/routes/discourse";
export default class DiscourseAiUsageRoute extends DiscourseRoute {
@service store;
model() {
return ajax("/admin/plugins/discourse-ai/ai-usage.json");
}
}
@@ -0,0 +1 @@
<AiSpam @model={{this.model}} />
@@ -0,0 +1,7 @@
<section class="ai-persona-tool-editor__current admin-detail pull-left">
<AiToolEditor
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
/>
</section>
@@ -1,5 +1,7 @@
<AiToolEditor
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
/>
<section class="ai-persona-tool-editor__current admin-detail pull-left">
<AiToolEditor
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
/>
</section>
@@ -1,5 +0,0 @@
<AiToolEditor
@tools={{this.allTools}}
@model={{this.model}}
@presets={{this.presets}}
/>
@@ -0,0 +1 @@
<AiUsage @model={{this.model}} />
@@ -0,0 +1,59 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiLlmQuotasController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
def index
quotas = LlmQuota.includes(:group)
render json: {
quotas:
ActiveModel::ArraySerializer.new(quotas, each_serializer: LlmQuotaSerializer),
}
end
def create
quota = LlmQuota.new(quota_params)
if quota.save
render json: LlmQuotaSerializer.new(quota), status: :created
else
render_json_error quota
end
end
def update
quota = LlmQuota.find(params[:id])
if quota.update(quota_params)
render json: LlmQuotaSerializer.new(quota)
else
render_json_error quota
end
end
def destroy
quota = LlmQuota.find(params[:id])
quota.destroy!
head :no_content
rescue ActiveRecord::RecordNotFound
render json: { error: I18n.t("not_found") }, status: 404
end
private
def quota_params
params.require(:quota).permit(
:group_id,
:llm_model_id,
:max_tokens,
:max_usages,
:duration_seconds,
)
end
end
end
end
@@ -6,7 +6,7 @@ module DiscourseAi
requires_plugin ::DiscourseAi::PLUGIN_NAME
def index
llms = LlmModel.all.order(:display_name)
llms = LlmModel.all.includes(:llm_quotas).order(:display_name)
render json: {
ai_llms:
@@ -30,13 +30,21 @@ module DiscourseAi
}
end
def show
def new
end
def edit
llm_model = LlmModel.find(params[:id])
render json: LlmModelSerializer.new(llm_model)
end
def create
llm_model = LlmModel.new(ai_llm_params)
# we could do nested attributes but the mechanics are not ideal leading
# to lots of complex debugging, this is simpler
quota_params.each { |quota| llm_model.llm_quotas.build(quota) } if quota_params
if llm_model.save
llm_model.toggle_companion_user
render json: LlmModelSerializer.new(llm_model), status: :created
@@ -48,6 +56,25 @@ module DiscourseAi
def update
llm_model = LlmModel.find(params[:id])
if params[:ai_llm].key?(:llm_quotas)
if quota_params
existing_quota_group_ids = llm_model.llm_quotas.pluck(:group_id)
new_quota_group_ids = quota_params.map { |q| q[:group_id] }
llm_model
.llm_quotas
.where(group_id: existing_quota_group_ids - new_quota_group_ids)
.destroy_all
quota_params.each do |quota_param|
quota = llm_model.llm_quotas.find_or_initialize_by(group_id: quota_param[:group_id])
quota.update!(quota_param)
end
else
llm_model.llm_quotas.destroy_all
end
end
if llm_model.seeded?
return render_json_error(I18n.t("discourse_ai.llm.cannot_edit_builtin"), status: 403)
end
@@ -107,6 +134,19 @@ module DiscourseAi
private
def quota_params
if params[:ai_llm][:llm_quotas].present?
params[:ai_llm][:llm_quotas].map do |quota|
mapped = {}
mapped[:group_id] = quota[:group_id].to_i
mapped[:max_tokens] = quota[:max_tokens].to_i if quota[:max_tokens].present?
mapped[:max_usages] = quota[:max_usages].to_i if quota[:max_usages].present?
mapped[:duration_seconds] = quota[:duration_seconds].to_i
mapped
end
end
end
def ai_llm_params(updating: nil)
return {} if params[:ai_llm].blank?
@@ -5,7 +5,7 @@ module DiscourseAi
class AiPersonasController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
before_action :find_ai_persona, only: %i[show update destroy create_user]
before_action :find_ai_persona, only: %i[edit update destroy create_user]
def index
ai_personas =
@@ -33,7 +33,10 @@ module DiscourseAi
render json: { ai_personas: ai_personas, meta: { tools: tools, llms: llms } }
end
def show
def new
end
def edit
render json: LocalizedAiPersonaSerializer.new(@ai_persona)
end
@@ -0,0 +1,137 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiSpamController < ::Admin::AdminController
requires_plugin "discourse-ai"
def show
render json: AiSpamSerializer.new(spam_config, root: false)
end
def update
updated_params = {}
if allowed_params.key?(:llm_model_id)
llm_model_id = updated_params[:llm_model_id] = allowed_params[:llm_model_id]
if llm_model_id.to_i < 0 &&
!SiteSetting.ai_spam_detection_model_allowed_seeded_models_map.include?(
llm_model_id.to_s,
)
return(
render_json_error(
I18n.t("discourse_ai.llm.configuration.invalid_seeded_model"),
status: 422,
)
)
end
end
updated_params[:data] = {
custom_instructions: allowed_params[:custom_instructions],
} if allowed_params.key?(:custom_instructions)
if updated_params.present?
# not using upsert cause we will not get the correct validation errors
if AiModerationSetting.spam
AiModerationSetting.spam.update!(updated_params)
else
AiModerationSetting.create!(updated_params.merge(setting_type: :spam))
end
end
is_enabled = ActiveModel::Type::Boolean.new.cast(allowed_params[:is_enabled])
if allowed_params.key?(:is_enabled)
if is_enabled && !AiModerationSetting.spam&.llm_model_id
return(
render_json_error(
I18n.t("discourse_ai.llm.configuration.must_select_model"),
status: 422,
)
)
end
SiteSetting.ai_spam_detection_enabled = is_enabled
end
render json: AiSpamSerializer.new(spam_config, root: false)
end
def test
url = params[:post_url].to_s
post = nil
if url.match?(/^\d+$/)
post_id = url.to_i
post = Post.find_by(id: post_id)
end
route = UrlHelper.rails_route_from_url(url) if !post
if route
if route[:controller] == "topics"
post_number = route[:post_number] || 1
post = Post.with_deleted.find_by(post_number: post_number, topic_id: route[:topic_id])
end
end
raise Discourse::NotFound if !post
result =
DiscourseAi::AiModeration::SpamScanner.test_post(
post,
custom_instructions: params[:custom_instructions],
llm_id: params[:llm_id],
)
render json: result
end
def fix_errors
case params[:error]
when "spam_scanner_not_admin"
begin
DiscourseAi::AiModeration::SpamScanner.fix_spam_scanner_not_admin
render json: success_json
rescue ActiveRecord::RecordInvalid
render_json_error(
I18n.t("discourse_ai.spam_detection.bot_user_update_failed"),
status: :unprocessable_entity,
)
rescue StandardError
render_json_error(
I18n.t("discourse_ai.spam_detection.unexpected"),
status: :internal_server_error,
)
end
else
render_json_error(
I18n.t("discourse_ai.spam_detection.invalid_error_type"),
status: :bad_request,
)
end
end
private
def allowed_params
params.permit(:is_enabled, :llm_model_id, :custom_instructions)
end
def spam_config
spam_config = {
enabled: SiteSetting.ai_spam_detection_enabled,
settings: AiModerationSetting.spam,
}
spam_config[:stats] = DiscourseAi::AiModeration::SpamReport.generate(min_date: 1.week.ago)
if spam_config[:stats].scanned_count > 0
spam_config[
:flagging_username
] = DiscourseAi::AiModeration::SpamScanner.flagging_user&.username
end
spam_config
end
end
end
end
@@ -5,14 +5,17 @@ module DiscourseAi
class AiToolsController < ::Admin::AdminController
requires_plugin ::DiscourseAi::PLUGIN_NAME
before_action :find_ai_tool, only: %i[test show update destroy]
before_action :find_ai_tool, only: %i[test edit update destroy]
def index
ai_tools = AiTool.all
render_serialized({ ai_tools: ai_tools }, AiCustomToolListSerializer, root: false)
end
def show
def new
end
def edit
render_serialized(@ai_tool, AiCustomToolSerializer)
end
@@ -0,0 +1,30 @@
# frozen_string_literal: true
module DiscourseAi
module Admin
class AiUsageController < ::Admin::AdminController
requires_plugin "discourse-ai"
def show
end
def report
render json: AiUsageSerializer.new(create_report, root: false)
end
private
def create_report
report =
DiscourseAi::Completions::Report.new(
start_date: params[:start_date]&.to_date || 30.days.ago,
end_date: params[:end_date]&.to_date || Time.current,
)
report = report.filter_by_feature(params[:feature]) if params[:feature].present?
report = report.filter_by_model(params[:model]) if params[:model].present?
report
end
end
end
end
@@ -0,0 +1,101 @@
# frozen_string_literal: true
module DiscourseAi
module AiBot
class ArtifactsController < ApplicationController
requires_plugin DiscourseAi::PLUGIN_NAME
before_action :require_site_settings!
skip_before_action :preload_json, :check_xhr, only: %i[show]
def show
artifact = AiArtifact.find(params[:id])
post = Post.find_by(id: artifact.post_id)
if artifact.metadata&.dig("public")
# no guardian needed
else
raise Discourse::NotFound if !post&.topic&.private_message?
raise Discourse::NotFound if !guardian.can_see?(post)
end
name = artifact.name
if params[:version].present?
artifact = artifact.versions.find_by(version_number: params[:version])
raise Discourse::NotFound if !artifact
end
js = artifact.js || ""
if !js.match?(%r{\A\s*<script.*</script>}mi)
mod = ""
mod = " type=\"module\"" if js.match?(/\A\s*import.*/)
js = "<script#{mod}>\n#{js}\n</script>"
end
# Prepare the inner (untrusted) HTML document
untrusted_html = <<~HTML
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>#{ERB::Util.html_escape(name)}</title>
<style>
#{artifact.css}
</style>
</head>
<body>
#{artifact.html}
#{js}
</body>
</html>
HTML
# Prepare the outer (trusted) HTML document
trusted_html = <<~HTML
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>#{ERB::Util.html_escape(name)}</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover, interactive-widget=resizes-content">
<style>
html, body, iframe {
margin: 0;
padding: 0;
width: 100%;
height: 100%;
border: 0;
overflow: hidden;
}
iframe {
overflow: auto;
}
</style>
</head>
<body>
<iframe sandbox="allow-scripts allow-forms" height="100%" width="100%" srcdoc="#{ERB::Util.html_escape(untrusted_html)}" frameborder="0"></iframe>
</body>
</html>
HTML
response.headers.delete("X-Frame-Options")
response.headers[
"Content-Security-Policy"
] = "script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval' https://unpkg.com https://cdnjs.cloudflare.com https://ajax.googleapis.com https://cdn.jsdelivr.net;"
response.headers["X-Robots-Tag"] = "noindex"
# Render the content
render html: trusted_html.html_safe, layout: false, content_type: "text/html"
end
private
def require_site_settings!
if !SiteSetting.discourse_ai_enabled ||
!SiteSetting.ai_artifact_security.in?(%w[lax strict])
raise Discourse::NotFound
end
end
end
end
end
@@ -7,7 +7,8 @@ module DiscourseAi
requires_login only: %i[create update destroy]
before_action :require_site_settings!
skip_before_action :preload_json, :check_xhr, only: %i[show]
skip_before_action :preload_json, :check_xhr, only: %i[show asset]
skip_before_action :verify_authenticity_token, only: ["asset"]
def create
ensure_allowed_create!
@@ -50,6 +51,30 @@ module DiscourseAi
end
end
def asset
no_cookies
name = params[:name]
path, content_type =
if name == "share"
%w[share.css text/css]
elsif name == "highlight"
%w[highlight.min.js application/javascript]
else
raise Discourse::NotFound
end
content = File.read(DiscourseAi.public_asset_path("ai-share/#{path}"))
# note, path contains a ":version" which automatically busts the cache
# based on file content, so this is safe
response.headers["Last-Modified"] = 10.years.ago.httpdate
response.headers["Content-Length"] = content.bytesize.to_s
immutable_for 1.year
render plain: content, disposition: :nil, content_type: content_type
end
def preview
ensure_allowed_preview!
data = SharedAiConversation.build_conversation_data(@topic, include_usernames: true)
@@ -35,7 +35,7 @@ module DiscourseAi
prompt.custom_instruction = params[:custom_prompt]
end
suggest_thumbnails(input) if prompt.id == CompletionPrompt::ILLUSTRATE_POST
return suggest_thumbnails(input) if prompt.id == CompletionPrompt::ILLUSTRATE_POST
hijack do
render json:
@@ -2,13 +2,29 @@
module DiscourseAi
module AiBot
module SharedAiConversationsHelper
# bump up version when assets change
# long term we may want to change this cause it is hard to remember
# to bump versions, but for now this does the job
VERSION = "1"
# keeping it here for caching
def self.share_asset_url(asset_name)
if !%w[share.css highlight.js].include?(asset_name)
raise StandardError, "unknown asset type #{asset_name}"
end
def share_asset_url(short_path)
::UrlHelper.absolute("/plugins/discourse-ai/ai-share/#{short_path}?#{VERSION}")
@urls ||= {}
url = @urls[asset_name]
return url if url
path = asset_name
path = "highlight.min.js" if asset_name == "highlight.js"
content = File.read(DiscourseAi.public_asset_path("ai-share/#{path}"))
sha1 = Digest::SHA1.hexdigest(content)
url = "/discourse-ai/ai-bot/shared-ai-conversations/asset/#{sha1}/#{asset_name}"
@urls[asset_name] = GlobalPath.cdn_path(url)
end
def share_asset_url(asset_name)
DiscourseAi::AiBot::SharedAiConversationsHelper.share_asset_url(asset_name)
end
end
end
+13
View File
@@ -0,0 +1,13 @@
# frozen_string_literal: true
module Jobs
class AiSpamScan < ::Jobs::Base
def execute(args)
return if !args[:post_id]
post = Post.find_by(id: args[:post_id])
return if !post
DiscourseAi::AiModeration::SpamScanner.perform_scan(post)
end
end
end
+1 -3
View File
@@ -18,9 +18,7 @@ module ::Jobs
target = target_type.constantize.find_by(id: target_id)
return if !target
truncation = DiscourseAi::Embeddings::Strategies::Truncation.new
vector_rep =
DiscourseAi::Embeddings::VectorRepresentations::Base.current_representation(truncation)
vector_rep = DiscourseAi::Embeddings::VectorRepresentations::Base.current_representation
tokenizer = vector_rep.tokenizer
chunk_tokens = target.rag_chunk_tokens
-17
View File
@@ -1,17 +0,0 @@
# frozen_string_literal: true
module Jobs
class EvaluatePostUploads < ::Jobs::Base
def execute(args)
return unless SiteSetting.ai_nsfw_detection_enabled
return if (post_id = args[:post_id]).blank?
post = Post.includes(:uploads).find_by_id(post_id)
return if post.nil? || post.uploads.empty?
return if post.uploads.none? { |u| FileHelper.is_supported_image?(u.url) }
DiscourseAi::PostClassificator.new(DiscourseAi::Nsfw::Classification.new).classify!(post)
end
end
end
@@ -1,23 +1,20 @@
# frozen_string_literal: true
module ::Jobs
class UpdateHotTopicGist < ::Jobs::Base
class FastTrackTopicGist < ::Jobs::Base
sidekiq_options retry: false
def execute(args)
return if !SiteSetting.discourse_ai_enabled
return if !SiteSetting.ai_summarization_enabled
return if SiteSetting.ai_summarize_max_hot_topics_gists_per_batch.zero?
return if !SiteSetting.ai_summary_gists_enabled
topic = Topic.find_by(id: args[:topic_id])
return if topic.blank?
return if !TopicHotScore.where(topic: topic).exists?
summarizer = DiscourseAi::Summarization.topic_gist(topic)
gist = summarizer.existing_summary
return if gist.blank?
return if !gist.outdated
return if gist.present? && (!gist.outdated || gist.created_at >= 5.minutes.ago)
summarizer.force_summarize(Discourse.system_user)
end

Some files were not shown because too many files have changed in this diff Show More