Natalie Tay 5d80a34589 FIX: Add a max token limit based on the text to be translated (#1507) ...

We're seeing that some LLMs are using 65000+ tokens for raw text that is only 10-1000 characters long.

This PR adds a max_token to be passed to the LLM API for each translation based on the length of the text.

2025-07-17 17:47:15 +08:00

..

ai_bot

FIX: A typo in bot filtration in ai-bot-header-icon (#1455)

2025-06-24 10:51:07 -03:00

ai_helper

FIX: title suggestions should return 5 unique titles (#1491)

2025-07-08 06:30:09 -07:00

ai_moderation

FEATURE: Use Persona's when scanning posts for spam (#1465)

2025-06-27 10:35:47 -03:00

automation

FIX: avoid using structured outputs for report runs (#1502)

2025-07-15 13:14:52 +10:00

completions

FIX: make AI helper more robust (#1484)

2025-07-04 14:47:11 +10:00

configuration

FIX: make AI helper more robust (#1484)

2025-07-04 14:47:11 +10:00

database

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

discord/bot

FEATURE: add inferred concepts system (#1330)

2025-06-02 14:29:20 -03:00

embeddings

FIX: Process succesfully generated embeddings even if some failed (#1500)

2025-07-10 17:51:01 -03:00

inference

FIX: Process succesfully generated embeddings even if some failed (#1500)

2025-07-10 17:51:01 -03:00

inferred_concepts

FIX: Proper default LLM detection for inferred concepts (#1392)

2025-06-02 17:56:47 -03:00

personas

DEV: Switch translations out of structured output as it returns only a single value (#1503)

2025-07-16 13:38:00 +08:00

sentiment

DEV: Move tokenizers to a gem (#1481)

2025-07-02 14:43:03 -03:00

summarization

DEV: Move tokenizers to a gem (#1481)

2025-07-02 14:43:03 -03:00

tasks

DEV: rake task to populate sentiment in dev env (#1176)

2025-03-10 14:21:00 -03:00

translation

FIX: Add a max token limit based on the text to be translated (#1507)

2025-07-17 17:47:15 +08:00

utils

FIX: make AI helper more robust (#1484)

2025-07-04 14:47:11 +10:00

automation.rb

FEATURE: Use Personas in Automation's llm_report script (#1499)

2025-07-14 12:47:21 -03:00

embeddings.rb

DEV: Robust check for embeddings enabled (#1116)

2025-02-06 12:18:55 -03:00

engine.rb

DEV: port directory structure to Zeitwerk (#319)

2023-11-29 15:17:46 +11:00

guardian_extensions.rb

FIX: Restore gists previous group access behavior. (#1247)

2025-04-07 12:04:30 -03:00

multisite_hash.rb

FIX: properly cache user locale (#593)

2024-04-26 09:28:35 -03:00

post_extensions.rb

FEATURE: add inferred concepts system (#1330)

2025-06-02 14:29:20 -03:00

summarization.rb

FEATURE: Personas powered summaries. (#1232)

2025-04-02 12:54:47 -03:00

topic_extensions.rb

FEATURE: add inferred concepts system (#1330)

2025-06-02 14:29:20 -03:00

translation.rb

DEV: Normalize locales that are similar (e.g. en and en_GB) so they do not get translated (#1495)

2025-07-09 22:21:51 +08:00