Roman Rizzi
9a79afcdbf
* DEV: Better strategies for summarization The strategy responsibility needs to be "Given a collection of texts, I know how to summarize them most efficiently, using the minimum amount of requests and maximizing token usage". There are different token limits for each model, so it all boils down to two different strategies: Fold all these texts into a single one, doing the summarization in chunks, and then build a summary from those. Build it by combining texts in a single prompt, and truncate it according to your token limits. While the latter is less than ideal, we need it for "bart-large-cnn-samsum" and "flan-t5-base-samsum", both with low limits. The rest will rely on folding. * Expose summarized chunks to users
77 lines
2.0 KiB
Ruby
77 lines
2.0 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
module DiscourseAi
|
|
module Tokenizer
|
|
class BasicTokenizer
|
|
class << self
|
|
def tokenizer
|
|
raise NotImplementedError
|
|
end
|
|
|
|
def tokenize(text)
|
|
tokenizer.encode(text).tokens
|
|
end
|
|
|
|
def size(text)
|
|
tokenize(text).size
|
|
end
|
|
|
|
def truncate(text, max_length)
|
|
# Fast track the common case where the text is already short enough.
|
|
return text if text.size < max_length
|
|
|
|
tokenizer.decode(tokenizer.encode(text).ids.take(max_length))
|
|
end
|
|
|
|
def can_expand_tokens?(text, addition, max_length)
|
|
return true if text.size + addition.size < max_length
|
|
|
|
tokenizer.encode(text).ids.length + tokenizer.encode(addition).ids.length < max_length
|
|
end
|
|
end
|
|
end
|
|
|
|
class BertTokenizer < BasicTokenizer
|
|
def self.tokenizer
|
|
@@tokenizer ||=
|
|
Tokenizers.from_file("./plugins/discourse-ai/tokenizers/bert-base-uncased.json")
|
|
end
|
|
end
|
|
|
|
class AnthropicTokenizer < BasicTokenizer
|
|
def self.tokenizer
|
|
@@tokenizer ||=
|
|
Tokenizers.from_file("./plugins/discourse-ai/tokenizers/claude-v1-tokenization.json")
|
|
end
|
|
end
|
|
|
|
class OpenAiTokenizer < BasicTokenizer
|
|
class << self
|
|
def tokenizer
|
|
@@tokenizer ||= Tiktoken.get_encoding("cl100k_base")
|
|
end
|
|
|
|
def tokenize(text)
|
|
tokenizer.encode(text)
|
|
end
|
|
|
|
def truncate(text, max_length)
|
|
# Fast track the common case where the text is already short enough.
|
|
return text if text.size < max_length
|
|
|
|
tokenizer.decode(tokenize(text).take(max_length))
|
|
rescue Tiktoken::UnicodeError
|
|
max_length = max_length - 1
|
|
retry
|
|
end
|
|
|
|
def can_expand_tokens?(text, addition, max_length)
|
|
return true if text.size + addition.size < max_length
|
|
|
|
tokenizer.encode(text).length + tokenizer.encode(addition).length < max_length
|
|
end
|
|
end
|
|
end
|
|
end
|
|
end
|