Skip to main content

pd_cipher::engine::async_adaptive

Async adaptive truncation engine for optimizing token lengths. Async adaptive truncation engine implementation.

This module implements the two-pass algorithm for finding optimal token truncation lengths that minimize size while maintaining uniqueness.

Structs​

AsyncAdaptiveTruncationEngine​

Async engine for performing adaptive token truncation.

The AsyncAdaptiveTruncationEngine implements a two-pass algorithm:

  1. Pass 1: Analyze the dataset to find optimal truncation lengths
  2. Pass 2: Generate truncated tokens using the analysis results
pub struct AsyncAdaptiveTruncationEngine<...>