Skip to main content

pd_cipher::pipeline

Token processing pipeline components for data preparation. Modular pipeline system for token processing.

This module provides a flexible, extensible component system for processing tokens through configurable pipelines. Components can be chained together to perform normalization, validation, transformation, and other operations.

Structs​

Pipeline​

A configurable pipeline for processing tokens through multiple components.

The pipeline executes components in sequence, passing the output of each component as input to the next. This allows for complex token processing workflows with modular, reusable components.

Examples

Basic Pipeline Usage​

use pd_cipher::pipeline::{Pipeline, Token};
use pd_cipher::pipeline::components::{NormalizationComponent, ValidationComponent};

# fn example() -> pd_cipher::Result<()> {
let mut pipeline = Pipeline::new();

// Add components in processing order
pipeline.add_component(Box::new(NormalizationComponent::new()));
pipeline.add_component(Box::new(ValidationComponent::new()));

let tokens = vec![
    "Hello World".as_bytes().to_vec(),
    "  Test Token  ".as_bytes().to_vec(),
];

let processed = pipeline.process(&tokens)?;
println!("Processed {} tokens", processed.len());
# Ok(())
# }

Pipeline with Custom Components​

use pd_cipher::pipeline::{Pipeline, Component, Token};
use pd_cipher::error::Result;

#[derive(Debug)]
struct PrefixComponent {
    prefix: String,
}

impl PrefixComponent {
    fn new(prefix: &str) -> Self {
        Self { prefix: prefix.to_string() }
    }
}

impl Component for PrefixComponent {
    fn name(&self) -> &str { "prefix" }

    fn process(&self, tokens: &[Token]) -> Result<Vec<Token>> {
        let processed = tokens.iter()
            .map(|token| {
                let mut prefixed = self.prefix.as_bytes().to_vec();
                prefixed.extend_from_slice(token);
                prefixed
            })
            .collect();
        Ok(processed)
    }
}

# fn example() -> pd_cipher::Result<()> {
let mut pipeline = Pipeline::new();
pipeline.add_component(Box::new(PrefixComponent::new("TOKEN:")));

let tokens = vec!["data".as_bytes().to_vec()];
let processed = pipeline.process(&tokens)?;
assert_eq!(processed[0], b"TOKEN:data");
# Ok(())
# }
pub struct Pipeline

Traits​

Component​

A component that processes tokens in a pipeline.

Components are the building blocks of the processing pipeline. Each component receives a collection of tokens, processes them according to its logic, and returns the processed tokens.

Design Principles

  • Stateless: Components should not maintain state between calls
  • Composable: Components can be chained together in any order
  • Idempotent: Processing the same input multiple times yields the same result
  • Error Handling: Components should validate input and handle errors gracefully

Examples

Basic Component Implementation​

use pd_cipher::pipeline::{Component, Token};
use pd_cipher::error::Result;

#[derive(Debug)]
struct UppercaseComponent;

impl Component for UppercaseComponent {
    fn name(&self) -> &str {
        "uppercase"
    }

    fn process(&self, tokens: &[Token]) -> Result<Vec<Token>> {
        let processed = tokens.iter()
            .map(|token| {
                String::from_utf8_lossy(token)
                    .to_uppercase()
                    .into_bytes()
            })
            .collect();
        Ok(processed)
    }
}

Using Components in a Pipeline​

use pd_cipher::pipeline::{Pipeline, Token};
use pd_cipher::pipeline::components::{NormalizationComponent, ValidationComponent};

# fn example() -> pd_cipher::Result<()> {
let mut pipeline = Pipeline::new();
pipeline.add_component(Box::new(NormalizationComponent::new()));
pipeline.add_component(Box::new(ValidationComponent::new()));

let input_tokens = vec![
    "Hello World".as_bytes().to_vec(),
    "Test Token".as_bytes().to_vec(),
];

let processed = pipeline.process(&input_tokens)?;
# Ok(())
# }
pub trait Component

Type Aliases​

Token​

Represents a token in the processing pipeline.

Tokens can be text, binary data, or structured data represented as bytes.

pub type Token = ...