LAO: Building a Rust-Based Local AI Orchestrator

Cloud AI is powerful, but it comes with trade-offs: your data leaves your control, you pay per token, and you're at the mercy of API rate limits and downtime. For security research, private data processing, or any workflow involving sensitive information, sending data to external APIs isn't acceptable.

I built LAO (Local AI Orchestrator) to solve this: a high-performance Rust orchestrator for local LLM workflows that processes sensitive inference tasks entirely on-device.

Design Goals

LAO was built around three principles:

Data never leaves the machine: All inference happens locally via Ollama, Whisper.cpp, and other local model runtimes
Memory safety without compromise: The plugin system and workflow engine are built in Rust to prevent the memory bugs that plague C/C++ AI tooling
Extensibility without vulnerability: Users can extend LAO with plugins and custom workflows without exposing the host system to external API vulnerabilities

Architecture

Secure Workflow Engine

LAO's core is a workflow engine that chains local AI models into pipelines. Each workflow is a directed acyclic graph (DAG) of processing steps:

struct Workflow {
    name: String,
    steps: Vec<WorkflowStep>,
    connections: Vec<(StepId, StepId)>,
}

struct WorkflowStep {
    id: StepId,
    model: ModelBackend,
    prompt_template: String,
    input_transforms: Vec<Transform>,
}

enum ModelBackend {
    Ollama { model: String, endpoint: String },
    WhisperCpp { model_path: PathBuf },
    Custom(Box<dyn LocalModel>),
}

The key security property: ModelBackend only supports local endpoints. There's no HttpApi variant — by design, LAO workflows can't accidentally send data to external services.

Memory-Safe Plugin System

LAO supports plugins for extending functionality — custom model backends, input preprocessors, output formatters. The plugin system uses Rust's trait system to enforce safety at compile time:

trait Plugin: Send + Sync {
    fn name(&self) -> &str;
    fn process(&self, input: &PluginInput) -> Result<PluginOutput>;
}

struct PluginManager {
    plugins: Vec<Box<dyn Plugin>>,
}

The Send + Sync bounds ensure plugins are safe to use across threads. Rust's ownership model means plugins can't accidentally share mutable state or create data races. And because plugins are loaded as Rust code (not dynamic libraries from the network), there's no risk of code injection.

CLI and Visual Workflows

LAO supports both CLI-driven and visual workflow creation:

CLI mode for automation and scripting:

lao run workflow.yaml --input "Summarize this document" --model llama3
lao chain "transcribe audio.wav | summarize | extract-entities"

Visual mode using egui for interactive workflow building — drag-and-drop nodes, connect models, and preview results in real time.

Both interfaces use the same underlying workflow engine, ensuring consistent behavior regardless of how workflows are created.

Local Model Integration

LAO integrates with multiple local AI runtimes:

Ollama: LLM inference (Llama, Mistral, CodeLlama, etc.)
Whisper.cpp: Audio transcription entirely on-device
Custom models: Any local model that implements the LocalModel trait

Each integration runs inference in a separate process with controlled I/O, so a misbehaving model can't affect the orchestrator's stability.

Why Rust for AI Tooling?

Most AI orchestration tools are written in Python. That's fine for prototyping, but for a tool that handles sensitive data and runs continuously:

Memory safety: No buffer overflows, use-after-free, or data races in the orchestrator itself
Performance: Workflow scheduling, I/O management, and result processing happen with zero GC pauses
Reliability: Rust's error handling (Result<T, E>) forces explicit handling of every failure mode
Type safety: The plugin system catches integration errors at compile time, not runtime

LAO is designed to be trusted infrastructure — the kind of tool you run on machines with sensitive data and forget about. Rust's guarantees make that possible.