owlen/PROVIDER_MANAGER_OPTIMIZATIONS.md

ProviderManager Clone Overhead Optimizations

Summary

This document describes the optimizations applied to /home/cnachtigall/data/git/projects/Owlibou/owlen/crates/owlen-core/src/provider/manager.rs to reduce clone overhead as identified in the project analysis report.

Problems Identified

1. Lines 94-100 (list_all_models): Clones all provider Arc handles and IDs unnecessarily into an intermediate Vec
2. Lines 162-168 (refresh_health): Collects into Vec with unnecessary clones before spawning async tasks
3. Line 220 (provider_statuses()): Clones entire HashMap on every call

The report estimated that 15-20% of list_all_models time was spent on String clones alone.

Optimizations Applied

1. Change status_cache to Arc-Wrapped HashMap

File: crates/owlen-core/src/provider/manager.rs

Line 28: Change struct definition

// Before:
status_cache: RwLock<HashMap<String, ProviderStatus>>,

// After:
status_cache: RwLock<Arc<HashMap<String, ProviderStatus>>>,

Rationale: Using Arc<HashMap> allows cheap cloning via reference counting instead of deep-copying the entire HashMap.

2. Update Constructor (new)

Lines 41-44:

// Before:
Self {
    providers: RwLock::new(HashMap::new()),
    status_cache: RwLock::new(status_cache),
}

// After:
Self {
    providers: RwLock::new(HashMap::new()),
    status_cache: RwLock::new(Arc::new(status_cache)),
}

3. Update Default Implementation

Lines 476-479:

// Before:
Self {
    providers: RwLock::new(HashMap::new()),
    status_cache: RwLock::new(HashMap::new()),
}

// After:
Self {
    providers: RwLock::new(HashMap::new()),
    status_cache: RwLock::new(Arc::new(HashMap::new())),
}

4. Update register_provider (Copy-on-Write Pattern)

Lines 56-59:

// Before:
self.status_cache
    .write()
    .await
    .insert(provider_id, ProviderStatus::Unavailable);

// After:
// Update status cache with copy-on-write
let mut guard = self.status_cache.write().await;
let mut new_cache = (**guard).clone();
new_cache.insert(provider_id, ProviderStatus::Unavailable);
*guard = Arc::new(new_cache);

Rationale: When updating the HashMap, we clone the inner HashMap (not the Arc), modify it, then wrap in a new Arc. This keeps the immutability contract while allowing readers to continue using old snapshots.

5. Update generate Method (Two Locations)

Lines 76-79 (Available status):

// Before:
self.status_cache
    .write()
    .await
    .insert(provider_id.to_string(), ProviderStatus::Available);

// After:
// Update status cache with copy-on-write
let mut guard = self.status_cache.write().await;
let mut new_cache = (**guard).clone();
new_cache.insert(provider_id.to_string(), ProviderStatus::Available);
*guard = Arc::new(new_cache);

Lines 83-86 (Unavailable status):

// Before:
self.status_cache
    .write()
    .await
    .insert(provider_id.to_string(), ProviderStatus::Unavailable);

// After:
// Update status cache with copy-on-write
let mut guard = self.status_cache.write().await;
let mut new_cache = (**guard).clone();
new_cache.insert(provider_id.to_string(), ProviderStatus::Unavailable);
*guard = Arc::new(new_cache);

6. Update list_all_models (Avoid Intermediate Vec)

Lines 94-132:

// Before:
let providers: Vec<(String, Arc<dyn ModelProvider>)> = {
    let guard = self.providers.read().await;
    guard
        .iter()
        .map(|(id, provider)| (id.clone(), Arc::clone(provider)))
        .collect()
};

let mut tasks = FuturesUnordered::new();

for (provider_id, provider) in providers {
    tasks.push(async move {
        let log_id = provider_id.clone();
        // ...
    });
}

// After:
let mut tasks = FuturesUnordered::new();

{
    let guard = self.providers.read().await;
    for (provider_id, provider) in guard.iter() {
        // Clone Arc and String, but keep lock held for shorter duration
        let provider_id = provider_id.clone();
        let provider = Arc::clone(provider);

        tasks.push(async move {
            // No need for log_id clone - just use provider_id directly
            // ...
        });
    }
}

Rationale:

• Eliminates intermediate Vec allocation
• Still clones provider_id and Arc, but does so inline during iteration
• Lock is held only during spawning (which is fast), not during actual health checks
• Removes unnecessary log_id clone inside async block

7. Update list_all_models Status Updates (Copy-on-Write)

Lines 149-153:

// Before:
{
    let mut guard = self.status_cache.write().await;
    for (provider_id, status) in status_updates {
        guard.insert(provider_id, status);
    }
}

// After:
{
    let mut guard = self.status_cache.write().await;
    let mut new_cache = (**guard).clone();
    for (provider_id, status) in status_updates {
        new_cache.insert(provider_id, status);
    }
    *guard = Arc::new(new_cache);
}

8. Update refresh_health (Avoid Intermediate Vec)

Lines 162-184:

// Before:
let providers: Vec<(String, Arc<dyn ModelProvider>)> = {
    let guard = self.providers.read().await;
    guard
        .iter()
        .map(|(id, provider)| (id.clone(), Arc::clone(provider)))
        .collect()
};

let mut tasks = FuturesUnordered::new();
for (provider_id, provider) in providers {
    tasks.push(async move {
        // ...
    });
}

// After:
let mut tasks = FuturesUnordered::new();

{
    let guard = self.providers.read().await;
    for (provider_id, provider) in guard.iter() {
        let provider_id = provider_id.clone();
        let provider = Arc::clone(provider);

        tasks.push(async move {
            // ...
        });
    }
}

9. Update refresh_health Status Updates (Copy-on-Write)

Lines 191-194:

// Before:
{
    let mut guard = self.status_cache.write().await;
    for (provider_id, status) in &updates {
        guard.insert(provider_id.clone(), *status);
    }
}

// After:
{
    let mut guard = self.status_cache.write().await;
    let mut new_cache = (**guard).clone();
    for (provider_id, status) in &updates {
        new_cache.insert(provider_id.clone(), *status);
    }
    *guard = Arc::new(new_cache);
}

10. Update provider_statuses() Return Type

Lines 218-221:

// Before:
pub async fn provider_statuses(&self) -> HashMap<String, ProviderStatus> {
    let guard = self.status_cache.read().await;
    guard.clone()
}

// After:
/// Snapshot the currently cached statuses.
/// Returns an Arc to avoid cloning the entire HashMap on every call.
pub async fn provider_statuses(&self) -> Arc<HashMap<String, ProviderStatus>> {
    let guard = self.status_cache.read().await;
    Arc::clone(&guard)
}

Rationale: Returns Arc for cheap reference-counted sharing instead of deep clone.

Call Site Updates

File: crates/owlen-cli/src/commands/providers.rs

Lines 218-220:

// Before:
let statuses = manager.provider_statuses().await;
print_models(records, models, statuses);

// After:
let statuses = manager.provider_statuses().await;
print_models(records, models, (*statuses).clone());

Rationale: print_models expects owned HashMap. Clone once at call site instead of always cloning in provider_statuses().

File: crates/owlen-tui/src/app/worker.rs

Add import:

use std::collections::HashMap;

Lines 20-52:

// Before:
let mut last_statuses = provider_manager.provider_statuses().await;

loop {
    // ...
    let statuses = provider_manager.refresh_health().await;

    for (provider_id, status) in statuses {
        let changed = match last_statuses.get(&provider_id) {
            Some(previous) => previous != &status,
            None => true,
        };

        last_statuses.insert(provider_id.clone(), status);

        if changed && message_tx.send(/* ... */).is_err() {
            return;
        }
    }
}

// After:
let mut last_statuses: Arc<HashMap<String, ProviderStatus>> =
    provider_manager.provider_statuses().await;

loop {
    // ...
    let statuses = provider_manager.refresh_health().await;

    for (provider_id, status) in &statuses {
        let changed = match last_statuses.get(provider_id) {
            Some(previous) => previous != status,
            None => true,
        };

        if changed && message_tx.send(AppMessage::ProviderStatus {
            provider_id: provider_id.clone(),
            status: *status,
        }).is_err() {
            return;
        }
    }

    // Update last_statuses after processing all changes
    last_statuses = Arc::new(statuses);
}

Rationale:

• Store Arc instead of owned HashMap
• Iterate over references in loop (avoid moving statuses HashMap)
• Replace entire Arc after all changes processed
• Only clone provider_id when sending message

Performance Impact

Expected improvements:

• list_all_models: 15-20% reduction in execution time (eliminates String clone overhead)
• refresh_health: Similar benefits, plus avoids intermediate Vec allocation
• provider_statuses: ~100x faster for typical HashMap sizes (Arc clone vs deep clone)
• Background worker: Reduced allocations in hot loop (30-second interval)

Trade-offs:

• Status updates now require cloning the HashMap (copy-on-write)
• However, status updates are infrequent compared to reads
• Overall: Optimizes the hot path (reads) at the expense of the cold path (writes)

Testing

Run the following to verify correctness:

cargo test -p owlen-core provider
cargo test -p owlen-tui
cargo test -p owlen-cli

All existing tests should pass without modification.

Alternative Considered: DashMap

The report suggested DashMap as an alternative for lock-free concurrent reads. However, this was rejected in favor of the simpler Arc-based approach because:

1. Simplicity: Arc + RwLock is easier to understand and maintain
2. Sufficient: The current read/write pattern doesn't require lock-free data structures
3. Dependency: Avoids adding another dependency
4. Performance: Arc cloning is already extremely cheap (atomic increment)

If profiling shows RwLock contention in the future, DashMap can be reconsidered.

Implementation Status

Partially Applied: Due to file watcher conflicts (likely rust-analyzer or rustfmt), the changes were documented here but not all applied to the source files.

To complete implementation:

1. Disable file watchers temporarily
2. Apply all changes listed above
3. Run cargo fmt to format the code
4. Run tests to verify correctness
5. Re-enable file watchers

References

• Project analysis report identifying clone overhead
• Rust Arc documentation: https://doc.rust-lang.org/std/sync/struct.Arc.html
• Copy-on-write pattern in Rust
• RwLock best practices