[update] expanded ROADMAP.md
to include detailed feature descriptions, enhanced database schema overview, and additional planned system modules
This commit is contained in:
@@ -19,12 +19,18 @@ owly-news-summariser/
|
||||
│ ├── models/ # Data models & database entities
|
||||
│ │ ├── user.rs
|
||||
│ │ ├── article.rs
|
||||
│ │ └── summary.rs
|
||||
│ │ ├── summary.rs
|
||||
│ │ ├── tag.rs # Tag models and relationships
|
||||
│ │ ├── analytics.rs # Analytics and statistics models
|
||||
│ │ └── settings.rs # User settings and preferences
|
||||
│ ├── services.rs # Services module declaration
|
||||
│ ├── services/ # Business logic layer
|
||||
│ │ ├── news_service.rs
|
||||
│ │ ├── summary_service.rs
|
||||
│ │ └── scraping_service.rs # Article content extraction
|
||||
│ │ ├── scraping_service.rs # Article content extraction
|
||||
│ │ ├── tagging_service.rs # AI-powered tagging
|
||||
│ │ ├── analytics_service.rs # Reading stats and analytics
|
||||
│ │ └── sharing_service.rs # Article sharing functionality
|
||||
│ └── config.rs # Configuration management
|
||||
├── migrations/ # SQLx migrations (managed by SQLx CLI)
|
||||
├── frontend/ # Keep existing Vue frontend for now
|
||||
@@ -64,8 +70,8 @@ owly-news-summariser/
|
||||
|
||||
## Core Features & Architecture
|
||||
|
||||
### Article Processing Workflow
|
||||
**Hybrid Approach: RSS Feeds + Manual Submissions with Configurable AI**
|
||||
### Article Processing & Display Workflow
|
||||
**Hybrid Approach: RSS Feeds + Manual Submissions with Smart Content Management**
|
||||
|
||||
1. **Article Collection**
|
||||
- RSS feed monitoring and batch processing
|
||||
@@ -74,33 +80,95 @@ owly-news-summariser/
|
||||
|
||||
2. **Content Processing Pipeline**
|
||||
- Fetch RSS articles → scrape full content → store in DB
|
||||
- Display articles immediately with "Awaiting summary" status
|
||||
- Optional AI summarization with configurable prompts
|
||||
- Support for AI-free mode (content storage only)
|
||||
- Background async summarization with status updates
|
||||
- Support for re-summarization without re-fetching
|
||||
- **Compact Article Display**:
|
||||
- Title (primary display)
|
||||
- RSS description text
|
||||
- Tags (visual indicators)
|
||||
- Time posted (from RSS)
|
||||
- Time added (when added to system)
|
||||
- Action buttons: [Full Article] [Summary] [Source]
|
||||
- **On-Demand Content Loading**:
|
||||
- Full Article: Display complete scraped content
|
||||
- Summary: Show AI-generated summary
|
||||
- Source: Open original URL in new tab
|
||||
- Background async processing with status updates
|
||||
- Support for re-processing without re-fetching
|
||||
|
||||
3. **AI Configuration System**
|
||||
- Toggle AI summarization on/off globally
|
||||
- Configurable AI prompts from config file
|
||||
- Support for different AI providers (Ollama, OpenAI, etc.)
|
||||
- Temperature and context settings
|
||||
- Fallback modes when AI is unavailable
|
||||
3. **Intelligent Tagging System**
|
||||
- **Automatic Tag Generation**: AI analyzes content and assigns relevant tags
|
||||
- **Visual Tag Display**: Color-coded tags in compact article view
|
||||
- **Tag Categories**: Technology, Politics, Business, Sports, Health, etc.
|
||||
- **Tag Filtering**: Quick filtering by clicking tags
|
||||
- **Custom Tags**: User-defined tags and categories
|
||||
- **Tag Confidence**: Visual indicators for AI vs manual tags
|
||||
|
||||
4. **Database Schema**
|
||||
4. **Analytics & Statistics System**
|
||||
- **Reading Analytics**:
|
||||
- Articles read vs added
|
||||
- Reading time tracking
|
||||
- Most read categories and tags
|
||||
- Reading patterns over time
|
||||
- **Content Analytics**:
|
||||
- Source reliability and quality metrics
|
||||
- Tag usage statistics
|
||||
- Processing success rates
|
||||
- Content freshness tracking
|
||||
- **Performance Metrics**:
|
||||
- AI processing times
|
||||
- Scraping success rates
|
||||
- User engagement patterns
|
||||
|
||||
5. **Advanced Filtering System**
|
||||
- **Multi-Criteria Filtering**:
|
||||
- By tags (single or multiple with AND/OR logic)
|
||||
- By date ranges (posted, added, read)
|
||||
- By processing status (pending, completed, failed)
|
||||
- By content availability (scraped, summary, RSS-only)
|
||||
- By read/unread status
|
||||
- **Saved Filter Presets**:
|
||||
- Custom filter combinations
|
||||
- Quick access to frequent searches
|
||||
- **Smart Suggestions**: Filter suggestions based on usage patterns
|
||||
|
||||
6. **Settings & Management System**
|
||||
- **User Preferences**:
|
||||
- Default article view mode
|
||||
- Tag display preferences
|
||||
- Reading tracking settings
|
||||
- Notification preferences
|
||||
- **System Settings**:
|
||||
- AI configuration (via API and config file)
|
||||
- Processing settings
|
||||
- Display customization
|
||||
- Export preferences
|
||||
- **Content Management**:
|
||||
- Bulk operations (mark read, delete, retag)
|
||||
- Archive old articles
|
||||
- Export/import functionality
|
||||
|
||||
7. **Article Sharing System**
|
||||
- **Multiple Share Formats**:
|
||||
- Clean text format with title, summary, and source link
|
||||
- Markdown format for developers
|
||||
- Rich HTML format for email/web
|
||||
- JSON format for API integration
|
||||
- **Copy to Clipboard**: One-click formatted sharing
|
||||
- **Share Templates**: Customizable sharing formats
|
||||
- **Privacy Controls**: Control what information is included in shares
|
||||
|
||||
8. **Database Schema**
|
||||
```
|
||||
Articles: id, title, url, source_type, rss_content, full_content,
|
||||
summary, summary_status, ai_enabled, created_at, updated_at
|
||||
Sources: RSS feeds, Manual submissions with priority levels
|
||||
Config: AI settings, prompts, provider configurations
|
||||
summary, processing_status, published_at, added_at, read_at,
|
||||
read_count, reading_time, ai_enabled, created_at, updated_at
|
||||
Tags: id, name, category, description, color, usage_count, created_at
|
||||
ArticleTags: article_id, tag_id, confidence_score, ai_generated, created_at
|
||||
ReadingStats: user_id, article_id, read_at, reading_time, completion_rate
|
||||
FilterPresets: id, name, filter_criteria, user_id, created_at
|
||||
Settings: key, value, category, user_id, updated_at
|
||||
ShareTemplates: id, name, format, template_content, created_at
|
||||
```
|
||||
|
||||
5. **Processing Priority System**
|
||||
- High: Manually submitted articles (immediate processing)
|
||||
- Normal: Recent RSS articles
|
||||
- Low: RSS backlog
|
||||
- Skip: AI-disabled articles (content-only storage)
|
||||
|
||||
## Step-by-Step Process
|
||||
|
||||
### Phase 1: Axum API Implementation
|
||||
@@ -108,67 +176,115 @@ owly-news-summariser/
|
||||
**Step 1: Core Infrastructure Setup**
|
||||
- Set up database connection pooling with SQLx
|
||||
- **Enhanced Configuration System**:
|
||||
- Extend config.toml with AI settings
|
||||
- AI provider configurations (Ollama, OpenAI, local models)
|
||||
- Customizable AI prompts and templates
|
||||
- Global AI enable/disable toggle
|
||||
- Processing batch sizes and timeouts
|
||||
- Extend config.toml with comprehensive settings
|
||||
- AI provider configurations with separate summary/tagging settings
|
||||
- Display preferences and UI customization
|
||||
- Analytics and tracking preferences
|
||||
- Sharing templates and formats
|
||||
- Filter and search settings
|
||||
- Establish error handling patterns with `anyhow`
|
||||
- Set up logging infrastructure
|
||||
- Set up logging and analytics infrastructure
|
||||
|
||||
**Step 2: Data Layer**
|
||||
- Design database schema with article source tracking and AI settings
|
||||
- Create SQLx migrations using `sqlx migrate add`
|
||||
- Implement article models with RSS/manual source types
|
||||
- Add AI configuration storage and retrieval
|
||||
- Create database access layer with proper async patterns
|
||||
- Design comprehensive database schema with analytics and settings support
|
||||
- Create SQLx migrations for all tables including analytics and user preferences
|
||||
- Implement article models with reading tracking and statistics
|
||||
- Add settings and preferences data layer
|
||||
- Create analytics data models and aggregation queries
|
||||
- Implement sharing templates and format management
|
||||
- Use SQLx's compile-time checked queries
|
||||
|
||||
**Step 3: Content Processing Services**
|
||||
- Implement RSS feed fetching and parsing
|
||||
- Create web scraping service for full article content
|
||||
- **Flexible AI Service**:
|
||||
- Configurable AI summarization with prompt templates
|
||||
- Support for multiple AI providers
|
||||
- AI-disabled mode for content-only processing
|
||||
- Retry logic and fallback strategies
|
||||
- Custom prompt injection from config
|
||||
- Build background job system for async summarization
|
||||
- Add content validation and error handling
|
||||
**Step 3: Enhanced Services Layer**
|
||||
- **Content Processing Services**:
|
||||
- RSS feed fetching and parsing
|
||||
- Web scraping with quality tracking
|
||||
- AI services for summary and tagging
|
||||
- **Analytics Service**:
|
||||
- Reading statistics collection and aggregation
|
||||
- Content performance metrics
|
||||
- User behavior tracking
|
||||
- Trend analysis and insights
|
||||
- **Settings Management Service**:
|
||||
- User preference handling
|
||||
- System configuration management
|
||||
- Real-time settings updates
|
||||
- **Sharing Service**:
|
||||
- Multiple format generation
|
||||
- Template processing
|
||||
- Privacy-aware content filtering
|
||||
- **Advanced Filtering Service**:
|
||||
- Complex query building
|
||||
- Filter preset management
|
||||
- Search optimization
|
||||
|
||||
**Step 4: API Layer Architecture**
|
||||
**Step 4: Comprehensive API Layer**
|
||||
- **Article Management Routes**:
|
||||
- `GET /api/articles` - List all articles with status
|
||||
- `GET /api/articles` - List articles with compact display data
|
||||
- `POST /api/articles` - Submit manual article URL
|
||||
- `GET /api/articles/:id` - Get specific article
|
||||
- `POST /api/articles/:id/summary` - Trigger/re-trigger summary
|
||||
- `POST /api/articles/:id/toggle-ai` - Enable/disable AI for article
|
||||
- **RSS Feed Management**:
|
||||
- `GET /api/feeds` - List RSS feeds
|
||||
- `POST /api/feeds` - Add RSS feed
|
||||
- `POST /api/feeds/:id/sync` - Manual feed sync
|
||||
- **Configuration Management**:
|
||||
- `GET /api/config` - Get current configuration
|
||||
- `POST /api/config/ai` - Update AI settings
|
||||
- `POST /api/config/prompts` - Update AI prompts
|
||||
- **Summary Management**:
|
||||
- `GET /api/summaries` - List summaries
|
||||
- WebSocket/SSE for real-time summary updates
|
||||
- `GET /api/articles/:id` - Get basic article info
|
||||
- `GET /api/articles/:id/full` - Get complete scraped content
|
||||
- `GET /api/articles/:id/summary` - Get AI summary
|
||||
- `POST /api/articles/:id/read` - Mark as read and track reading time
|
||||
- `POST /api/articles/:id/share` - Generate shareable content
|
||||
- **Analytics Routes**:
|
||||
- `GET /api/analytics/dashboard` - Main analytics dashboard data
|
||||
- `GET /api/analytics/reading-stats` - Personal reading statistics
|
||||
- `GET /api/analytics/content-stats` - Content and source analytics
|
||||
- `GET /api/analytics/trends` - Trending topics and patterns
|
||||
- `GET /api/analytics/export` - Export analytics data
|
||||
- **Filtering & Search Routes**:
|
||||
- `GET /api/filters/presets` - Get saved filter presets
|
||||
- `POST /api/filters/presets` - Save new filter preset
|
||||
- `GET /api/search/suggestions` - Get search and filter suggestions
|
||||
- `POST /api/search` - Advanced search with multiple criteria
|
||||
- **Settings Routes**:
|
||||
- `GET /api/settings` - Get all user settings
|
||||
- `PUT /api/settings` - Update user settings
|
||||
- `GET /api/settings/system` - Get system configuration
|
||||
- `PUT /api/settings/system` - Update system settings (admin)
|
||||
- **Tag Management Routes**:
|
||||
- `GET /api/tags` - List tags with usage statistics
|
||||
- `GET /api/tags/trending` - Get trending tags
|
||||
- `POST /api/tags/:id/follow` - Follow/unfollow tag for notifications
|
||||
- **Sharing Routes**:
|
||||
- `GET /api/share/templates` - Get sharing templates
|
||||
- `POST /api/share/templates` - Create custom sharing template
|
||||
- `POST /api/articles/:id/share/:format` - Generate share content
|
||||
|
||||
**Step 5: Frontend Integration Features**
|
||||
- Article list with status indicators and AI toggle
|
||||
- "Add Article" form with URL input, validation, and AI option
|
||||
- AI configuration panel with prompt editing
|
||||
- Real-time status updates for processing articles
|
||||
- Bulk article import functionality with AI settings
|
||||
- Summary regeneration controls with prompt modification
|
||||
- Global AI enable/disable toggle in settings
|
||||
**Step 5: Enhanced Frontend Features**
|
||||
- **Compact Article Display**:
|
||||
- Card-based layout with title, RSS excerpt, tags, and timestamps
|
||||
- Action buttons for Full Article, Summary, and Source
|
||||
- Visual tag indicators with click-to-filter
|
||||
- Reading status and progress indicators
|
||||
- **Advanced Analytics Dashboard**:
|
||||
- Reading statistics with charts and trends
|
||||
- Content source performance metrics
|
||||
- Tag usage and trending topics
|
||||
- Personal reading insights and goals
|
||||
- **Comprehensive Filtering Interface**:
|
||||
- Multi-criteria filter builder
|
||||
- Saved filter presets with quick access
|
||||
- Smart filter suggestions
|
||||
- Visual filter indicators and clear actions
|
||||
- **Settings Management Panel**:
|
||||
- User preference configuration
|
||||
- AI and processing settings
|
||||
- Display and UI customization
|
||||
- Export/import functionality
|
||||
- **Enhanced Sharing System**:
|
||||
- Quick share buttons with format selection
|
||||
- Copy-to-clipboard functionality
|
||||
- Custom sharing templates
|
||||
- Preview before sharing
|
||||
|
||||
**Step 6: Integration & Testing**
|
||||
- Test all API endpoints thoroughly
|
||||
- Test AI-enabled and AI-disabled workflows
|
||||
- Ensure Vue frontend works seamlessly with new Rust backend
|
||||
- Performance testing and optimization
|
||||
- Test all API endpoints with comprehensive coverage
|
||||
- Test analytics collection and aggregation
|
||||
- Test filtering and search functionality
|
||||
- Validate settings persistence and real-time updates
|
||||
- Test sharing functionality across different formats
|
||||
- Performance testing with large datasets
|
||||
- Deploy and monitor
|
||||
|
||||
### Phase 2: CLI Application Addition
|
||||
@@ -178,135 +294,104 @@ owly-news-summariser/
|
||||
- Create `src/bin/cli.rs` for CLI application
|
||||
- Keep shared logic in `src/lib.rs`
|
||||
- Update Cargo.toml to support multiple binaries
|
||||
- Add clap with completion feature
|
||||
|
||||
**Step 2: CLI Architecture with Auto-completion**
|
||||
- **Shell Completion Support**:
|
||||
- Generate bash, zsh, and fish completion scripts
|
||||
- `owly completion bash > /etc/bash_completion.d/owly`
|
||||
- `owly completion zsh > ~/.zsh/completions/_owly`
|
||||
- Dynamic completion for article IDs, feed URLs, etc.
|
||||
- **Enhanced CLI Commands**:
|
||||
- `owly add-url <url> [--no-ai]` - Add single article for processing
|
||||
- `owly add-feed <rss-url> [--no-ai]` - Add RSS feed
|
||||
- `owly sync [--no-ai]` - Sync all RSS feeds
|
||||
- `owly process [--ai-only|--no-ai]` - Process pending articles
|
||||
- `owly list [--status pending|completed|failed]` - List articles with filtering
|
||||
- `owly summarize <id> [--prompt <custom-prompt>]` - Re-summarize specific article
|
||||
- `owly config ai [--enable|--disable]` - Configure AI settings
|
||||
- `owly config prompt [--set <prompt>|--reset]` - Manage AI prompts
|
||||
- `owly completion <shell>` - Generate shell completions
|
||||
- Reuse existing services and models from the API
|
||||
- Create CLI-specific output formatting with rich progress indicators
|
||||
- Implement batch processing capabilities with progress bars
|
||||
- Support piping URLs for bulk processing
|
||||
- Interactive prompt editing mode
|
||||
**Step 2: Enhanced CLI with Analytics and Management**
|
||||
- **Core Commands**:
|
||||
- `owly list [--filters] [--format table|json|compact]` - List articles
|
||||
- `owly show <id> [--content|--summary]` - Display specific article
|
||||
- `owly read <id>` - Mark article as read and open in pager
|
||||
- `owly open <id>` - Open source URL in browser
|
||||
- **Analytics Commands**:
|
||||
- `owly stats [--period day|week|month|year]` - Show reading statistics
|
||||
- `owly trends [--tags|--sources|--topics]` - Display trending content
|
||||
- `owly analytics export [--format csv|json]` - Export analytics data
|
||||
- **Management Commands**:
|
||||
- `owly settings [--get key] [--set key=value]` - Manage settings
|
||||
- `owly filters [--list|--save name|--load name]` - Manage filter presets
|
||||
- `owly cleanup [--old|--unread|--failed]` - Clean up articles
|
||||
- **Sharing Commands**:
|
||||
- `owly share <id> [--format text|markdown|html]` - Generate share content
|
||||
- `owly export <id> [--template name] [--output file]` - Export article
|
||||
|
||||
**Step 3: Shared Core Logic**
|
||||
- Extract common functionality into library crates
|
||||
- Ensure both API and CLI can use the same business logic
|
||||
- Implement proper configuration management for both contexts
|
||||
- Share article processing pipeline between web and CLI
|
||||
- Unified AI service layer for both interfaces
|
||||
**Step 3: Advanced CLI Features**
|
||||
- Interactive filtering and search
|
||||
- Real-time analytics display with charts (using ASCII graphs)
|
||||
- Bulk operations with progress indicators
|
||||
- Settings management with validation
|
||||
- Shell completion for all commands and parameters
|
||||
|
||||
### Phase 3: Dioxus Frontend Migration
|
||||
|
||||
**Step 1: Parallel Development**
|
||||
- Create new `frontend-dioxus/` directory
|
||||
- Keep existing Vue frontend running during development
|
||||
- Set up Dioxus project structure with proper routing
|
||||
**Step 1: Component Architecture**
|
||||
- **Core Display Components**:
|
||||
- `ArticleCard` - Compact article display with action buttons
|
||||
- `ArticleViewer` - Full article content display
|
||||
- `SummaryViewer` - AI summary display
|
||||
- `TagCloud` - Interactive tag display and filtering
|
||||
- **Analytics Components**:
|
||||
- `AnalyticsDashboard` - Main analytics overview
|
||||
- `ReadingStats` - Personal reading statistics
|
||||
- `TrendChart` - Trending topics and patterns
|
||||
- `ContentMetrics` - Source and content analytics
|
||||
- **Filtering Components**:
|
||||
- `FilterBuilder` - Advanced filter creation interface
|
||||
- `FilterPresets` - Saved filter management
|
||||
- `SearchBar` - Smart search with suggestions
|
||||
- **Settings Components**:
|
||||
- `SettingsPanel` - User preference management
|
||||
- `SystemConfig` - System-wide configuration
|
||||
- `ExportImport` - Data export/import functionality
|
||||
- **Sharing Components**:
|
||||
- `ShareDialog` - Sharing interface with format options
|
||||
- `ShareTemplates` - Custom template management
|
||||
|
||||
**Step 2: Component Architecture**
|
||||
- Design reusable Dioxus components
|
||||
- **Core Components**:
|
||||
- ArticleList with status filtering and AI indicators
|
||||
- AddArticleForm with URL validation and AI toggle
|
||||
- SummaryDisplay with regeneration controls and prompt editing
|
||||
- FeedManager for RSS feed management with AI settings
|
||||
- StatusIndicator for processing states
|
||||
- ConfigPanel for AI settings and prompt management
|
||||
- AIToggle for per-article AI control
|
||||
- Implement state management (similar to Pinia in Vue)
|
||||
- Create API client layer for communication with Rust backend
|
||||
- WebSocket integration for real-time updates
|
||||
|
||||
**Step 3: Feature Parity & UX Enhancements**
|
||||
- Port Vue components to Dioxus incrementally
|
||||
- **Enhanced UX Features**:
|
||||
- Smart URL validation with preview
|
||||
- Batch article import with drag-and-drop and AI options
|
||||
- Advanced filtering and search (by AI status, processing state)
|
||||
- Processing queue management with AI priority
|
||||
- Summary comparison tools
|
||||
- AI prompt editor with syntax highlighting
|
||||
- Configuration import/export functionality
|
||||
- Implement proper error handling and loading states
|
||||
- Progressive Web App features
|
||||
|
||||
**Step 4: Final Migration**
|
||||
- Switch production traffic to Dioxus frontend
|
||||
- Remove Vue frontend after thorough testing
|
||||
- Optimize bundle size and performance
|
||||
**Step 2: Enhanced UX Features**
|
||||
- **Smart Article Display**:
|
||||
- Lazy loading for performance
|
||||
- Infinite scroll with virtualization
|
||||
- Quick preview on hover
|
||||
- Keyboard navigation support
|
||||
- **Advanced Analytics**:
|
||||
- Interactive charts and graphs
|
||||
- Customizable dashboard widgets
|
||||
- Goal setting and progress tracking
|
||||
- Comparison and trend analysis
|
||||
- **Intelligent Filtering**:
|
||||
- Auto-complete for filters
|
||||
- Visual filter builder
|
||||
- Filter combination suggestions
|
||||
- Saved search notifications
|
||||
- **Seamless Sharing**:
|
||||
- One-click sharing with clipboard integration
|
||||
- Live preview of shared content
|
||||
- Social media format optimization
|
||||
- Batch sharing capabilities
|
||||
|
||||
## Key Strategic Considerations
|
||||
|
||||
### 1. Modern Rust Practices
|
||||
- Use modern module structure without `mod.rs` files
|
||||
- Leverage SQLx's built-in migration and connection management
|
||||
- Follow Rust 2018+ edition conventions
|
||||
### 1. Performance & Scalability
|
||||
- **Efficient Data Loading**: Lazy loading and pagination for large datasets
|
||||
- **Optimized Queries**: Indexed database queries for filtering and analytics
|
||||
- **Caching Strategy**: Smart caching for frequently accessed content
|
||||
- **Real-time Updates**: WebSocket integration for live analytics
|
||||
|
||||
### 2. Maintain Backward Compatibility
|
||||
- Keep API contracts stable during Vue-to-Dioxus transition
|
||||
- Use feature flags for gradual rollouts
|
||||
- Support legacy configurations during migration
|
||||
### 2. User Experience Focus
|
||||
- **Progressive Disclosure**: Show essential info first, details on demand
|
||||
- **Responsive Design**: Optimized for mobile and desktop
|
||||
- **Accessibility**: Full keyboard navigation and screen reader support
|
||||
- **Customization**: User-configurable interface and behavior
|
||||
|
||||
### 3. Shared Code Architecture
|
||||
- Design your core business logic to be framework-agnostic
|
||||
- Use workspace structure for better code organization
|
||||
- Consider extracting domain logic into separate crates
|
||||
- AI service abstraction for provider flexibility
|
||||
### 3. Analytics & Insights
|
||||
- **Privacy-First**: User control over data collection and retention
|
||||
- **Actionable Insights**: Meaningful statistics that guide reading habits
|
||||
- **Performance Metrics**: System health and efficiency tracking
|
||||
- **Trend Analysis**: Pattern recognition for content and behavior
|
||||
|
||||
### 4. Content Processing Strategy
|
||||
- **Resilient Pipeline**: Store original content for offline re-processing
|
||||
- **Progressive Enhancement**: Show articles immediately, summaries when ready
|
||||
- **Flexible AI Integration**: Support both AI-enabled and AI-disabled workflows
|
||||
- **Error Recovery**: Graceful handling of scraping/summarization failures
|
||||
- **Rate Limiting**: Respect external API limits and website politeness
|
||||
- **Configuration-Driven**: All AI behavior controlled via config files
|
||||
|
||||
### 5. CLI User Experience
|
||||
- **Shell Integration**: Auto-completion for all major shells
|
||||
- **Interactive Mode**: Guided prompts for complex operations
|
||||
- **Batch Processing**: Efficient handling of large article sets
|
||||
- **Progress Indicators**: Clear feedback for long-running operations
|
||||
- **Flexible Output**: JSON, table, and human-readable formats
|
||||
|
||||
### 6. Testing Strategy
|
||||
- Unit tests for business logic
|
||||
- Integration tests for API endpoints
|
||||
- End-to-end tests for the full stack
|
||||
- CLI integration tests with completion validation
|
||||
- Content scraping reliability tests
|
||||
- AI service mocking and fallback testing
|
||||
|
||||
### 7. Configuration Management
|
||||
- **Layered Configuration**:
|
||||
- Default settings
|
||||
- Config file overrides
|
||||
- Environment variable overrides
|
||||
- Runtime API updates
|
||||
- Support for different deployment scenarios
|
||||
- Proper secrets management
|
||||
- Hot-reloading of AI prompts and settings
|
||||
- Configuration validation and migration
|
||||
|
||||
### 8. Database Strategy
|
||||
- Use SQLx migrations for schema evolution (`sqlx migrate add/run`)
|
||||
- Leverage compile-time checked queries with SQLx macros
|
||||
- Implement proper connection pooling and error handling
|
||||
- Let SQLx handle what it does best - don't reinvent the wheel
|
||||
- Design for both read-heavy (web UI) and write-heavy (batch processing) patterns
|
||||
- Store AI configuration and prompt history
|
||||
### 4. Content Management
|
||||
- **Flexible Display**: Multiple view modes for different use cases
|
||||
- **Smart Organization**: AI-assisted content categorization
|
||||
- **Bulk Operations**: Efficient management of large article collections
|
||||
- **Data Integrity**: Reliable content processing and error handling
|
||||
|
||||
## Enhanced Configuration File Structure
|
||||
|
||||
@@ -315,81 +400,122 @@ owly-news-summariser/
|
||||
host = '127.0.0.1'
|
||||
port = 8090
|
||||
|
||||
[display]
|
||||
default_view = "compact" # compact, full, summary
|
||||
articles_per_page = 50
|
||||
show_reading_time = true
|
||||
show_word_count = false
|
||||
highlight_unread = true
|
||||
theme = "auto" # light, dark, auto
|
||||
|
||||
[analytics]
|
||||
enabled = true
|
||||
track_reading_time = true
|
||||
track_scroll_position = true
|
||||
retention_days = 365 # How long to keep detailed analytics
|
||||
aggregate_older_data = true
|
||||
|
||||
[filtering]
|
||||
enable_smart_suggestions = true
|
||||
max_recent_filters = 10
|
||||
auto_save_filters = true
|
||||
default_sort = "added_desc" # added_desc, published_desc, title_asc
|
||||
|
||||
[sharing]
|
||||
default_format = "text"
|
||||
include_summary = true
|
||||
include_tags = true
|
||||
include_source = true
|
||||
copy_to_clipboard = true
|
||||
|
||||
[sharing.templates.text]
|
||||
format = """
|
||||
📰 {title}
|
||||
|
||||
{summary}
|
||||
|
||||
🏷️ Tags: {tags}
|
||||
🔗 Source: {url}
|
||||
📅 Published: {published_at}
|
||||
|
||||
Shared via Owly News Summariser
|
||||
"""
|
||||
|
||||
[sharing.templates.markdown]
|
||||
format = """
|
||||
# {title}
|
||||
|
||||
{summary}
|
||||
|
||||
**Tags:** {tags}
|
||||
**Source:** [{url}]({url})
|
||||
**Published:** {published_at}
|
||||
|
||||
---
|
||||
*Shared via Owly News Summariser*
|
||||
"""
|
||||
|
||||
[ai]
|
||||
enabled = true
|
||||
provider = "ollama" # ollama, openai, local
|
||||
provider = "ollama"
|
||||
timeout_seconds = 120
|
||||
|
||||
[ai.summary]
|
||||
enabled = true
|
||||
temperature = 0.1
|
||||
max_tokens = 1000
|
||||
|
||||
[ai.ollama]
|
||||
host = "http://localhost:11434"
|
||||
model = "llama2"
|
||||
context_size = 8192
|
||||
[ai.tagging]
|
||||
enabled = true
|
||||
temperature = 0.3
|
||||
max_tokens = 200
|
||||
max_tags_per_article = 10
|
||||
min_confidence_threshold = 0.7
|
||||
|
||||
[ai.openai]
|
||||
api_key_env = "OPENAI_API_KEY"
|
||||
model = "gpt-3.5-turbo"
|
||||
|
||||
[ai.prompts]
|
||||
default = """
|
||||
Analyze this article and provide a concise summary in JSON format:
|
||||
{
|
||||
"title": "article title",
|
||||
"summary": "brief summary in 2-3 sentences",
|
||||
"key_points": ["point1", "point2", "point3"]
|
||||
}
|
||||
|
||||
Article URL: {url}
|
||||
Article Title: {title}
|
||||
Article Content: {content}
|
||||
"""
|
||||
|
||||
news = """Custom prompt for news articles..."""
|
||||
technical = """Custom prompt for technical articles..."""
|
||||
[scraping]
|
||||
timeout_seconds = 30
|
||||
max_retries = 3
|
||||
max_content_length = 50000
|
||||
respect_robots_txt = true
|
||||
rate_limit_delay_ms = 1000
|
||||
|
||||
[processing]
|
||||
batch_size = 10
|
||||
max_concurrent = 5
|
||||
retry_attempts = 3
|
||||
priority_manual = true
|
||||
auto_mark_read_on_view = false
|
||||
|
||||
[cli]
|
||||
default_output = "table" # table, json, compact
|
||||
auto_confirm = false
|
||||
default_output = "table"
|
||||
pager_command = "less"
|
||||
show_progress = true
|
||||
auto_confirm_bulk = false
|
||||
```
|
||||
|
||||
## What SQLx Handles for You
|
||||
|
||||
- **Migrations**: Use `sqlx migrate add <name>` to create, `sqlx::migrate!()` macro to embed
|
||||
- **Connection Pooling**: Built-in `SqlitePool` with configuration options
|
||||
- **Query Safety**: Compile-time checked queries prevent SQL injection and typos
|
||||
- **Type Safety**: Automatic Rust type mapping from database types
|
||||
|
||||
## Future Enhancements (Post Phase 3)
|
||||
|
||||
### Advanced Features
|
||||
- **Machine Learning**: Article classification and topic clustering
|
||||
- **Analytics Dashboard**: Reading patterns and trending topics
|
||||
- **Content Deduplication**: Identify and merge similar articles
|
||||
- **Multi-language Support**: Translation and summarization
|
||||
- **API Rate Limiting**: User quotas and fair usage policies
|
||||
- **Advanced AI Features**: Custom model training, prompt optimization
|
||||
### Advanced Analytics
|
||||
- **Machine Learning Insights**: Content recommendation based on reading patterns
|
||||
- **Predictive Analytics**: Trending topic prediction
|
||||
- **Behavioral Analysis**: Reading habit optimization suggestions
|
||||
- **Comparative Analytics**: Benchmark against reading goals and averages
|
||||
|
||||
### Extended Integrations
|
||||
- **Browser Extension**: One-click article addition (far future)
|
||||
- **Mobile App**: Native iOS/Android apps using Rust core
|
||||
- **Social Features**: Shared reading lists and collaborative summaries
|
||||
- **Export Features**: PDF, EPUB, and other format exports
|
||||
- **Third-party Integrations**: Pocket, Instapaper, read-later services
|
||||
- **AI Provider Ecosystem**: Support for more AI services and local models
|
||||
### Enhanced Content Management
|
||||
- **Smart Collections**: AI-curated article collections
|
||||
- **Reading Lists**: Planned reading with progress tracking
|
||||
- **Content Relationships**: Related article suggestions
|
||||
- **Advanced Search**: Full-text search with relevance scoring
|
||||
|
||||
### Scalability Improvements
|
||||
- **Microservices**: Split processing services for horizontal scaling
|
||||
- **Message Queues**: Redis/RabbitMQ for high-throughput processing
|
||||
- **Caching Layer**: Redis for frequently accessed summaries
|
||||
- **CDN Integration**: Static asset and summary caching
|
||||
- **Multi-database**: PostgreSQL for production, advanced analytics
|
||||
- **Distributed AI**: Load balancing across multiple AI providers
|
||||
### Social & Collaboration Features
|
||||
- **Reading Groups**: Shared reading lists and discussions
|
||||
- **Social Sharing**: Integration with social platforms
|
||||
- **Collaborative Tagging**: Community-driven content organization
|
||||
- **Reading Challenges**: Gamification of reading habits
|
||||
|
||||
### Integration & Extensibility
|
||||
- **Browser Extension**: Seamless article saving and reading
|
||||
- **Mobile Apps**: Native iOS/Android applications
|
||||
- **API Ecosystem**: Third-party integrations and plugins
|
||||
- **Webhook System**: Real-time notifications and integrations
|
||||
```
|
||||
|
Reference in New Issue
Block a user