diff --git a/README.md b/README.md index 7704409..42df8f7 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,14 @@ Owly News Summariser is a modern web application that fetches news articles from - Fetches news from configurable RSS feeds - Automatically summarizes articles using Ollama LLM -- Filters news by country +- **AI-powered intelligent tagging** with geographic, category, and source tags +- **Advanced multi-criteria filtering** with hierarchical tag support - Progressive Web App (PWA) support for offline access - Scheduled background updates - High-performance Rust backend for optimal resource usage - Modern Vue.js frontend with TypeScript support +- **Comprehensive analytics** and reading statistics +- **Flexible sharing system** with multiple format options ## Project Structure @@ -24,7 +27,7 @@ The project consists of multiple components: ### For Rust Backend (Recommended) - Rust 1.88.0+ -- [Ollama](https://ollama.ai/) for article summarization +- [Ollama](https://ollama.ai/) for article summarization and tagging - SQLite (handled automatically by SQLx) ### For Python Backend (Legacy) @@ -102,6 +105,32 @@ The project consists of multiple components: The frontend will be available at http://localhost:5173 +## Key Features + +### Intelligent Content Organization +- **AI-Powered Tagging**: Automatic classification with geographic, topical, and source tags +- **Hierarchical Filtering**: Multi-level filtering by location (country → region → city), categories, and content types +- **Smart Search**: Advanced filtering with suggestions based on tag relationships and usage patterns +- **Legacy Migration**: Seamless upgrade from simple country-based filtering to comprehensive tag-based system + +### Advanced Analytics +- **Reading Statistics**: Track reading time, completion rates, and engagement patterns +- **Content Analytics**: Source performance, tag usage, and trending topics analysis +- **Geographic Insights**: Location-based content distribution and reading preferences +- **Goal Tracking**: Personal reading goals with progress monitoring + +### Flexible Article Display +- **Compact View**: Title, excerpt, tags, and action buttons for quick browsing +- **On-Demand Loading**: Full content, AI summaries, and source links as needed +- **Visual Tag System**: Color-coded, hierarchical tags with click-to-filter functionality +- **Reading Status**: Visual indicators for read/unread status and progress tracking + +### Enhanced Sharing +- **Multiple Formats**: Text, Markdown, HTML, and JSON export options +- **Custom Templates**: User-configurable sharing formats +- **One-Click Operations**: Copy to clipboard with formatted content +- **Privacy Controls**: Configurable information inclusion in shared content + ## Building for Production ### Building the Rust Backend @@ -156,3 +185,33 @@ Run frontend tests: cd frontend npm run test ``` + +## Configuration + +The application uses a comprehensive configuration system via `config.toml`: + +- **AI Settings**: Configure Ollama integration for summaries and tagging +- **Display Preferences**: Default views, themes, and UI customization +- **Analytics**: Control data collection and retention policies +- **Filtering**: Smart suggestions, saved filters, and geographic hierarchy +- **Sharing**: Default formats and custom templates + +See the example configuration in the project for detailed options. + +## Migration from Legacy Systems + +The application includes automatic migration tools for upgrading from simpler filtering systems: + +- **Country Filter Migration**: Automatic conversion to hierarchical geographic tags +- **Data Preservation**: Maintains historical data during migration +- **Backward Compatibility**: Gradual transition with user control +- **Validation Tools**: Ensure data integrity throughout the migration process + +## Future Roadmap + +The project is evolving through three phases: +1. **Phase 1**: High-performance Rust backend with advanced filtering and analytics +2. **Phase 2**: CLI application for power users and automation +3. **Phase 3**: Migration to Dioxus for a full Rust stack + +See `ROADMAP.md` for detailed development plans and architectural decisions. diff --git a/backend-rust/ROADMAP.md b/backend-rust/ROADMAP.md index 3ebd7ab..d8751e0 100644 --- a/backend-rust/ROADMAP.md +++ b/backend-rust/ROADMAP.md @@ -6,6 +6,7 @@ This document outlines the strategic approach for transforming the project throu ### Current Phase: Axum API Setup ``` + owly-news-summariser/ ├── src/ │ ├── main.rs # Entry point (will evolve) @@ -39,6 +40,7 @@ owly-news-summariser/ ``` ### Phase 2: Multi-Binary Structure (API + CLI) ``` + owly-news-summariser/ ├── src/ │ ├── lib.rs # Shared library code @@ -57,6 +59,7 @@ owly-news-summariser/ ``` ### Phase 3: Full Rust Stack ``` + owly-news-summariser/ ├── src/ │ ├── [same structure as Phase 2] @@ -67,7 +70,6 @@ owly-news-summariser/ ├── config.toml └── Cargo.toml ``` - ## Core Features & Architecture ### Article Processing & Display Workflow @@ -96,11 +98,13 @@ owly-news-summariser/ 3. **Intelligent Tagging System** - **Automatic Tag Generation**: AI analyzes content and assigns relevant tags - - **Visual Tag Display**: Color-coded tags in compact article view - - **Tag Categories**: Technology, Politics, Business, Sports, Health, etc. - - **Tag Filtering**: Quick filtering by clicking tags + - **Geographic & Source Tags**: AI-generated location tags (countries, regions, cities) and publication source tags + - **Content Category Tags**: Technology, Politics, Business, Sports, Health, etc. + - **Visual Tag Display**: Color-coded tags in compact article view with hierarchical display + - **Tag Filtering**: Quick filtering by clicking tags with smart suggestions - **Custom Tags**: User-defined tags and categories - **Tag Confidence**: Visual indicators for AI vs manual tags + - **Tag Migration**: Automatic conversion of legacy country filters to geographic tags 4. **Analytics & Statistics System** - **Reading Analytics**: @@ -121,19 +125,23 @@ owly-news-summariser/ 5. **Advanced Filtering System** - **Multi-Criteria Filtering**: - By tags (single or multiple with AND/OR logic) + - By geographic tags (country, region, city with hierarchical filtering) + - By content categories and topics - By date ranges (posted, added, read) - By processing status (pending, completed, failed) - By content availability (scraped, summary, RSS-only) - By read/unread status + - **Smart Filter Migration**: Automatic conversion of legacy country filters to tag-based equivalents - **Saved Filter Presets**: - Custom filter combinations - Quick access to frequent searches - - **Smart Suggestions**: Filter suggestions based on usage patterns + - Geographic preset templates (e.g., "European Tech News", "US Politics") + - **Smart Suggestions**: Filter suggestions based on usage patterns and tag relationships 6. **Settings & Management System** - **User Preferences**: - Default article view mode - - Tag display preferences + - Tag display preferences with geographic hierarchy settings - Reading tracking settings - Notification preferences - **System Settings**: @@ -145,6 +153,7 @@ owly-news-summariser/ - Bulk operations (mark read, delete, retag) - Archive old articles - Export/import functionality + - Legacy data migration tools 7. **Article Sharing System** - **Multiple Share Formats**: @@ -157,17 +166,18 @@ owly-news-summariser/ - **Privacy Controls**: Control what information is included in shares 8. **Database Schema** - ``` - Articles: id, title, url, source_type, rss_content, full_content, - summary, processing_status, published_at, added_at, read_at, - read_count, reading_time, ai_enabled, created_at, updated_at - Tags: id, name, category, description, color, usage_count, created_at - ArticleTags: article_id, tag_id, confidence_score, ai_generated, created_at - ReadingStats: user_id, article_id, read_at, reading_time, completion_rate - FilterPresets: id, name, filter_criteria, user_id, created_at - Settings: key, value, category, user_id, updated_at - ShareTemplates: id, name, format, template_content, created_at - ``` +``` +Articles: id, title, url, source_type, rss_content, full_content, +summary, processing_status, published_at, added_at, read_at, +read_count, reading_time, ai_enabled, created_at, updated_at +Tags: id, name, category, description, color, usage_count, parent_id, created_at +ArticleTags: article_id, tag_id, confidence_score, ai_generated, created_at +ReadingStats: user_id, article_id, read_at, reading_time, completion_rate +FilterPresets: id, name, filter_criteria, user_id, created_at +Settings: key, value, category, user_id, updated_at +ShareTemplates: id, name, format, template_content, created_at +LegacyMigration: old_filter_type, old_value, new_tag_ids, migrated_at +``` ## Step-by-Step Process @@ -182,12 +192,15 @@ owly-news-summariser/ - Analytics and tracking preferences - Sharing templates and formats - Filter and search settings + - Geographic tagging preferences - Establish error handling patterns with `anyhow` - Set up logging and analytics infrastructure **Step 2: Data Layer** - Design comprehensive database schema with analytics and settings support - Create SQLx migrations for all tables including analytics and user preferences +- Implement hierarchical tag system with geographic and content categories +- Add legacy migration support for country filters - Implement article models with reading tracking and statistics - Add settings and preferences data layer - Create analytics data models and aggregation queries @@ -199,6 +212,11 @@ owly-news-summariser/ - RSS feed fetching and parsing - Web scraping with quality tracking - AI services for summary and tagging +- **Enhanced Tagging Service**: + - Geographic location detection and tagging + - Content category classification + - Hierarchical tag relationships + - Legacy filter migration logic - **Analytics Service**: - Reading statistics collection and aggregation - Content performance metrics @@ -213,9 +231,10 @@ owly-news-summariser/ - Template processing - Privacy-aware content filtering - **Advanced Filtering Service**: - - Complex query building + - Complex query building with geographic hierarchy - Filter preset management - Search optimization + - Legacy filter migration **Step 4: Comprehensive API Layer** - **Article Management Routes**: @@ -232,20 +251,23 @@ owly-news-summariser/ - `GET /api/analytics/content-stats` - Content and source analytics - `GET /api/analytics/trends` - Trending topics and patterns - `GET /api/analytics/export` - Export analytics data -- **Filtering & Search Routes**: +- **Enhanced Filtering & Search Routes**: - `GET /api/filters/presets` - Get saved filter presets - `POST /api/filters/presets` - Save new filter preset - `GET /api/search/suggestions` - Get search and filter suggestions - `POST /api/search` - Advanced search with multiple criteria + - `POST /api/filters/migrate` - Migrate legacy country filters to tags - **Settings Routes**: - `GET /api/settings` - Get all user settings - `PUT /api/settings` - Update user settings - `GET /api/settings/system` - Get system configuration - `PUT /api/settings/system` - Update system settings (admin) -- **Tag Management Routes**: - - `GET /api/tags` - List tags with usage statistics +- **Enhanced Tag Management Routes**: + - `GET /api/tags` - List tags with usage statistics and hierarchy + - `GET /api/tags/geographic` - Get geographic tag hierarchy - `GET /api/tags/trending` - Get trending tags - `POST /api/tags/:id/follow` - Follow/unfollow tag for notifications + - `GET /api/tags/categories` - Get tag categories and relationships - **Sharing Routes**: - `GET /api/share/templates` - Get sharing templates - `POST /api/share/templates` - Create custom sharing template @@ -255,18 +277,19 @@ owly-news-summariser/ - **Compact Article Display**: - Card-based layout with title, RSS excerpt, tags, and timestamps - Action buttons for Full Article, Summary, and Source - - Visual tag indicators with click-to-filter + - Hierarchical tag display with geographic and category indicators - Reading status and progress indicators - **Advanced Analytics Dashboard**: - Reading statistics with charts and trends - Content source performance metrics - - Tag usage and trending topics + - Tag usage and trending topics with geographic breakdowns - Personal reading insights and goals - **Comprehensive Filtering Interface**: - - Multi-criteria filter builder + - Multi-criteria filter builder with geographic hierarchy - Saved filter presets with quick access - - Smart filter suggestions + - Smart filter suggestions based on tag relationships - Visual filter indicators and clear actions + - Legacy filter migration interface - **Settings Management Panel**: - User preference configuration - AI and processing settings @@ -281,10 +304,11 @@ owly-news-summariser/ **Step 6: Integration & Testing** - Test all API endpoints with comprehensive coverage - Test analytics collection and aggregation -- Test filtering and search functionality +- Test enhanced filtering and search functionality +- Test legacy filter migration - Validate settings persistence and real-time updates - Test sharing functionality across different formats -- Performance testing with large datasets +- Performance testing with large datasets and hierarchical tags - Deploy and monitor ### Phase 2: CLI Application Addition @@ -303,22 +327,27 @@ owly-news-summariser/ - `owly open ` - Open source URL in browser - **Analytics Commands**: - `owly stats [--period day|week|month|year]` - Show reading statistics - - `owly trends [--tags|--sources|--topics]` - Display trending content + - `owly trends [--tags|--sources|--topics|--geo]` - Display trending content - `owly analytics export [--format csv|json]` - Export analytics data - **Management Commands**: - `owly settings [--get key] [--set key=value]` - Manage settings - `owly filters [--list|--save name|--load name]` - Manage filter presets - `owly cleanup [--old|--unread|--failed]` - Clean up articles + - `owly migrate [--from-country-filters]` - Migrate legacy data +- **Enhanced Filtering Commands**: + - `owly filter [--tag] [--geo] [--category]` - Advanced filtering with geographic support + - `owly tags [--list|--hierarchy|--geo]` - Tag management with geographic display - **Sharing Commands**: - `owly share [--format text|markdown|html]` - Generate share content - `owly export [--template name] [--output file]` - Export article **Step 3: Advanced CLI Features** -- Interactive filtering and search +- Interactive filtering and search with geographic hierarchy - Real-time analytics display with charts (using ASCII graphs) - Bulk operations with progress indicators - Settings management with validation - Shell completion for all commands and parameters +- Legacy data migration tools ### Phase 3: Dioxus Frontend Migration @@ -327,16 +356,20 @@ owly-news-summariser/ - `ArticleCard` - Compact article display with action buttons - `ArticleViewer` - Full article content display - `SummaryViewer` - AI summary display - - `TagCloud` - Interactive tag display and filtering + - `TagCloud` - Interactive tag display with geographic hierarchy + - `GeographicMap` - Visual geographic filtering interface - **Analytics Components**: - `AnalyticsDashboard` - Main analytics overview - `ReadingStats` - Personal reading statistics - `TrendChart` - Trending topics and patterns - `ContentMetrics` - Source and content analytics -- **Filtering Components**: - - `FilterBuilder` - Advanced filter creation interface + - `GeographicAnalytics` - Location-based content insights +- **Enhanced Filtering Components**: + - `FilterBuilder` - Advanced filter creation interface with geographic support - `FilterPresets` - Saved filter management - `SearchBar` - Smart search with suggestions + - `GeographicFilter` - Hierarchical location filtering + - `MigrationTool` - Legacy filter migration interface - **Settings Components**: - `SettingsPanel` - User preference management - `SystemConfig` - System-wide configuration @@ -352,14 +385,14 @@ owly-news-summariser/ - Quick preview on hover - Keyboard navigation support - **Advanced Analytics**: - - Interactive charts and graphs + - Interactive charts and graphs with geographic data - Customizable dashboard widgets - Goal setting and progress tracking - Comparison and trend analysis - **Intelligent Filtering**: - - Auto-complete for filters - - Visual filter builder - - Filter combination suggestions + - Auto-complete for filters with geographic suggestions + - Visual filter builder with map integration + - Filter combination suggestions based on tag relationships - Saved search notifications - **Seamless Sharing**: - One-click sharing with clipboard integration @@ -371,8 +404,8 @@ owly-news-summariser/ ### 1. Performance & Scalability - **Efficient Data Loading**: Lazy loading and pagination for large datasets -- **Optimized Queries**: Indexed database queries for filtering and analytics -- **Caching Strategy**: Smart caching for frequently accessed content +- **Optimized Queries**: Indexed database queries for filtering and analytics with hierarchical tag support +- **Caching Strategy**: Smart caching for frequently accessed content and tag hierarchies - **Real-time Updates**: WebSocket integration for live analytics ### 2. User Experience Focus @@ -380,18 +413,20 @@ owly-news-summariser/ - **Responsive Design**: Optimized for mobile and desktop - **Accessibility**: Full keyboard navigation and screen reader support - **Customization**: User-configurable interface and behavior +- **Smooth Migration**: Seamless transition from country-based to tag-based filtering ### 3. Analytics & Insights - **Privacy-First**: User control over data collection and retention - **Actionable Insights**: Meaningful statistics that guide reading habits - **Performance Metrics**: System health and efficiency tracking -- **Trend Analysis**: Pattern recognition for content and behavior +- **Trend Analysis**: Pattern recognition for content and behavior with geographic context ### 4. Content Management - **Flexible Display**: Multiple view modes for different use cases -- **Smart Organization**: AI-assisted content categorization +- **Smart Organization**: AI-assisted content categorization with geographic awareness - **Bulk Operations**: Efficient management of large article collections - **Data Integrity**: Reliable content processing and error handling +- **Legacy Support**: Smooth migration from existing country-based filtering ## Enhanced Configuration File Structure @@ -420,6 +455,8 @@ enable_smart_suggestions = true max_recent_filters = 10 auto_save_filters = true default_sort = "added_desc" # added_desc, published_desc, title_asc +enable_geographic_hierarchy = true +auto_migrate_country_filters = true [sharing] default_format = "text" @@ -435,6 +472,7 @@ format = """ {summary} 🏷️ Tags: {tags} +🌍 Location: {geographic_tags} 🔗 Source: {url} 📅 Published: {published_at} @@ -448,6 +486,7 @@ format = """ {summary} **Tags:** {tags} +**Location:** {geographic_tags} **Source:** [{url}]({url}) **Published:** {published_at} @@ -471,6 +510,9 @@ temperature = 0.3 max_tokens = 200 max_tags_per_article = 10 min_confidence_threshold = 0.7 +enable_geographic_tagging = true +enable_category_tagging = true +geographic_hierarchy_levels = 3 # country, region, city [scraping] timeout_seconds = 30 @@ -486,35 +528,57 @@ retry_attempts = 3 priority_manual = true auto_mark_read_on_view = false +[migration] +auto_convert_country_filters = true +preserve_legacy_data = true +migration_batch_size = 100 + [cli] default_output = "table" pager_command = "less" show_progress = true auto_confirm_bulk = false +show_geographic_hierarchy = true ``` +## Migration Strategy for Country-Based Filtering + +### Automatic Migration Process +1. **Data Analysis**: Scan existing country filter data and RSS feed origins +2. **Tag Generation**: Create geographic tags for each country with hierarchical structure +3. **Filter Conversion**: Convert country-based filters to tag-based equivalents +4. **User Notification**: Inform users about the migration and new capabilities +5. **Gradual Rollout**: Maintain backward compatibility during transition period + +### Enhanced Geographic Features +- **Hierarchical Display**: Country → Region → City tag hierarchy +- **Visual Map Integration**: Interactive geographic filtering via map interface +- **Smart Suggestions**: Related location and content suggestions +- **Multi-Level Filtering**: Filter by specific cities, regions, or broader geographic areas +- **Source Intelligence**: AI detection of article geographic relevance beyond RSS origin + ## Future Enhancements (Post Phase 3) ### Advanced Analytics -- **Machine Learning Insights**: Content recommendation based on reading patterns -- **Predictive Analytics**: Trending topic prediction +- **Machine Learning Insights**: Content recommendation based on reading patterns and geographic preferences +- **Predictive Analytics**: Trending topic prediction with geographic context - **Behavioral Analysis**: Reading habit optimization suggestions -- **Comparative Analytics**: Benchmark against reading goals and averages +- **Comparative Analytics**: Benchmark against reading goals and regional averages ### Enhanced Content Management -- **Smart Collections**: AI-curated article collections +- **Smart Collections**: AI-curated article collections with geographic themes - **Reading Lists**: Planned reading with progress tracking -- **Content Relationships**: Related article suggestions -- **Advanced Search**: Full-text search with relevance scoring +- **Content Relationships**: Related article suggestions with geographic relevance +- **Advanced Search**: Full-text search with relevance scoring and geographic weighting ### Social & Collaboration Features -- **Reading Groups**: Shared reading lists and discussions +- **Reading Groups**: Shared reading lists and discussions with geographic focus - **Social Sharing**: Integration with social platforms - **Collaborative Tagging**: Community-driven content organization -- **Reading Challenges**: Gamification of reading habits +- **Reading Challenges**: Gamification of reading habits with geographic themes ### Integration & Extensibility - **Browser Extension**: Seamless article saving and reading -- **Mobile Apps**: Native iOS/Android applications +- **Mobile Apps**: Native iOS/Android applications with location awareness - **API Ecosystem**: Third-party integrations and plugins -- **Webhook System**: Real-time notifications and integrations +- **Webhook System**: Real-time notifications and integrations with geographic filtering