vikingowl/owly-news
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[update] enhanced ROADMAP.md with advanced geographic tagging, hierarchical filtering, migration strategies, and expanded analytics features, clarified project phase details, and incorporated tagging upgrades into future plans

Browse Source
This commit is contained in:
vikingowl
2025-08-06 12:38:21 +02:00
parent 0f1632ad65
commit b2d82892ef
2 changed files with 175 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

63
README.md
View File

@@ -6,11 +6,14 @@ Owly News Summariser is a modern web application that fetches news articles from
- Fetches news from configurable RSS feeds
- Automatically summarizes articles using Ollama LLM
- Filters news by country
- **AI-powered intelligent tagging** with geographic, category, and source tags
- **Advanced multi-criteria filtering** with hierarchical tag support
- Progressive Web App (PWA) support for offline access
- Scheduled background updates
- High-performance Rust backend for optimal resource usage
- Modern Vue.js frontend with TypeScript support
- **Comprehensive analytics** and reading statistics
- **Flexible sharing system** with multiple format options
## Project Structure
@@ -24,7 +27,7 @@ The project consists of multiple components:
### For Rust Backend (Recommended)
- Rust 1.88.0+
- [Ollama](https://ollama.ai/) for article summarization
- [Ollama](https://ollama.ai/) for article summarization and tagging
- SQLite (handled automatically by SQLx)
### For Python Backend (Legacy)
@@ -102,6 +105,32 @@ The project consists of multiple components:
The frontend will be available at http://localhost:5173
## Key Features
### Intelligent Content Organization
- **AI-Powered Tagging**: Automatic classification with geographic, topical, and source tags
- **Hierarchical Filtering**: Multi-level filtering by location (country → region → city), categories, and content types
- **Smart Search**: Advanced filtering with suggestions based on tag relationships and usage patterns
- **Legacy Migration**: Seamless upgrade from simple country-based filtering to comprehensive tag-based system
### Advanced Analytics
- **Reading Statistics**: Track reading time, completion rates, and engagement patterns
- **Content Analytics**: Source performance, tag usage, and trending topics analysis
- **Geographic Insights**: Location-based content distribution and reading preferences
- **Goal Tracking**: Personal reading goals with progress monitoring
### Flexible Article Display
- **Compact View**: Title, excerpt, tags, and action buttons for quick browsing
- **On-Demand Loading**: Full content, AI summaries, and source links as needed
- **Visual Tag System**: Color-coded, hierarchical tags with click-to-filter functionality
- **Reading Status**: Visual indicators for read/unread status and progress tracking
### Enhanced Sharing
- **Multiple Formats**: Text, Markdown, HTML, and JSON export options
- **Custom Templates**: User-configurable sharing formats
- **One-Click Operations**: Copy to clipboard with formatted content
- **Privacy Controls**: Configurable information inclusion in shared content
## Building for Production
### Building the Rust Backend
@@ -156,3 +185,33 @@ Run frontend tests:
cd frontend
npm run test
```
## Configuration
The application uses a comprehensive configuration system via `config.toml`:
- **AI Settings**: Configure Ollama integration for summaries and tagging
- **Display Preferences**: Default views, themes, and UI customization
- **Analytics**: Control data collection and retention policies
- **Filtering**: Smart suggestions, saved filters, and geographic hierarchy
- **Sharing**: Default formats and custom templates
See the example configuration in the project for detailed options.
## Migration from Legacy Systems
The application includes automatic migration tools for upgrading from simpler filtering systems:
- **Country Filter Migration**: Automatic conversion to hierarchical geographic tags
- **Data Preservation**: Maintains historical data during migration
- **Backward Compatibility**: Gradual transition with user control
- **Validation Tools**: Ensure data integrity throughout the migration process
## Future Roadmap
The project is evolving through three phases:
1. **Phase 1**: High-performance Rust backend with advanced filtering and analytics
2. **Phase 2**: CLI application for power users and automation
3. **Phase 3**: Migration to Dioxus for a full Rust stack
See `ROADMAP.md` for detailed development plans and architectural decisions.

164
backend-rust/ROADMAP.md
View File

@@ -6,6 +6,7 @@ This document outlines the strategic approach for transforming the project throu
### Current Phase: Axum API Setup
```
owly-news-summariser/
├── src/
│ ├── main.rs # Entry point (will evolve)
@@ -39,6 +40,7 @@ owly-news-summariser/
```
### Phase 2: Multi-Binary Structure (API + CLI)
```
owly-news-summariser/
├── src/
│ ├── lib.rs # Shared library code
@@ -57,6 +59,7 @@ owly-news-summariser/
```
### Phase 3: Full Rust Stack
```
owly-news-summariser/
├── src/
│ ├── [same structure as Phase 2]
@@ -67,7 +70,6 @@ owly-news-summariser/
├── config.toml
└── Cargo.toml
```
## Core Features & Architecture
### Article Processing & Display Workflow
@@ -96,11 +98,13 @@ owly-news-summariser/
3. **Intelligent Tagging System**
- **Automatic Tag Generation**: AI analyzes content and assigns relevant tags
- **Visual Tag Display**: Color-coded tags in compact article view
- **Tag Categories**: Technology, Politics, Business, Sports, Health, etc.
- **Tag Filtering**: Quick filtering by clicking tags
- **Geographic & Source Tags**: AI-generated location tags (countries, regions, cities) and publication source tags
- **Content Category Tags**: Technology, Politics, Business, Sports, Health, etc.
- **Visual Tag Display**: Color-coded tags in compact article view with hierarchical display
- **Tag Filtering**: Quick filtering by clicking tags with smart suggestions
- **Custom Tags**: User-defined tags and categories
- **Tag Confidence**: Visual indicators for AI vs manual tags
- **Tag Migration**: Automatic conversion of legacy country filters to geographic tags
4. **Analytics & Statistics System**
- **Reading Analytics**:
@@ -121,19 +125,23 @@ owly-news-summariser/
5. **Advanced Filtering System**
- **Multi-Criteria Filtering**:
- By tags (single or multiple with AND/OR logic)
- By geographic tags (country, region, city with hierarchical filtering)
- By content categories and topics
- By date ranges (posted, added, read)
- By processing status (pending, completed, failed)
- By content availability (scraped, summary, RSS-only)
- By read/unread status
- **Smart Filter Migration**: Automatic conversion of legacy country filters to tag-based equivalents
- **Saved Filter Presets**:
- Custom filter combinations
- Quick access to frequent searches
- **Smart Suggestions**: Filter suggestions based on usage patterns
- Geographic preset templates (e.g., "European Tech News", "US Politics")
- **Smart Suggestions**: Filter suggestions based on usage patterns and tag relationships
6. **Settings & Management System**
- **User Preferences**:
- Default article view mode
- Tag display preferences
- Tag display preferences with geographic hierarchy settings
- Reading tracking settings
- Notification preferences
- **System Settings**:
@@ -145,6 +153,7 @@ owly-news-summariser/
- Bulk operations (mark read, delete, retag)
- Archive old articles
- Export/import functionality
- Legacy data migration tools
7. **Article Sharing System**
- **Multiple Share Formats**:
@@ -157,17 +166,18 @@ owly-news-summariser/
- **Privacy Controls**: Control what information is included in shares
8. **Database Schema**
```
Articles: id, title, url, source_type, rss_content, full_content,
summary, processing_status, published_at, added_at, read_at,
read_count, reading_time, ai_enabled, created_at, updated_at
Tags: id, name, category, description, color, usage_count, created_at
ArticleTags: article_id, tag_id, confidence_score, ai_generated, created_at
ReadingStats: user_id, article_id, read_at, reading_time, completion_rate
FilterPresets: id, name, filter_criteria, user_id, created_at
Settings: key, value, category, user_id, updated_at
ShareTemplates: id, name, format, template_content, created_at
```
```
Articles: id, title, url, source_type, rss_content, full_content,
summary, processing_status, published_at, added_at, read_at,
read_count, reading_time, ai_enabled, created_at, updated_at
Tags: id, name, category, description, color, usage_count, parent_id, created_at
ArticleTags: article_id, tag_id, confidence_score, ai_generated, created_at
ReadingStats: user_id, article_id, read_at, reading_time, completion_rate
FilterPresets: id, name, filter_criteria, user_id, created_at
Settings: key, value, category, user_id, updated_at
ShareTemplates: id, name, format, template_content, created_at
LegacyMigration: old_filter_type, old_value, new_tag_ids, migrated_at
```
## Step-by-Step Process
@@ -182,12 +192,15 @@ owly-news-summariser/
- Analytics and tracking preferences
- Sharing templates and formats
- Filter and search settings
- Geographic tagging preferences
- Establish error handling patterns with `anyhow`
- Set up logging and analytics infrastructure
**Step 2: Data Layer**
- Design comprehensive database schema with analytics and settings support
- Create SQLx migrations for all tables including analytics and user preferences
- Implement hierarchical tag system with geographic and content categories
- Add legacy migration support for country filters
- Implement article models with reading tracking and statistics
- Add settings and preferences data layer
- Create analytics data models and aggregation queries
@@ -199,6 +212,11 @@ owly-news-summariser/
- RSS feed fetching and parsing
- Web scraping with quality tracking
- AI services for summary and tagging
- **Enhanced Tagging Service**:
- Geographic location detection and tagging
- Content category classification
- Hierarchical tag relationships
- Legacy filter migration logic
- **Analytics Service**:
- Reading statistics collection and aggregation
- Content performance metrics
@@ -213,9 +231,10 @@ owly-news-summariser/
- Template processing
- Privacy-aware content filtering
- **Advanced Filtering Service**:
- Complex query building
- Complex query building with geographic hierarchy
- Filter preset management
- Search optimization
- Legacy filter migration
**Step 4: Comprehensive API Layer**
- **Article Management Routes**:
@@ -232,20 +251,23 @@ owly-news-summariser/
- `GET /api/analytics/content-stats` - Content and source analytics
- `GET /api/analytics/trends` - Trending topics and patterns
- `GET /api/analytics/export` - Export analytics data
- **Filtering & Search Routes**:
- **Enhanced Filtering & Search Routes**:
- `GET /api/filters/presets` - Get saved filter presets
- `POST /api/filters/presets` - Save new filter preset
- `GET /api/search/suggestions` - Get search and filter suggestions
- `POST /api/search` - Advanced search with multiple criteria
- `POST /api/filters/migrate` - Migrate legacy country filters to tags
- **Settings Routes**:
- `GET /api/settings` - Get all user settings
- `PUT /api/settings` - Update user settings
- `GET /api/settings/system` - Get system configuration
- `PUT /api/settings/system` - Update system settings (admin)
- **Tag Management Routes**:
- `GET /api/tags` - List tags with usage statistics
- **Enhanced Tag Management Routes**:
- `GET /api/tags` - List tags with usage statistics and hierarchy
- `GET /api/tags/geographic` - Get geographic tag hierarchy
- `GET /api/tags/trending` - Get trending tags
- `POST /api/tags/:id/follow` - Follow/unfollow tag for notifications
- `GET /api/tags/categories` - Get tag categories and relationships
- **Sharing Routes**:
- `GET /api/share/templates` - Get sharing templates
- `POST /api/share/templates` - Create custom sharing template
@@ -255,18 +277,19 @@ owly-news-summariser/
- **Compact Article Display**:
- Card-based layout with title, RSS excerpt, tags, and timestamps
- Action buttons for Full Article, Summary, and Source
- Visual tag indicators with click-to-filter
- Hierarchical tag display with geographic and category indicators
- Reading status and progress indicators
- **Advanced Analytics Dashboard**:
- Reading statistics with charts and trends
- Content source performance metrics
- Tag usage and trending topics
- Tag usage and trending topics with geographic breakdowns
- Personal reading insights and goals
- **Comprehensive Filtering Interface**:
- Multi-criteria filter builder
- Multi-criteria filter builder with geographic hierarchy
- Saved filter presets with quick access
- Smart filter suggestions
- Smart filter suggestions based on tag relationships
- Visual filter indicators and clear actions
- Legacy filter migration interface
- **Settings Management Panel**:
- User preference configuration
- AI and processing settings
@@ -281,10 +304,11 @@ owly-news-summariser/
**Step 6: Integration & Testing**
- Test all API endpoints with comprehensive coverage
- Test analytics collection and aggregation
- Test filtering and search functionality
- Test enhanced filtering and search functionality
- Test legacy filter migration
- Validate settings persistence and real-time updates
- Test sharing functionality across different formats
- Performance testing with large datasets
- Performance testing with large datasets and hierarchical tags
- Deploy and monitor
### Phase 2: CLI Application Addition
@@ -303,22 +327,27 @@ owly-news-summariser/
- `owly open <id>` - Open source URL in browser
- **Analytics Commands**:
- `owly stats [--period day|week|month|year]` - Show reading statistics
- `owly trends [--tags|--sources|--topics]` - Display trending content
- `owly trends [--tags|--sources|--topics|--geo]` - Display trending content
- `owly analytics export [--format csv|json]` - Export analytics data
- **Management Commands**:
- `owly settings [--get key] [--set key=value]` - Manage settings
- `owly filters [--list|--save name|--load name]` - Manage filter presets
- `owly cleanup [--old|--unread|--failed]` - Clean up articles
- `owly migrate [--from-country-filters]` - Migrate legacy data
- **Enhanced Filtering Commands**:
- `owly filter [--tag] [--geo] [--category]` - Advanced filtering with geographic support
- `owly tags [--list|--hierarchy|--geo]` - Tag management with geographic display
- **Sharing Commands**:
- `owly share <id> [--format text|markdown|html]` - Generate share content
- `owly export <id> [--template name] [--output file]` - Export article
**Step 3: Advanced CLI Features**
- Interactive filtering and search
- Interactive filtering and search with geographic hierarchy
- Real-time analytics display with charts (using ASCII graphs)
- Bulk operations with progress indicators
- Settings management with validation
- Shell completion for all commands and parameters
- Legacy data migration tools
### Phase 3: Dioxus Frontend Migration
@@ -327,16 +356,20 @@ owly-news-summariser/
- `ArticleCard` - Compact article display with action buttons
- `ArticleViewer` - Full article content display
- `SummaryViewer` - AI summary display
- `TagCloud` - Interactive tag display and filtering
- `TagCloud` - Interactive tag display with geographic hierarchy
- `GeographicMap` - Visual geographic filtering interface
- **Analytics Components**:
- `AnalyticsDashboard` - Main analytics overview
- `ReadingStats` - Personal reading statistics
- `TrendChart` - Trending topics and patterns
- `ContentMetrics` - Source and content analytics
- **Filtering Components**:
- `FilterBuilder` - Advanced filter creation interface
- `GeographicAnalytics` - Location-based content insights
- **Enhanced Filtering Components**:
- `FilterBuilder` - Advanced filter creation interface with geographic support
- `FilterPresets` - Saved filter management
- `SearchBar` - Smart search with suggestions
- `GeographicFilter` - Hierarchical location filtering
- `MigrationTool` - Legacy filter migration interface
- **Settings Components**:
- `SettingsPanel` - User preference management
- `SystemConfig` - System-wide configuration
@@ -352,14 +385,14 @@ owly-news-summariser/
- Quick preview on hover
- Keyboard navigation support
- **Advanced Analytics**:
- Interactive charts and graphs
- Interactive charts and graphs with geographic data
- Customizable dashboard widgets
- Goal setting and progress tracking
- Comparison and trend analysis
- **Intelligent Filtering**:
- Auto-complete for filters
- Visual filter builder
- Filter combination suggestions
- Auto-complete for filters with geographic suggestions
- Visual filter builder with map integration
- Filter combination suggestions based on tag relationships
- Saved search notifications
- **Seamless Sharing**:
- One-click sharing with clipboard integration
@@ -371,8 +404,8 @@ owly-news-summariser/
### 1. Performance & Scalability
- **Efficient Data Loading**: Lazy loading and pagination for large datasets
- **Optimized Queries**: Indexed database queries for filtering and analytics
- **Caching Strategy**: Smart caching for frequently accessed content
- **Optimized Queries**: Indexed database queries for filtering and analytics with hierarchical tag support
- **Caching Strategy**: Smart caching for frequently accessed content and tag hierarchies
- **Real-time Updates**: WebSocket integration for live analytics
### 2. User Experience Focus
@@ -380,18 +413,20 @@ owly-news-summariser/
- **Responsive Design**: Optimized for mobile and desktop
- **Accessibility**: Full keyboard navigation and screen reader support
- **Customization**: User-configurable interface and behavior
- **Smooth Migration**: Seamless transition from country-based to tag-based filtering
### 3. Analytics & Insights
- **Privacy-First**: User control over data collection and retention
- **Actionable Insights**: Meaningful statistics that guide reading habits
- **Performance Metrics**: System health and efficiency tracking
- **Trend Analysis**: Pattern recognition for content and behavior
- **Trend Analysis**: Pattern recognition for content and behavior with geographic context
### 4. Content Management
- **Flexible Display**: Multiple view modes for different use cases
- **Smart Organization**: AI-assisted content categorization
- **Smart Organization**: AI-assisted content categorization with geographic awareness
- **Bulk Operations**: Efficient management of large article collections
- **Data Integrity**: Reliable content processing and error handling
- **Legacy Support**: Smooth migration from existing country-based filtering
## Enhanced Configuration File Structure
@@ -420,6 +455,8 @@ enable_smart_suggestions = true
max_recent_filters = 10
auto_save_filters = true
default_sort = "added_desc" # added_desc, published_desc, title_asc
enable_geographic_hierarchy = true
auto_migrate_country_filters = true
[sharing]
default_format = "text"
@@ -435,6 +472,7 @@ format = """
{summary}
🏷️ Tags: {tags}
🌍 Location: {geographic_tags}
🔗 Source: {url}
📅 Published: {published_at}
@@ -448,6 +486,7 @@ format = """
{summary}
**Tags:** {tags}
**Location:** {geographic_tags}
**Source:** [{url}]({url})
**Published:** {published_at}
@@ -471,6 +510,9 @@ temperature = 0.3
max_tokens = 200
max_tags_per_article = 10
min_confidence_threshold = 0.7
enable_geographic_tagging = true
enable_category_tagging = true
geographic_hierarchy_levels = 3 # country, region, city
[scraping]
timeout_seconds = 30
@@ -486,35 +528,57 @@ retry_attempts = 3
priority_manual = true
auto_mark_read_on_view = false
[migration]
auto_convert_country_filters = true
preserve_legacy_data = true
migration_batch_size = 100
[cli]
default_output = "table"
pager_command = "less"
show_progress = true
auto_confirm_bulk = false
show_geographic_hierarchy = true
```
## Migration Strategy for Country-Based Filtering
### Automatic Migration Process
1. **Data Analysis**: Scan existing country filter data and RSS feed origins
2. **Tag Generation**: Create geographic tags for each country with hierarchical structure
3. **Filter Conversion**: Convert country-based filters to tag-based equivalents
4. **User Notification**: Inform users about the migration and new capabilities
5. **Gradual Rollout**: Maintain backward compatibility during transition period
### Enhanced Geographic Features
- **Hierarchical Display**: Country → Region → City tag hierarchy
- **Visual Map Integration**: Interactive geographic filtering via map interface
- **Smart Suggestions**: Related location and content suggestions
- **Multi-Level Filtering**: Filter by specific cities, regions, or broader geographic areas
- **Source Intelligence**: AI detection of article geographic relevance beyond RSS origin
## Future Enhancements (Post Phase 3)
### Advanced Analytics
- **Machine Learning Insights**: Content recommendation based on reading patterns
- **Predictive Analytics**: Trending topic prediction
- **Machine Learning Insights**: Content recommendation based on reading patterns and geographic preferences
- **Predictive Analytics**: Trending topic prediction with geographic context
- **Behavioral Analysis**: Reading habit optimization suggestions
- **Comparative Analytics**: Benchmark against reading goals and averages
- **Comparative Analytics**: Benchmark against reading goals and regional averages
### Enhanced Content Management
- **Smart Collections**: AI-curated article collections
- **Smart Collections**: AI-curated article collections with geographic themes
- **Reading Lists**: Planned reading with progress tracking
- **Content Relationships**: Related article suggestions
- **Advanced Search**: Full-text search with relevance scoring
- **Content Relationships**: Related article suggestions with geographic relevance
- **Advanced Search**: Full-text search with relevance scoring and geographic weighting
### Social & Collaboration Features
- **Reading Groups**: Shared reading lists and discussions
- **Reading Groups**: Shared reading lists and discussions with geographic focus
- **Social Sharing**: Integration with social platforms
- **Collaborative Tagging**: Community-driven content organization
- **Reading Challenges**: Gamification of reading habits
- **Reading Challenges**: Gamification of reading habits with geographic themes
### Integration & Extensibility
- **Browser Extension**: Seamless article saving and reading
- **Mobile Apps**: Native iOS/Android applications
- **Mobile Apps**: Native iOS/Android applications with location awareness
- **API Ecosystem**: Third-party integrations and plugins
- **Webhook System**: Real-time notifications and integrations
- **Webhook System**: Real-time notifications and integrations with geographic filtering