* Fix race condition in concurrent crawling with unique source IDs
- Add unique hash-based source_id generation to prevent conflicts
- Separate source identification from display with three fields:
- source_id: 16-char SHA256 hash for unique identification
- source_url: Original URL for tracking
- source_display_name: Human-friendly name for UI
- Add comprehensive test suite validating the fix
- Migrate existing data with backward compatibility
* Fix title generation to use source_display_name for better AI context
- Pass source_display_name to title generation function
- Use display name in AI prompt instead of hash-based source_id
- Results in more specific, meaningful titles for each source
* Skip AI title generation when display name is available
- Use source_display_name directly as title to avoid unnecessary AI calls
- More efficient and predictable than AI-generated titles
- Keep AI generation only as fallback for backward compatibility
* Fix critical issues from code review
- Add missing os import to prevent NameError crash
- Remove unused imports (pytest, Mock, patch, hashlib, urlparse, etc.)
- Fix GitHub API capitalization consistency
- Reuse existing DocumentStorageService instance
- Update test expectations to match corrected capitalization
Addresses CodeRabbit review feedback on PR #472
* Add safety improvements from code review
- Truncate display names to 100 chars when used as titles
- Document hash collision probability (negligible for <1M sources)
Simple, pragmatic fixes per KISS principle
* Fix code extraction to use hash-based source_ids and improve display names
- Fixed critical bug where code extraction was using old domain-based source_ids
- Updated code extraction service to accept source_id as parameter instead of extracting from URL
- Added special handling for llms.txt and sitemap.xml files in display names
- Added comprehensive tests for source_id handling in code extraction
- Removed unused urlparse import from code_extraction_service.py
This fixes the foreign key constraint errors that were preventing code examples
from being stored after the source_id architecture refactor.
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix critical variable shadowing and source_type determination issues
- Fixed variable shadowing in document_storage_operations.py where source_url parameter
was being overwritten by document URLs, causing incorrect source_url in database
- Fixed source_type determination to use actual URLs instead of hash-based source_id
- Added comprehensive tests for source URL preservation
- Ensure source_type is correctly set to "file" for file uploads, "url" for web crawls
The variable shadowing bug was causing sitemap sources to have the wrong source_url
(last crawled page instead of sitemap URL). The source_type bug would mark all
sources as "url" even for file uploads due to hash-based IDs not starting with "file_".
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix URL canonicalization and document metrics calculation
- Implement proper URL canonicalization to prevent duplicate sources
- Remove trailing slashes (except root)
- Remove URL fragments
- Remove tracking parameters (utm_*, gclid, fbclid, etc.)
- Sort query parameters for consistency
- Remove default ports (80 for HTTP, 443 for HTTPS)
- Normalize scheme and domain to lowercase
- Fix avg_chunks_per_doc calculation to avoid division by zero
- Track processed_docs count separately from total crawl_results
- Handle all-empty document sets gracefully
- Show processed/total in logs for better visibility
- Add comprehensive tests for both fixes
- 10 test cases for URL canonicalization edge cases
- 4 test cases for document metrics calculation
This prevents database constraint violations when crawling the same
content with URL variations and provides accurate metrics in logs.
* Fix synchronous extract_source_summary blocking async event loop
- Run extract_source_summary in thread pool using asyncio.to_thread
- Prevents blocking the async event loop during AI summary generation
- Preserves exact error handling and fallback behavior
- Variables (source_id, combined_content) properly passed to thread
Added comprehensive tests verifying:
- Function runs in thread without blocking
- Error handling works correctly with fallback
- Multiple sources can be processed
- Thread safety with variable passing
* Fix synchronous update_source_info blocking async event loop
- Run update_source_info in thread pool using asyncio.to_thread
- Prevents blocking the async event loop during database operations
- Preserves exact error handling and fallback behavior
- All kwargs properly passed to thread execution
Added comprehensive tests verifying:
- Function runs in thread without blocking
- Error handling triggers fallback correctly
- All kwargs are preserved when passed to thread
- Existing extract_source_summary tests still pass
* Fix race condition in source creation using upsert
- Replace INSERT with UPSERT for new sources to prevent PRIMARY KEY violations
- Handles concurrent crawls attempting to create the same source
- Maintains existing UPDATE behavior for sources that already exist
Added comprehensive tests verifying:
- Concurrent source creation doesn't fail
- Upsert is used for new sources (not insert)
- Update is still used for existing sources
- Async concurrent operations work correctly
- Race conditions with delays are handled
This prevents database constraint errors when multiple crawls target
the same URL simultaneously.
* Add migration detection UI components
Add MigrationBanner component with clear user instructions for database schema updates. Add useMigrationStatus hook for periodic health check monitoring with graceful error handling.
* Integrate migration banner into main app
Add migration status monitoring and banner display to App.tsx. Shows migration banner when database schema updates are required.
* Enhance backend startup error instructions
Add detailed Docker restart instructions and migration script guidance. Improves user experience when encountering startup failures.
* Add database schema caching to health endpoint
Implement smart caching for schema validation to prevent repeated database queries. Cache successful validations permanently and throttle failures to 30-second intervals. Replace debug prints with proper logging.
* Clean up knowledge API imports and logging
Remove duplicate import statements and redundant logging. Improves code clarity and reduces log noise.
* Remove unused instructions prop from MigrationBanner
Clean up component API by removing instructions prop that was accepted but never rendered. Simplifies the interface and eliminates dead code while keeping the functional hardcoded migration steps.
* Add schema_valid flag to migration_required health response
Add schema_valid: false flag to health endpoint response when database schema migration is required. Improves API consistency without changing existing behavior.
---------
Co-authored-by: Claude <noreply@anthropic.com>
- Add include_archived parameter to TaskService.list_tasks()
- Service now conditionally applies archived filter based on parameter
- Add 'archived' field to task DTO for client visibility
- Update API endpoints to pass include_archived down to service
- Remove redundant client-side filtering in API layer
- Fix type hints in integration tests (dict[str, Any] | None)
- Use pytest.skip() instead of return for proper test reporting
These fixes address the functional bug identified by CodeRabbit where
archived tasks couldn't be retrieved even when explicitly requested.
- Add include_content parameter to ProjectService.list_projects()
- Add exclude_large_fields parameter to TaskService.list_tasks()
- Add include_content parameter to DocumentService.list_documents()
- Update all MCP tools to use lightweight responses by default
- Fix critical N+1 query problem in ProjectService (was making separate query per project)
- Add response size monitoring and logging for validation
- Add comprehensive unit and integration tests
Results:
- Projects endpoint: 99.3% token reduction (27,055 -> 194 tokens)
- Tasks endpoint: 98.2% token reduction (12,750 -> 226 tokens)
- Documents endpoint: Returns metadata with content_size instead of full content
- Maintains full backward compatibility with default parameters
- Single query optimization eliminates N+1 performance issue
- Fixed test_update_task_status to use individual parameters
- Added test_update_task_no_fields for validation testing
- All MCP tests passing (44 tests)
Applied automated linting and formatting:
- Fixed missing newlines at end of files
- Adjusted line wrapping for better readability
- Fixed multi-line string formatting in tests
- No functional changes, only style improvements
All 43 tests still passing after formatting changes.
Based on latest PR #306 review feedback:
Fixed Issues:
- Replaced last remaining basic error handling with MCPErrorFormatter
in version_tools.py get_version function
- Added proper error handling for invalid env vars in get_max_polling_attempts
- Improved type hints with TaskUpdateFields TypedDict for better validation
- All tools now consistently use get_default_timeout() (verified with grep)
Test Improvements:
- Added comprehensive tests for MCPErrorFormatter utility (10 tests)
- Added tests for timeout_config utility (13 tests)
- All 43 MCP tests passing with new utilities
- Tests verify structured error format and timeout configuration
Type Safety:
- Created TaskUpdateFields TypedDict to specify exact allowed fields
- Documents valid statuses and assignees in type comments
- Improves IDE support and catches type errors at development time
This completes all priority actions from the review:
✅ Fixed inconsistent timeout usage (was already done)
✅ Fixed error handling inconsistency
✅ Improved type hints for update_fields
✅ Added tests for utility modules
Comprehensive update to MCP server error handling:
Error Handling Improvements:
- Applied MCPErrorFormatter to all remaining MCP tool files
- Replaced all hardcoded timeout values with configurable timeout system
- Converted all simple string errors to structured error format
- Added proper httpx exception handling with detailed context
Tools Updated:
- document_tools.py: All 5 document management tools
- version_tools.py: All 4 version management tools
- feature_tools.py: Project features tool
- project_tools.py: Remaining 3 project tools (get, list, delete)
- task_tools.py: Remaining 4 task tools (get, list, update, delete)
Test Improvements:
- Removed backward compatibility checks from all tests
- Tests now enforce structured error format (dict not string)
- Any string error response is now considered a bug
- All 20 tests passing with new strict validation
This completes the error handling refactor for all MCP tools,
ensuring consistent client experience and better debugging.
- Create test structure mirroring features folder organization
- Add tests for document tools (create, list, update, delete)
- Add tests for version tools (create, list, restore, invalid field handling)
- Add tests for task tools (create with sources, list with filters, update, delete)
- Add tests for project tools (create with polling, list, get)
- Add tests for feature tools (get features with various structures)
- Mock HTTP client for all external API calls
- Test both success and error scenarios
- 100% test coverage for critical tool functions
- Use python-jose (already in dependencies) instead of PyJWT for JWT decoding
- Make unknown Supabase key roles fail fast per alpha principles
- Skip all JWT validations (not just signature) when checking role
- Update tests to expect failure for unknown roles
Fixes:
- No need to add PyJWT dependency - python-jose provides JWT functionality
- Unknown key types now raise ConfigurationError instead of warning
- JWT decode properly skips all validations to only check role claim
- Add backend validation to detect and warn about anon vs service keys
- Prevent startup with incorrect Supabase key configuration
- Consolidate frontend state management following KISS principles
- Remove duplicate state tracking and sessionStorage polling
- Add clear error display when backend fails to start
- Improve .env.example documentation with detailed key selection guide
- Add comprehensive test coverage for validation logic
- Remove unused test results checking to eliminate 404 errors
The implementation now warns users about key misconfiguration while
maintaining backward compatibility. Frontend state is simplified with
MainLayout as the single source of truth for backend status.
- Add is_binary_file() method to URLHandler to detect 40+ binary extensions
- Update RecursiveCrawlStrategy to filter binary URLs before crawl queue
- Add comprehensive unit tests for binary file detection
- Prevents net::ERR_ABORTED errors when crawler encounters ZIP, PDF, etc.
This fixes the issue where the crawler was treating binary file URLs
(like .zip downloads) as navigable web pages, causing errors in crawl4ai.
