* fixed the llms.txt/fulls-llm.txt/llms.md etc. to be crawleed finally. intelligently determines if theres links in the llms.txt and crawls them as it should. tested fully everything works!
* updated coderabbits suggestion - resolved
* refined to code rabbits suggestions take 2, should be final take. didnt add the max link paramter suggestion though.
* 3rd times the charm, added nit picky thing from code rabbit. code rabbit makes me crave nicotine
* Fixed progress bar accuracy and OpenAI API compatibility issues
Changes Made:
1. Progress Bar Fix: Fixed llms.txt crawling progress jumping to 90% then regressing to 45% by adjusting batch crawling progress ranges (20-30% instead of 40-90%) and using consistent ProgressMapper ranges
2. OpenAI API Compatibility: Added robust fallback logic in contextual embedding service to handle newer models (GPT-5) that require max_completion_tokens instead of max_tokens and don't support custom temperature values
Files Modified:
- src/server/services/crawling/crawling_service.py - Fixed progress ranges
- src/server/services/crawling/progress_mapper.py - Restored original stage ranges
- src/server/services/embeddings/contextual_embedding_service.py - Added fallback API logic
Result:
- Progress bar now smoothly progresses 030% (crawling) 35-80% (storage) 100%
- Automatic compatibility with both old (GPT-4.1-nano) and new (GPT-5-nano) OpenAI models
- Eliminates max_tokens not supported and temperature not supported errors
* removed gpt-5-handlings since thats a seprate issue and doesnt pertain to here, definitley recommend looking at that though since gpt-5-nano is considered a reasoning model and doesnt use max_tokens, requires a diffrent output. also removed my upsert fix from documentstorage since thats not apart of this exact issue and i have another PR open for it. checked in code rabbit in my ide no issues, no nitpicks. should be good? might flag me for the UPSERT logic not being in here. owell has nothing to do with this was pr, was submitted in the last revision by mistake. everythings tested and good to go!
* fixed the llms-full.txt crawling issue. now crawls just that page when crawling llms-full.txt. fixed the 100% crawl url when multiple urls are present and hasnt finished crawling. also fixed a styling issue in CrawlingProgressCard.tsx , when batching code examples the batching progress bar would sometimes glitch out of the ui fixed it to where it wont do that now.
* fixed a few things so it will work with the current branch!
* added some enhancemments to ui rendering aswell and other little misc. fixes from code rabbit
---------
Co-authored-by: Chillbruhhh <joshchesser97@gmail.com>
Co-authored-by: Claude Code <claude@anthropic.com>
8172067413