# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

KeywordMatcherPro is a Laravel 10 SEO tool that analyzes e-commerce websites to recommend optimal target keywords for each page. The application processes Screaming Frog CSV exports through a queue-based pipeline that fetches content, extracts keywords using AI, retrieves search metrics, and generates Excel reports.

**Target deployment:** o2switch Pro hosting at `https://kodepilot.com/keywordmatcherpro`

## Essential Commands

### Development Setup
```bash
# Install dependencies
composer install

# Setup environment
cp .env.example .env
php artisan key:generate

# Database
php artisan migrate
php artisan migrate:fresh  # Reset database

# Clear all caches
php artisan cache:clear && php artisan config:clear && php artisan route:clear
```

### Queue Management
```bash
# Process queue (required for page analysis)
php artisan queue:work

# Process single job for testing
php artisan queue:work --once

# View failed jobs
php artisan queue:failed

# Retry all failed jobs
php artisan queue:retry all

# Clear failed jobs
php artisan queue:flush
```

### Testing & Debugging
```bash
# Laravel tinker for testing services
php artisan tinker

# View logs in real-time
tail -f storage/logs/laravel.log

# Test a single page analysis (in tinker)
$page = App\Models\Page::first();
App\Jobs\ProcessPage::dispatch($page);
```

### Production Deployment
```bash
# Deploy to o2switch
composer install --no-dev --optimize-autoloader
php artisan migrate --force
php artisan config:cache
php artisan route:cache
```

## Architecture

### Processing Pipeline

The application follows a multi-stage asynchronous pipeline:

1. **CSV Import** (`ScreamingFrogImporter`) → Creates `Project` and `Page` records
2. **Queue Dispatch** (`ProcessProject` job) → Batches pages for processing
3. **Page Processing** (`ProcessPage` job) → Four-step pipeline:
   - Fetch content via Jina.ai (with 30-day cache)
   - Extract 15-20 keywords via OpenAI
   - Fetch search metrics via DataForSEO (optional, with 7-day cache)
   - Calculate composite scores
4. **Report Generation** (`GenerateReport` job) → Creates Excel file when project completes

### Service Layer Architecture

**Key Services** (`app/Services/`):

- **JinaContentExtractor**: Fetches clean content from URLs
  - Implements aggressive 30-day caching via `JinaCache` model
  - Rate limiting: 100ms between requests
  - Returns max 1000 characters

- **AIKeywordExtractor**: OpenAI GPT-4o-mini integration
  - Returns structured JSON with keywords, relevance scores, search intent
  - Temperature: 0.3 for consistency
  - Rate limiting: 50ms between requests

- **DataForSEOClient**: Search volume and competition data
  - Batches up to 1000 keywords per request
  - Implements 7-day caching via `KeywordMetricsCache` model
  - Can be disabled via `DATAFORSEO_ENABLED=false`

- **ScoringEngine**: Composite scoring algorithm
  - **With search data**: 40% AI relevance, 25% search volume, 15% competition, 15% intent, 5% specificity
  - **Without search data**: 60% AI relevance, 25% intent, 15% specificity
  - Logarithmic scale for search volume normalization
  - Inverts competition (lower is better)

- **ScreamingFrogImporter**: CSV parsing and validation
  - Auto-detects page types from URL patterns
  - Skips non-HTML files (.pdf, .jpg, .css, etc.)
  - Maps common Screaming Frog column variations

- **ExcelReportGenerator**: Multi-sheet Excel reports using PhpSpreadsheet
  - Summary sheet: Project metadata
  - Recommendations sheet: One row per URL with primary keyword
  - Page Type Breakdown: Statistics by page type

### Database Schema

**Core Tables:**
- `projects`: Client projects with status tracking
- `pages`: Individual URLs with status progression (pending → content_fetched → analyzed → completed)
- `keyword_recommendations`: Extracted keywords with scores (linked to pages)
- `jina_cache`: 30-day URL content cache (keyed by MD5 hash)
- `keyword_metrics_cache`: 7-day search metrics cache (keyed by keyword+location MD5)
- `api_usage_logs`: Cost tracking for all API calls

**Page Status Flow:**
```
pending → content_fetched → analyzed → completed
                                    ↓
                                 failed
```

### Configuration Architecture

**Environment-Based Config** (`config/services.php`):
- All API credentials in `.env`
- Feature flags: `DATAFORSEO_ENABLED`
- Rate limits configurable per service
- Cache durations configurable

**Critical .env Variables:**
```env
OPENAI_API_KEY=sk-...          # Required
JINA_API_KEY=jina_...          # Required
DATAFORSEO_ENABLED=false       # Optional feature
QUEUE_CONNECTION=database      # Must be database for o2switch
```

### Caching Strategy

**Two-tier caching system:**

1. **Content Cache** (`JinaCache` model):
   - Key: `md5($url)`
   - TTL: 30 days (configurable via `JINA_CACHE_DAYS`)
   - Prevents duplicate Jina API calls (80%+ hit rate on re-analysis)

2. **Metrics Cache** (`KeywordMetricsCache` model):
   - Key: `md5($keyword . $locationCode)`
   - TTL: 7 days (configurable via `METRICS_CACHE_DAYS`)
   - Reduces DataForSEO costs significantly

**Cache checking pattern** (ALWAYS check cache before API):
```php
// Example from JinaContentExtractor
$cached = JinaCache::getCachedContent($url);
if ($cached) {
    return $cached;  // Skip API call
}
// Only then make API request
```

### Queue System

**Database-driven queue** (required for o2switch):
- Jobs stored in `jobs` table
- Failed jobs in `failed_jobs` table
- Processed via cron: `* * * * * php artisan schedule:run`

**Job Hierarchy:**
```
ProcessProject (dispatched once per project)
    └─> ProcessPage (dispatched per page, batched by 10)
            └─> GenerateReport (dispatched when all pages complete)
```

**Job Retry Strategy:**
- `ProcessPage`: 3 retries, 300s timeout
- `GenerateReport`: 3 retries, 300s timeout
- Failed jobs update page status to 'failed'

## API Integration Notes

### Rate Limiting
All API services implement rate limiting via `usleep()`:
- Jina: 100ms between requests
- OpenAI: 50ms between requests
- DataForSEO: 50ms between requests (+ 5s between batches)

### Cost Optimization
Per 100 pages (with caching):
- Jina: ~$0.10 (free tier eligible)
- OpenAI: ~$0.15
- DataForSEO: ~$0.10 (optional)
- **Total: ~$0.35** (~$0.05 on re-analysis)

### Error Handling
All services log errors via Laravel's Log facade and continue processing:
- API failures update page status to 'failed'
- Project continues processing remaining pages
- Error messages stored in `error_message` column

## Testing a Complete Flow

```php
// In tinker
use App\Models\Project;
use App\Services\ScreamingFrogImporter;
use App\Jobs\ProcessProject;

// 1. Create project
$project = Project::create([
    'user_id' => 1,
    'client_name' => 'Test Client',
    'domain' => 'example.com',
]);

// 2. Import CSV
$importer = new ScreamingFrogImporter();
$count = $importer->import('/path/to/screaming-frog-export.csv', $project);

// 3. Dispatch processing
ProcessProject::dispatch($project);

// 4. Run queue worker in separate terminal
// php artisan queue:work

// 5. Monitor progress
$project->refresh();
echo "Processed: {$project->processed_pages}/{$project->total_pages}\n";

// 6. Check for completion
$project->status;  // Should be 'completed' when done
$project->excel_report_path;  // Path to generated report
```

## Common Issues

### Queue Not Processing
- **Symptom**: Pages stuck in 'pending' status
- **Solution**: Ensure cron is running: `* * * * * php artisan schedule:run`
- **Debug**: Run `php artisan queue:work --once` manually

### API Errors
- **Jina 404**: URL not accessible or blocked by firewall → marked as failed, processing continues
- **OpenAI rate limit**: Increase `OPENAI_RATE_LIMIT_MS` in .env
- **DataForSEO auth failed**: Check login/password are correct (not account password, but API password)

### Cache Not Working
- Check `jina_cache` and `keyword_metrics_cache` tables for entries
- Verify `expires_at` timestamps are in future
- Clear expired: `App\Models\JinaCache::clearExpired()`

### Memory Issues
- Increase PHP memory limit in `.htaccess` or `php.ini`
- Reduce batch size: `BATCH_SIZE=5` in `.env`
- Limit keywords per page: `MAX_KEYWORDS_PER_PAGE=10`

## Development Workflow

When adding new features:

1. **Models**: Update migrations first, then models with relationships
2. **Services**: Add new service classes in `app/Services/` with dependency injection
3. **Jobs**: Create queueable jobs implementing `ShouldQueue` interface
4. **Testing**: Test services in isolation via tinker before integrating into jobs
5. **Logging**: Use `Log::info()` for key events, `Log::error()` for failures
6. **Caching**: Always implement caching for external API calls (follow JinaCache pattern)
7. **Configuration**: Add new config to `config/services.php` and document in `.env.example`

## File Locations

**Reports**: `storage/app/exports/keyword-report-{project_id}-{timestamp}.xlsx`
**Logs**: `storage/logs/laravel.log`
**Uploads**: `storage/app/uploads/` (for CSV files)
**API Docs**: `/docs` folder contains complete API documentation for Jina, OpenAI, DataForSEO

## Deployment Notes

- Production runs on **o2switch Pro** shared hosting (PHP 8.2, MySQL 8.0)
- Queue processing via **cron**, not Supervisor (shared hosting limitation)
- **No Node.js/npm** - pure Laravel backend
- Subdomain setup: Document root must point to `public/` folder
- See `DEPLOYMENT.md` for complete o2switch deployment instructions
- See `API_SETUP.md` for obtaining all required API keys